Bimodal fusion sensor calibration method and device, sensor and storage medium

Abstract

The invention provides a bimodal fusion sensor calibration method and device, a sensor and a storage medium, and the method comprises the steps: obtaining a calibration image file collected by a bimodal sensor, and the calibration image file comprises a test voltage value corresponding to an EVS pixel and a test gray value corresponding to an APS pixel; obtaining an interpolation gray value of the EVS pixel according to the test gray value of the adjacent APS pixel corresponding to the EVS pixel; according to the interpolation gray values of the multiple EVS pixels and the test voltage values of the multiple EVS pixels, the mapping relation between the voltage values collected by the bimodal sensor and the gray values is constructed; according to the mapping relation and the real-time voltage value collected by the bimodal sensor, the mapping gray value of the EVS pixel is obtained through calculation, and the method aims at accurately determining the gray value of the EVS pixel through the information collected by the EVS pixel, so that the quality of a gray image collected by the bimodal fusion sensor is improved.

Description

Technical Field

[0001] This application relates to the field of image sensors, and in particular, to a calibration method, device, sensor, and storage medium for a dual-modal fusion sensor. Background Art

[0002] The dynamic event vision sensor (EVS) is a new type of sensor that mimics the human retina and generates event signals in response to pixel point pulses of brightness changes caused by motion. Therefore, it can capture the brightness changes of a scene at an extremely high frame rate, record events at specific time points and specific positions in an image, and form an event stream rather than a frame stream. The dynamic event vision sensor solves problems such as high power consumption, low frame rate, and poor dynamic range of APS pixel sensors (APS, Active Pixel Sensor). However, in imaging, it only retains the motion edge information and loses the rich detail information of the object.

[0003] In related technologies, a dual-modal fusion sensor has been proposed to integrate the functions of APS pixels and EVS pixels. Currently, when using the dual-modal fusion sensor to output a grayscale image, only the grayscale image corresponding to the APS pixels can be output, while the EVS pixels cannot collect grayscale information and thus do not participate in the grayscale image output. This causes the EVS pixels corresponding to the dynamic event vision sensor to lack grayscale values, resulting in a low quality of the grayscale image collected by the dual-modal fusion sensor. Summary of the Invention

[0004] This application provides a calibration method, device, sensor, and storage medium for a dual-modal fusion sensor, aiming to solve the technical problem that the EVS pixels corresponding to the EVS pixels lack grayscale values, resulting in a low quality of the grayscale image collected by the dual-modal fusion sensor.

[0005] On the one hand, this application provides a calibration method for a dual-modal fusion sensor, and the method includes:

[0006] Obtain a calibration image file collected by the dual-modal fusion sensor, where the calibration image file includes the test voltage value corresponding to the EVS pixels and the test grayscale value corresponding to the APS pixels;

[0007] Obtain the interpolated grayscale value of the EVS pixels according to the test grayscale values of the adjacent APS pixels corresponding to the EVS pixels;

[0008] Construct a mapping relationship between the voltage value and the grayscale value collected by the dual-modal fusion sensor according to the interpolated grayscale values of multiple EVS pixels and their test voltage values;

[0009] Calculate the mapped gray value of the EVS pixel according to the mapping relationship and the real-time voltage value collected by the bimodal fusion sensor.

[0010] In some embodiments of the present application, after obtaining the calibrated image file collected by the bimodal fusion sensor, the method further includes:

[0011] For each APS pixel, determine the pixel gray value continuous type corresponding to the APS pixel according to the neighborhood gray difference value of the test gray values between the APS pixel and the neighborhood APS pixels of the APS pixel;

[0012] Obtain the gray level corresponding to each APS pixel according to the preset gray distribution rule and the test gray value of each APS pixel;

[0013] Take the gray level corresponding to the APS pixel with the pixel gray value continuous type of gray gradient pixel as the effective gray level, and count the types of effective gray levels based on the same effective gray level;

[0014] If the number of types of the effective gray levels is greater than the preset type threshold, obtain the interpolation gray value of the EVS pixel according to the test gray values of the adjacent APS pixels corresponding to the EVS pixel.

[0015] In some embodiments of the present application, the step of if the number of types of the effective gray levels is greater than the preset type threshold, obtain the interpolation gray value of the EVS pixel according to the test gray values of the adjacent APS pixels corresponding to the EVS pixel, includes:

[0016] If the number of types of the effective gray levels is greater than the preset type threshold, obtain the effective pixel position of the target APS pixel in the types of the effective gray levels;

[0017] Determine the target EVS pixels adjacent to the effective pixel position according to the preset approximation algorithm;

[0018] Obtain the interpolation gray value of the target EVS pixel according to the test gray values of the adjacent APS pixels corresponding to the target EVS pixel.

[0019] In some embodiments of the present application, the step of for each APS pixel, determine the pixel gray value continuous type corresponding to the APS pixel according to the neighborhood gray difference value of the test gray values between the APS pixel and the neighborhood APS pixels of the APS pixel, includes:

[0020] For each APS pixel, compare the neighborhood gray difference value with the preset gray difference value;

[0021] Count the number of APS pixels corresponding to the APS pixels within a preset area where the neighborhood gray - scale difference is less than the preset gray - scale difference;

[0022] If the number of pixels is greater than or equal to a preset number of pixels, determine that the APS pixel is a gray - scale gradually changing pixel;

[0023] If the number of pixels is less than the preset number of pixels, determine that the APS pixel is a gray - scale suddenly changing pixel.

[0024] In some embodiments of the present application, before obtaining the interpolated gray - scale value of the EVS pixel according to the measured gray - scale values of the adjacent APS pixels corresponding to the EVS pixel, the method further includes:

[0025] Obtain the temperature change value when the dual - mode fusion sensor collects the calibration image file;

[0026] If the temperature change value is greater than a preset temperature change threshold, re - obtain the calibration image file collected by the dual - mode fusion sensor;

[0027] If the temperature change value is less than or equal to the preset temperature change threshold, obtain the interpolated gray - scale value of the EVS pixel according to the measured gray - scale values of the adjacent APS pixels corresponding to the EVS pixel.

[0028] In some embodiments of the present application, after obtaining the calibration image file collected by the dual - mode fusion sensor, the method further includes:

[0029] Divide the pixel area according to the arrangement of the EVS pixels, where the pixel area includes at least one EVS pixel and at least one APS pixel;

[0030] For each pixel area, establish a correspondence relationship between the EVS pixels and the measured APS pixels in the pixel area.

[0031] In some embodiments of the present application, after calculating and obtaining the mapped gray - scale value of the EVS pixel according to the mapping relationship and the real - time voltage value collected by the dual - mode fusion sensor, it further includes:

[0032] Obtain the actual gray - scale value of the APS pixel collected by the dual - mode fusion sensor at the same moment when collecting the real - time voltage value;

[0033] According to the real - time gray - scale value of the APS pixel and the mapped gray - scale value of the EVS pixel, combine and generate and output a target gray - scale image.

[0034] On the other hand, the present application provides a dual - mode fusion sensor calibration device, and the dual - mode fusion sensor calibration device includes:

[0035] An acquisition module, configured to acquire a calibration image file collected by a dual-modal fusion sensor, where the calibration image file includes a test voltage value corresponding to an EVS pixel and a test gray value corresponding to an APS pixel;

[0036] An analog module, configured to obtain an interpolated gray value of the EVS pixel according to the test gray values of adjacent APS pixels corresponding to the EVS pixel;

[0037] A construction module, configured to construct a mapping relationship between the voltage value and the gray value collected by the dual-modal fusion sensor according to the interpolated gray values of multiple EVS pixels and their test voltage values;

[0038] A calculation module, configured to calculate a mapped gray value of the EVS pixel according to the mapping relationship and the real-time voltage value collected by the dual-modal fusion sensor.

[0039] On the other hand, the present application further provides a dual-modal fusion sensor, where the dual-modal fusion sensor includes:

[0040] One or more processors;

[0041] A memory; and

[0042] One or more applications, where the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the dual-modal fusion sensor calibration method.

[0043] On the other hand, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the dual-modal fusion sensor calibration method.

[0044] The technical solution of the embodiment of the present application includes: obtaining a calibration image file collected by a dual-modal fusion sensor, where the calibration image file includes a test voltage value corresponding to an EVS pixel and a test gray value corresponding to an APS pixel; obtaining an interpolated gray value of the EVS pixel according to the test gray value of the adjacent APS pixel corresponding to the EVS pixel; constructing a mapping relationship between the voltage value and the gray value collected by the dual-modal fusion sensor according to the interpolated gray values of multiple EVS pixels and their test voltage values; and calculating a mapped gray value of the EVS pixel according to the mapping relationship and the real-time voltage value collected by the dual-modal fusion sensor. In the technical solution of the embodiment of the present application, during the calibration phase, the test gray value of the APS pixel corresponding to the EVS pixel in the calibration image file collected by the dual-modal sensor is interpolated to obtain an interpolated gray value simulating the gray value of the EVS pixel, so as to construct a mapping relationship between the voltage value and the gray value. In the application phase, according to the mapping relationship, the real-time voltage value collected by the dual-modal fusion sensor is calculated and converted to obtain the mapped gray value of the EVS pixel. In this way, the mapped gray value is obtained by using the voltage value collected by the EVS pixel in the dual-modal fusion sensor to fill the gray value gap of the EVS pixel in the gray image collected by the dual-modal fusion sensor, so that a gray image with higher resolution, higher frame rate, and wider dynamic range can be combined and generated, improving the quality of the gray image output by the dual-modal fusion sensor. Description of the Drawings

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0046] Figure 1 It is a schematic flowchart of an embodiment of the dual-modal fusion sensor calibration method provided in the embodiment of the present application;

[0047] Figure 2 It is a schematic diagram of the image corresponding to the calibration image file collected by the dual-modal fusion sensor involved in the embodiment of the present application;

[0048] Figure 3 It is a schematic flowchart of an embodiment of determining whether the calibration image file can be used to construct a mapping relationship in the dual-modal fusion sensor calibration method provided in the embodiment of the present application;

[0049] Figure 4 It is a schematic flowchart of an embodiment of screening EVS pixels for constructing a mapping relationship according to the types of effective gray levels in the dual-modal fusion sensor calibration method provided in the embodiment of the present application;

[0050] Figure 5 It is a schematic flowchart of another embodiment for determining whether a calibration image file can be used to construct a mapping relationship in the dual-modal fusion sensor calibration method provided in the embodiments of the present application;

[0051] Figure 6 It is a schematic structural diagram of an embodiment of the dual-modal fusion sensor calibration device provided in the embodiments of the present application;

[0052] Figure 7 It is a schematic structural diagram of an embodiment of the dual-modal fusion sensor provided in the embodiments of the present application. Detailed implementation manners

[0053] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of the present invention.

[0054] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation to the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0055] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or instance". Any embodiment described as "exemplary" in this application is not necessarily to be construed as more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the present invention. In the following description, details are set forth for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but rather to be in line with the broadest scope consistent with the principles and features disclosed in this application.

[0056] In the related art, a dual - mode fusion sensor has been proposed to fuse the functions of APS pixels and EVS pixels. However, when currently using the dual - mode fusion sensor to output a grayscale image, only the grayscale image corresponding to the APS pixels can be output, and the information collected by the EVS pixels does not participate in the output of the grayscale image, resulting in poor quality of the grayscale image output by the dual - mode fusion sensor.

[0057] Based on this, a calibration method, device, sensor, and storage medium for a dual - mode fusion sensor in the embodiments of this application are proposed, which will be described in detail below respectively.

[0058] The calibration method for the dual - mode fusion sensor in the embodiments of the present invention is applied to a calibration device for the dual - mode fusion sensor. The calibration device for the dual - mode fusion sensor can be set in the dual - mode fusion sensor or in other terminal devices, such as a computer, a mobile phone, etc. One or more processors, a memory, and one or more application programs are provided in the dual - mode fusion sensor. One or more of the application programs are stored in the memory and configured to be executed by the processor to implement the calibration method for the dual - mode fusion sensor.

[0059] The dual - mode fusion sensor in the embodiments of the present invention is mainly used for:

[0060] Obtaining a calibration image file collected by the dual - mode fusion sensor, where the calibration image file includes a test voltage value corresponding to the EVS pixels and a test grayscale value corresponding to the APS pixels; obtaining an interpolated grayscale value of the EVS pixels according to the test grayscale values of the adjacent APS pixels corresponding to the EVS pixels; constructing a mapping relationship between the voltage value and the grayscale value collected by the dual - mode fusion sensor according to the interpolated grayscale values of multiple EVS pixels and their test voltage values; and calculating a mapped grayscale value of the EVS pixels according to the mapping relationship and the real - time voltage value collected by the dual - mode fusion sensor.

[0061] In this way, the voltage values collected by the EVS pixels in the dual-modal fusion sensor are used to calculate and convert the mapped gray values, so as to fill the gray value gaps of the EVS pixels in the gray image collected by the dual-modal fusion sensor, thereby generating a gray image with higher resolution, higher frame rate, and wider dynamic range, improving the quality of the gray image output by the dual-modal fusion sensor.

[0062] Hereinafter, through specific exemplary solutions, the content claimed in the claims of the present invention will be explained and illustrated, so that those skilled in the art can better understand the protection scope of the claims of the present invention. It can be understood that the following exemplary solutions do not limit the protection scope of the present invention, but are only used to explain the present invention.

[0063] Exemplarily, as Figure 1 shown, Figure 1 FIG. is a schematic flowchart of an embodiment of the dual-modal fusion sensor calibration method in the embodiment of the present application. The dual-modal fusion sensor calibration method includes the following steps 101 to 104:

[0064] 101. Obtain a calibration image file collected by the dual-modal fusion sensor, where the calibration image file includes the test voltage value corresponding to the EVS pixel and the test gray value corresponding to the APS pixel;

[0065] In this embodiment, the dual-modal fusion sensor calibration method can be applied to a dual-modal fusion sensor calibration device. The dual-modal fusion sensor calibration device can be set in the dual-modal fusion sensor or can be a terminal device connected to the dual-modal fusion sensor.

[0066] The dual-modal fusion sensor can collect a calibration image file for a target scene. The calibration image file can be a file in RAW format. RAW refers to the binary format in which the dual-modal fusion sensor converts the captured light source signal into a data signal.

[0067] The dual - mode fusion sensor includes multiple sensing units. The calibration image file includes information collected by the sensing units of the corresponding dual - mode fusion sensor, and each sensing unit corresponds to a pixel. According to the different data types collected by the sensing units, the pixels corresponding to the sensing units are also different. The sensing units in the dual - mode fusion sensor can be divided into dynamic event vision pixels, that is, EVS (EVS, Event - based Vision Sensor) pixels and active pixels, that is, APS (APS, Active Pixel Sensor) pixels. In this embodiment, the sensing area in the dual - mode fusion sensor can be divided into multiple sub - sensing areas, and each sub - sensing area includes at least one EVS pixel and at least one APS pixel. In this way, compared with the dual - mode vision sensor composed of multiple discrete sensor modules, the device volume of the sensor is effectively compressed, which is more conducive to the miniaturization of the overall architecture, and eliminates the inspection and image fusion matching problems between different sensors.

[0068] In actual application, when the dual - mode fusion sensor collects the calibration image file, the EVS pixels and APS pixels in the dual - mode fusion sensor can be synchronously exposed. The EVS pixel is configured with a photodiode, which is integrated with a capacitor that accumulates charge. It generates a photocurrent in response to the incident light intensity, and then generates a corresponding voltage value according to the photocurrent.

[0069] Among them, the data collected by the EVS pixel is a voltage value, and in the calibration stage, this voltage value is used as the test voltage value in this embodiment; the APS pixel in the dual - mode fusion sensor collects a gray value, and in the calibration stage, this gray value is used as the test gray value in this embodiment. The voltage value of the EVS pixel and the gray value corresponding to the adjacent APS pixel together form the above - mentioned calibration image file.

[0070] Exemplarily, each sensing area in the dual - mode fusion sensor includes at least one EVS pixel and at least one APS pixel. Please refer to Figure 2 , the EVS pixels and APS pixels are arranged in an array, where A represents an APS pixel and A / E represents an EVS pixel. It should be understood that Figure 2 the array size in is only for illustrative purposes and should not be understood as the only limitation on the array size of the overall pixel array in actual applications. And the layout and quantity of the two types of pixels in the overall pixel array can be determined according to the actual application scenario, and this embodiment does not make a unique limitation.

[0071] 102. Obtain the interpolated gray value of the EVS pixel according to the test gray value of the adjacent APS pixel corresponding to the EVS pixel;

[0072] In this embodiment, the process, pixel arrangement, etc. of the calibration image files collected by the dual-modal fusion sensor during the calibration stage and the calibration image files collected during the application stage can be the same. The calibration image files collected during the calibration stage include the test voltage values corresponding to the EVS pixels and the test gray values corresponding to the APS pixels.

[0073] In this embodiment, it is necessary to obtain the mapping relationship between the voltage values collected by the dual-modal fusion sensor and the corresponding gray values, that is, the mapping relationship between the voltage values collected by the EVS pixels and the corresponding gray values. Then, during the calibration stage, it is necessary to obtain the test voltage values and gray values corresponding to the EVS pixels. Each EVS pixel corresponds to a number of APS pixels. The APS pixels can be the APS pixels within a certain range near the EVS pixel or adjacent APS pixels.

[0074] Optionally, the APS pixels adjacent to the EVS pixels can be used as the APS pixels corresponding to the EVS.

[0075] Optionally, after obtaining the calibration image file collected by the dual-modal fusion sensor, the method further includes:

[0076] Dividing the pixel area according to the arrangement of the EVS pixels, where the pixel area includes at least one EVS pixel and at least one APS pixel;

[0077] For each pixel area, establishing the correspondence between the EVS pixels and the test APS pixels in the pixel area.

[0078] Dividing the image corresponding to the calibration image file into multiple pixel areas. The division rule can be determined according to the arrangement of the EVS pixels in the dual-modal fusion sensor, so that each pixel area includes at least one EVS pixel and at least one APS pixel. For each pixel area, the APS pixels in the same pixel area as the EVS pixels are used as the APS pixels corresponding to the EVS pixels in that area, thereby establishing the correspondence between the EVS pixels and the APS pixels in the pixel area. Refer to Figure 2 , the image corresponding to the calibration image file is divided into 9 array pixel areas. Each array pixel area includes 2*2 pixels, and each array pixel area includes at least one APS pixel and at least one EVS pixel.

[0079] In this way, taking the EVS pixels as the standard, different pixel areas are divided, so that each APS pixel and the EVS pixel are in the same pixel area, thereby improving the authenticity of the interpolated gray value obtained by fusing the gray values of the EVS pixels and improving the authenticity of the preset mapping relationship of the calibration.

[0080] In this embodiment, for EVS pixels, the gray value of the corresponding APS pixels can be referred to, the measured gray values of the respective APS pixels corresponding to the EVS pixels can be fused, and the interpolated gray value of the EVS pixel can be calculated. Refer to Figure 2 , for an array pixel region including 2×2 pixels, the interpolated gray value of the tested EVS pixel can be calculated based on the measured gray values of 3 adjacent APS pixels in each array pixel region.

[0081] Optionally, the method of fusing the measured gray values of the respective APS pixels corresponding to the EVS pixel can be to perform weighted average calculation on the measured gray values corresponding to the APS pixels, and use the result of the weighted average calculation as the interpolated gray value of the EVS pixel.

[0082] Furthermore, the relative distance between the APS pixel and the EVS pixel is obtained, and the weight of the measured gray value corresponding to each APS pixel is determined according to the ratio between the relative distances corresponding to the respective APS pixels and the EVS pixel. The greater the relative distance, the smaller the corresponding weight. For example, a larger weight can be assigned to the measured gray value corresponding to the APS pixel that is closer to the position of the EVS pixel, while a larger weight can be assigned to the measured gray value corresponding to the APS pixel that is farther from the position of the EVS pixel.

[0083] 103. According to the interpolated gray values of multiple said EVS pixels and their measured voltage values, construct the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor;

[0084] In this embodiment, the measured voltage value can be collected based on the EVS pixel. The corresponding interpolated gray value is obtained through the measured gray value of the corresponding APS pixel. According to the measured voltage value corresponding to the EVS pixel and its interpolated gray value, they are combined into sample data, so as to determine the mapping relationship between the voltage value and the gray value collected by the EVS pixel. The more EVS pixels used to construct the mapping relationship, the more corresponding measured voltage values and the more sample data obtained by combining their interpolated gray values. The mapping relationship can be in the form of a mapping table, a mapping function, etc., and is selected according to specific requirements. The mapping table composed of the measured voltage values corresponding to multiple EVS pixels and their interpolated gray values can be used as the mapping relationship, or the function constructed according to the measured voltage values corresponding to multiple EVS pixels and their interpolated gray values can be used as the mapping relationship.

[0085] Optionally, the sample data composed of the measured voltage value corresponding to the EVS pixel and its interpolated gray value is input into a preset fitting function, the fitting parameters in the preset fitting function are calculated, and the fitting parameters are substituted into the preset fitting function to obtain the mapping relationship.

[0086] In this embodiment, the test voltage value corresponding to the EVS pixel and the sample data that can be formed by its interpolated gray value. The test voltage values corresponding to multiple EVS pixels and their interpolated gray values can form multiple sample data. The form of each sample data is (X: test voltage value, Y: interpolated gray value). According to the mapping between the interpolated gray value and the test voltage value actually output by the EVS pixel, the mapping relationship can be obtained. According to the calibration image file output by the dual-mode fusion sensor at the same calibration working moment, the sample data corresponding to different EVS pixels can be obtained respectively, and then the globally unified mapping parameters can be obtained by combining the sample data and the preset fitting function.

[0087] Input the sample data composed of the test voltage value corresponding to the EVS pixel and its interpolated gray value into the preset fitting function to calculate the fitting parameters in the fitting function. Specifically, the fitting function can be expressed as: I = f(V out ), where V out represents the test voltage value corresponding to the EVS pixel, and I represents the interpolated gray value corresponding to the EVS pixel. It should be understood that the mapping relationship of f(·) in the above fitting function can be in the form of exponential, linear, logarithmic, polynomial, etc., and this embodiment does not make a unique limitation on this.

[0088] In one implementation, if the above fitting function is implemented using a linear mapping relationship, the fitting function can be expressed as:

[0089] I = a * V out + b

[0090] where V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, and a and b represent the fitting parameters.

[0091] In another implementation, if the above fitting function is implemented using a logarithmic mapping relationship, the fitting function can be expressed as:

[0092] I = t * I 0 * log[(V - V out ) / k]

[0093] where V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, a and b represent the fitting parameters, t represents the exposure time of the APS pixel, and I 0 , V and k represent the fitting parameters.

[0094] In yet another implementation, if the above fitting function is implemented using a polynomial mapping relationship, the fitting function can be expressed as:

[0095]

[0096] Among them, V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, and a and b represent fitting parameters.

[0097] In another implementation, if the above fitting function is implemented using an exponential mapping relationship, the fitting function can be expressed as:

[0098]

[0099] Among them, V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, t represents the exposure time of the APS pixel, I 0 , V, and k represent fitting parameters.

[0100] Furthermore, the value of the above m is the natural constant e, that is, the above fitting function can be further expressed as:

[0101]

[0102] Of course, in practical applications, the value of the above constant m can be flexibly set according to the application scenario, and this embodiment does not make a unique limitation on this. In this embodiment, a preset fitting function is predefined, and the form of this fitting function is not limited to the above form provided in this embodiment, and can also be Fourier expansion, etc.

[0103] In this embodiment, the test voltage value corresponding to the EVS pixel and its interpolated gray value are both input as known quantities into the above fitting function based on the exponential mapping relationship, etc., and the unknown quantities a, b, or I 0 , V, and k in the fitting function are solved to obtain the fitting parameters. Finally, the obtained solution is substituted into the fitting function to obtain a complete fitting function I = f(V out ), and this fitting function can be used as the mapping relationship between the voltage value and the gray value collected by the EVS pixel or the dual-modal fusion sensor.

[0104] It should be noted that the obtained mapping relationship can be used as the global mapping relationship of all EVS pixels in the dual-modal fusion sensor, associated with all EVS pixels in the dual-modal fusion sensor, because the configurations and structures of all EVS pixels in the dual-modal fusion sensor can be the same, thereby reducing the calibration cost. The obtained mapping relationship can be used as the mapping relationship of the EVS pixels corresponding to the EVS pixels in the same area as the EVS pixel, thereby improving the accuracy of the mapping relationship, improving the accuracy of the calibrated mapped gray value, and improving the quality of the gray image output by the dual-modal fusion sensor.

[0105] 104. Calculate and convert to obtain the mapped grayscale value of the EVS pixel according to the mapping relationship and the real-time voltage value collected by the bimodal fusion sensor.

[0106] In this embodiment, the mapping relationship is the mapping relationship between the voltage value collected by the bimodal fusion sensor (or EVS pixel) and the grayscale value corresponding to the EVS pixel. This mapping relationship is determined by calibration. In the calibration stage, it is determined based on the sample data composed of the test voltage value and the interpolated grayscale value of the EVS pixel obtained through testing. According to the real-time voltage value corresponding to the EVS pixel collected by the current bimodal fusion sensor, the mapped grayscale value corresponding to the EVS pixel is calibrated from the mapping relationship. This mapped grayscale value can be used as the grayscale value collected by the bimodal fusion sensor based on dynamic events. Based on this grayscale value, a grayscale image with higher quality can be obtained from the bimodal fusion sensor.

[0107] Optionally, after calculating and obtaining the mapped grayscale value of the EVS pixel according to the mapping relationship and the real-time voltage value collected by the bimodal fusion sensor, it further includes:

[0108] Obtain the actual grayscale value of the APS pixel collected by the bimodal fusion sensor at the same moment when collecting the real-time voltage value.

[0109] Generate and output a target grayscale image according to the real-time grayscale value of the APS pixel and the mapped grayscale value of the EVS pixel.

[0110] In this embodiment, when the bimodal fusion sensor collects the real-time voltage value corresponding to the EVS pixel, it can collect the actual grayscale value corresponding to the APS pixel through the bimodal fusion sensor at the same moment. For example, the bimodal sensor collects a real-time image file, which includes the real-time voltage value corresponding to the EVS pixel and the actual grayscale value corresponding to the APS pixel. When collecting the real-time image file, the EVS pixel and the APS pixel of the bimodal fusion sensor are exposed simultaneously, and the real-time voltage value corresponding to the EVS pixel and the actual grayscale value corresponding to the APS pixel are collected simultaneously. After determining the mapped grayscale value of the EVS pixel according to the real-time voltage value corresponding to the EVS pixel and the mapping relationship, since the pixel positions of the EVS pixel and the APS pixel are different, based on the pixel positions of the EVS pixel and the APS pixel, the mapped grayscale value of the EVS pixel can be combined with the real-time grayscale value of the APS pixel collected at the same moment to generate a target grayscale image. The pixel positions on the target grayscale image all correspond to the APS pixel or the EVS pixel. The generation of the target grayscale image uses both the grayscale information collected by the APS pixel and the dynamic voltage information collected by the EVS pixel, making the details of the target grayscale image richer and of higher quality.

[0111] In the embodiment of the present application, a calibration image file collected by a bimodal fusion sensor is obtained. The calibration image file includes a test voltage value corresponding to an EVS pixel and a test gray value corresponding to an APS pixel. According to the test gray value of the adjacent APS pixel corresponding to the EVS pixel, an interpolated gray value of the EVS pixel is obtained. According to the interpolated gray values of a plurality of the EVS pixels and their test voltage values, the mapping relationship between the voltage value and the gray value collected by the bimodal fusion sensor is calibrated. According to the mapping relationship and the real-time voltage value collected by the bimodal fusion sensor, the mapped gray value of the EVS pixel is calculated. In this way, the voltage value collected based on dynamic events in the bimodal fusion sensor can be used to calibrate the mapped gray value corresponding to the EVS pixel. By using the mapped gray value corresponding to the EVS pixel, a gray image output by the bimodal fusion sensor can be obtained better. A gray image output by the bimodal fusion sensor can be generated according to the combination of the mapped gray values corresponding to the EVS pixels. A gray image output by the bimodal fusion sensor can also be generated according to the combination of the mapped gray values corresponding to the EVS pixels and the real-time gray values corresponding to the APS pixels collected by the APS pixels at the same time. The information collected by the EVS pixels and the APS pixels in the bimodal fusion sensor is combined to fill the gap of the gray value of the EVS pixels in the gray image collected by the bimodal fusion sensor, and the problems of low resolution, low frame rate, and narrow dynamic range of the gray image output by the bimodal fusion sensor are solved.

[0112] As Figure 3 shown, Figure 3 FIG. is a schematic flowchart of an embodiment for determining whether a calibration image file can be used to construct a mapping relationship in the bimodal fusion sensor calibration method provided in the embodiment of the present application, including steps 301 to 304:

[0113] 301. For each APS pixel, determine the pixel gray continuity type corresponding to the APS pixel according to the neighborhood gray difference between the test gray value between the APS pixel and the neighborhood APS pixels of the APS pixel.

[0114] In this embodiment, according to the test gray values corresponding to the APS pixels in the calibration image file, the gray-scale image output by the dual-mode fusion sensor based on the APS pixels can be determined. The gray-scale image includes the test gray values corresponding to each APS pixel. The gray-scale image can be further decomposed to obtain gray-scale images of different color channels, and the test gray values can also correspondingly obtain the test gray value components corresponding to different color channels. Next, the test gray values of the APS pixels and the test gray values of the neighboring APS pixels can be analyzed as a whole to determine the gray-scale continuity type of the APS pixels. It is also possible to determine the gray-scale continuity type of the APS pixels by color channel: decompose the test gray values of the APS pixels and the test gray values of the neighboring APS pixels by color channel to obtain the test gray value components of the APS pixels and the test gray value components of the neighboring APS pixels under each color channel, and analyze the test gray value components of the APS pixels and the test gray value components of the neighboring APS pixels under the same color channel to respectively obtain the pixel gray-scale continuity types of each APS pixel corresponding to different color channels. The pixel gray-scale continuity type represents a parameter of the gray-scale difference between an APS pixel and its neighboring APS pixels. The pixel gray-scale continuity type can include gray-scale mutation pixels and gray-scale gradual change pixels. The APS pixels belonging to the gray-scale mutation pixels are more prominent relative to their nearby areas, and the gray values change greatly; the APS pixels belonging to the gray-scale gradual change pixels are more coherent with the gray scale of their nearby areas, and the gray values change little.

[0115] Optionally, for each APS pixel, determine the pixel gray-scale continuity type corresponding to the APS pixel according to the neighborhood gray-scale difference between the test gray values of the APS pixel and the neighboring APS pixels of the APS pixel:

[0116] For each APS pixel, compare the neighborhood gray-scale difference with a preset gray-scale difference;

[0117] Count the number of pixels corresponding to the APS pixels in the preset area where the neighborhood gray-scale difference is less than the preset gray-scale difference;

[0118] If the number of pixels is greater than or equal to the preset number of pixels, determine that the APS pixel is a gray-scale gradual change pixel;

[0119] If the number of pixels is less than the preset number of pixels, determine that the APS pixel is a gray-scale mutation pixel.

[0120] A gray-scale image I can be generated according to the test gray values of the APS pixels in the calibration image file (m,n) =1,2,3...2 k (I∈R N / τ×M / ν I∈R N / τ×M / v, τ and v are the periods of APS pixels in the X and Y directions, k is the bit depth of the gray scale, generally 8, 12, 16, etc., m and n are the length and width of the image respectively, m ∈ 1, 2, 3...M, n ∈ 1, 2, 3...N, and M and N are the length and width of the picture respectively), which can be understood as the gray scale image collected by APS pixels in the dual-modal fusion sensor, or the component gray scale images corresponding to the gray scale image in different color channels. For APS pixels, adjacent APS pixels are determined along the X and Y directions as their corresponding neighboring APS pixels, and I (m,n) = 1, 2, 3...2 k The difference is taken between the test gray scale value corresponding to the APS pixel in k and the test gray scale value of the neighboring APS pixel, and the absolute value of the difference result is taken to obtain the neighborhood gray scale difference between the APS pixel and the neighboring APS pixel. A preset gray scale difference I threshold is set in the X and Y directions of the APS pixel. When the neighborhood gray scale difference is less than the preset gray scale difference I threshold , the spatial neighborhood event of the APS pixel is determined to be 0. When the neighborhood gray scale difference is greater than the preset gray scale difference I threshOld , the spatial neighborhood event of the APS pixel is determined to be 1. Since the APS pixel performs subtraction in the X and Y directions, then according to the spatial neighborhood events of each APS pixel, a spatial neighborhood event image

[0121] The gray scale change gradient and continuity of I (m,n) = 1, 2, 3...2 k are judged by synthesizing the spatial neighborhood event image. For each APS pixel, the number of spatial neighborhood events with a value of 0 in the preset area where the APS pixel is located is counted. If the number of 0 neighborhood events is greater than or equal to the preset event number, the gray scale difference between the APS pixel and most of the pixel points in its preset area is not large, the gray scale connection with the nearby area is relatively coherent, and the gray scale value changes little. The APS pixel is determined to be a gray scale gradually changing pixel, and the neighborhood gray scale continuity value is set to 0; otherwise, if the number of 0 neighborhood events is less than the preset event number, the gray scale difference between the APS pixel and most of the pixel points in its preset area is large, the gray scale connection with the nearby area is relatively incoherent, and the gray scale value changes greatly. The APS pixel is determined to be a gray scale suddenly changing pixel, and the neighborhood gray scale continuity value is set to 1. Thus, a neighborhood gray scale continuity map can be generated. The specific process can be expressed by the following formula:

[0122]

[0123]

[0124]

[0125] 302. Obtain the gray level corresponding to each APS pixel according to the preset gray level distribution rule and the measured gray level value of each APS pixel;

[0126] In this embodiment, a corresponding gray scale map Ι can be obtained according to the measured gray level value of each APS pixel (m,n) = 1, 2, 3, … 2 k , and plan Ι according to the preset gray level distribution rule (m,n) = 1, 2, 3, … 2 k to obtain a gray level map The gray level of the APS pixel can be determined according to the gray level map. Plan Ι according to the preset gray level distribution rule (m,n) = 1, 2, 3, … 2 k to obtain a gray level map including but not limited to the following methods:

[0127] G = 1, 2, 3, …, That is

[0128] G = 0.1, 0.2, 0.3, …,

[0129] That is

[0130] G = 10, 20, 30, …, That is

[0131] Optionally, in order to avoid overly dark or overexposed scenes, the lowest and highest levels in its gray level map can be made invalid.

[0132] 303. Use the gray level corresponding to the APS pixel whose pixel gray level continuous type is a gray scale gradient pixel as the effective gray level, and count the types of effective gray levels based on the same effective gray level;

[0133] In this embodiment, a comprehensive judgment map is obtained according to the gray level map and the neighborhood gray level continuity map :

[0134]

[0135] It can be determined according to the comprehensive judgment map that when When the pixel gray - level continuous type corresponding to the APS pixel is a gray - level gradient pixel, then further determine that in the comprehensive determination diagram, the comprehensive determination value corresponding to the position of the APS pixel is a valid gray - level. When When the pixel gray - level continuous type corresponding to the APS pixel is a gray - level mutation pixel, then in the comprehensive determination diagram, the comprehensive determination value corresponding to the position of the APS pixel is 0; thus, the number of types of gray - levels with gentle transformation in the comprehensive determination diagram can be determined. For example, there are 25 types of gray - levels (1, 2, 3, 4...25) in the gray - level diagram. After excluding the gray - level mutation pixels of non - continuous neighboring pixels through the neighborhood gray - level continuity diagram, the comprehensive determination diagram only contains the following gray - levels (1, 2, 3, 8, 8, 4, 4, 5, 4, 5), and all are valid gray - levels, with 6 types of valid gray - levels (1, 2, 3, 4, 5, 8). The determination of the types of valid gray - levels can be realized through the following formula:

[0136]

[0137] Among them, unique is the operation of searching for all unique elements, is the vector composed of unique elements in the valid gray - levels.

[0138] For the calculation method of the types of valid gray - levels by dividing different color channels, it is necessary to merge the types of valid gray - levels corresponding to a single color channel, and make a determination according to the merged types of valid gray - levels.

[0139] Specifically, if a single RGB color channel obtains a type of valid gray - level, then the total number of types of valid gray - levels after merging the R, G, and B color channels is calculated by the following formula:

[0140]

[0141] By calculating the vector length of the vector that is, the number of types of valid gray - levels; among them, length is the operation of counting the length of the vector The vector is the specific gray - level. When the vector length of the vector is greater than the preset threshold, it is determined that the APS pixel corresponding to the non - zero element position in the comprehensive determination diagram is the target APS pixel, and the gray - level corresponding to the target APS pixel is valid.

[0142] 304. If the number of types of the valid gray - levels is greater than the preset type threshold, then according to the measured gray - level values of the adjacent APS pixels corresponding to the EVS pixel, the interpolated gray - level value of the EVS pixel is obtained.

[0143] In this embodiment, after the number of valid gray level types in the comprehensive determination diagram is greater than the preset type threshold, the gray level change of the image corresponding to the currently acquired calibration image file has a sufficiently flat area, and there are multiple valid gray level types to enhance the diversity of sample data, which meets the requirements for constructing the mapping relationship, and the next step can be executed.

[0144] Optionally, if the number of valid gray level types is less than or equal to the preset type threshold, then the currently acquired calibration image file does not have enough valid gray level types to enhance the diversity of sample data, which does not meet the requirements for determining the preset mapping relationship, and it is necessary to re-acquire the calibration image file, control the dual-mode fusion sensor to re-acquire the calibration image file, and repeat the above steps.

[0145] Optionally, if the number of valid level types is less than or equal to the preset type threshold, save the valid gray level types At the same time, save the corresponding image, the effective flat area, and the corresponding adjacent APS pixel test gray value and EVS test voltage value, and issue an instruction to re-acquire a new image file, repeat the above steps, and after obtaining the latest acquired calibration image file update through the following formula:

[0146]

[0147] Obtain the target valid gray level types, compare the number of valid gray level types with the preset type threshold, and re-determine whether the newly acquired calibration image file combined with the saved image file meets the calibration requirements.

[0148] In the technical solution disclosed in this embodiment, the number of valid gray level types is determined through the spatial neighborhood events of APS pixels and the corresponding gray levels, and then the currently acquired calibration image file is determined whether it meets the requirements for constructing the mapping relationship based on the number of valid gray level types. When it does not meet the requirements, it is necessary to re-acquire a new calibration image file, so as to ensure that the number of flat areas in the total calibration image file is sufficient, ensure the accuracy of interpolating the EVS gray value from the APS neighborhood, enrich the diversity of sample data, be more conducive to fitting an accurate mapping relationship, thereby improving the accuracy of calibrating the mapped gray value of EVS pixels, and improving the quality of the gray image output by the dual-mode fusion sensor.

[0149] As Figure 4 shown, Figure 4 is a flowchart of an embodiment of screening EVS pixels for constructing a mapping relationship according to the number of valid gray level types in the dual-mode fusion sensor calibration method provided in the embodiment of the present application, including steps 401-step 403:

[0150] 401. If the number of types of effective gray levels is greater than a preset type threshold, obtain the effective pixel positions of the target APS pixels among the types of effective gray levels;

[0151] In this embodiment, if the number of types of effective gray levels is greater than the preset type threshold, the currently acquired calibration image file can be continued to be used to determine the interpolated gray value of the EVS pixels, and then the mapping relationship can be constructed. If the gray level of the APS pixel belongs to the above-mentioned types of effective gray levels, then the APS pixel is in the gray level slow change region, and it is used as the target APS pixel to determine the effective pixel position corresponding to the target APS pixel.

[0152] Exemplarily, select the effective pixel positions according to the types of effective gray levels. find is the operation of searching for the position of a specific element. is the χ-th element of the selected vector.

[0153] 402. Determine the target EVS pixels adjacent to the effective pixel positions according to a preset approximation algorithm;

[0154] In this embodiment, for EVS pixels, when they belong to the gray level slow change region, their gray levels can be interpolated from approximately adjacent APS pixels, that is, if the EVS pixels belong to the gray level slow change region, then through the measured gray value of their corresponding APS pixels, a more realistic interpolated gray value can be obtained.

[0155] For example: Suppose a bimodal fusion sensor takes a picture containing items such as a table, a blackboard, and flowers. Usually, for items like flowers that belong to the region with uneven spatial transformation, the gray values in their spatial neighborhood differ greatly, and the mapping relationship constructed based on the sample data in this region to calculate the interpolated gray value of the EVS pixels is relatively inaccurate, and this region can be identified as an invalid region. On the contrary, for items like the blackboard with relatively gentle spatial transformation, the gray values in their spatial neighborhood differ little, and the mapping relationship constructed based on the sample data in this region is relatively accurate. Therefore, the interpolated gray value of the EVS pixels calculated according to the mapping relationship and the EVS test voltage value is relatively accurate.

[0156] The region where the target APS pixel is located is the gray level slow change region, and the effective pixel position where the target APS pixel is located points to the gray level slow change region. Therefore, through the formula (transform is the conversion from the pixel position of the APS pixel to the pixel position of the EVS pixel, and the conversion can include operations such as near-neighbor interpolation approximation and other preset approximation algorithms), convert the effective pixel position of the target APS pixel into the target pixel position of the corresponding EVS pixel, and use the EVS pixel at the target pixel position as the target EVS pixel.

[0157] 403. Obtain the interpolated gray value of the target EVS pixel according to the measured gray values of the corresponding adjacent APS pixels of the target EVS pixel.

[0158] In this embodiment, since the target EVS pixel and the target APS pixel are adjacent and also in the gray-scale slow-varying region, the interpolated gray value of the neighboring target EVS pixel can be obtained according to the measured gray value of the target APS pixel. Then, by using the sample data composed of the interpolated gray value and the measured voltage value determined by the target EVS pixel, a more accurate mapping relationship can be obtained.

[0159] In the technical solution disclosed in this embodiment, the effective pixel positions of the target APS pixels in the gray-scale slow-varying region are determined by the types of effective gray levels, and then the target EVS pixels belonging to the gray-scale slow-varying region are found according to the preset approximation algorithm and the effective pixel positions. The interpolated gray values of the target EVS pixels in the gray-scale slow-varying region are more similar to the measured gray values of their corresponding APS pixels. For the target EVS pixels, more accurate interpolated gray values can be found, and the preset mapping relationship determined according to the measured voltage value and the interpolated gray value of the target EVS pixel is more accurate. Therefore, the mapped gray values calibrated according to the mapping relationship are also more accurate, further improving the quality of the output gray-scale image.

[0160] As Figure 5 shown, Figure 5 is a schematic flowchart of an embodiment for determining whether a calibration image file can be used to construct a mapping relationship in the dual-modal fusion sensor calibration method provided in an embodiment of the present application, including steps 501 to 503:

[0161] 501. Obtain the temperature change value when the dual-modal fusion sensor acquires the calibration image file.

[0162] In this embodiment, a temperature sensor is also provided in the dual-modal fusion sensor. The temperature and temperature change of the dual-modal fusion sensor at different times can be detected in the dual-modal fusion sensor. The installation position of the temperature sensor is not limited. When the dual-modal fusion sensor acquires an image, a temperature change may occur. The greater the temperature change, the more abnormal the state of the dual-modal fusion sensor, and the lower the quality of the obtained calibration image file. Therefore, the temperature change value when acquiring the calibration image file needs to be used as the judgment basis for whether the calibration image file can be used for test calibration. A temperature change threshold ΔT threshold is preset. When the dual-modal fusion sensor acquires the calibration image file, the temperature change value at this time is obtained through the temperature sensor.

[0163] 502. If the temperature change value is greater than the preset temperature change threshold, re-acquire the calibration image file acquired by the dual-modal fusion sensor.

[0164] In this embodiment, the temperature change value when collecting the calibration image file is compared with the preset temperature change threshold ΔT threshold If ΔT threshold > ΔT, the state of the dual - mode fusion sensor is abnormal when collecting the calibration image file, and it is necessary to re - obtain the calibration image file for calibration, and re - obtain the calibration image file collected by the dual - mode fusion sensor until the temperature change value when collecting the calibration image file is less than the preset temperature change threshold.

[0165] 503. If the temperature change value is less than or equal to the preset temperature change threshold, the interpolated gray value of the EVS pixel is obtained according to the measured gray values of the adjacent APS pixels corresponding to the EVS pixel.

[0166] In this embodiment, the temperature change value ΔT when collecting the calibration image file is compared with the preset temperature change threshold ΔT threshold If ΔT 4hreshold ≤ ΔT, the state of the dual - mode fusion sensor is normal when collecting the calibration image file, and it is not necessary to re - obtain the calibration image file for calibration. The next step can be executed based on the currently collected calibration image file, that is, fusing the mapped gray value corresponding to the EVS pixel and the measured gray values of each APS pixel to obtain a complete, high - resolution, and high - dynamic - range gray - scale image.

[0167] Further, when the temperature change value ΔT when collecting the calibration image file is less than or equal to the preset temperature change threshold ΔT threshold , before re - obtaining the calibration image file collected by the dual - mode fusion sensor, the following operations can also be performed: Obtain the operating temperature of the dual - mode fusion sensor after the change of this temperature change value, and detect whether there is a pre - stored mapping relationship between the voltage value and the gray value collected by the dual - mode fusion sensor associated with this operating temperature, or a pre - stored mapping relationship associated with the temperature range where this operating temperature is located. If there is a corresponding pre - stored mapping relationship, the pre - stored mapping relationship associated with this operating temperature is used as the mapping relationship for calculating the mapped gray value of the EVS pixel later; if there is no corresponding pre - stored mapping relationship, it is necessary to re - obtain the calibration image file collected by the dual - mode fusion sensor, obtain a newly calculated mapping relationship, and associate and save this new mapping relationship with the above - mentioned operating temperature or the temperature range where the above - mentioned operating temperature is located. In this way, during the calibration stage, it is not necessary to reconstruct the mapping relationship each time when the temperature change value is less than or equal to the preset temperature change threshold, thereby improving the efficiency of constructing the mapping relationship.

[0168] To better implement the dual - mode fusion sensor calibration method in the embodiments of the present application, based on the dual - mode fusion sensor calibration method, an embodiment of the present application also provides a dual - mode fusion sensor calibration device, as Figure 6 shown Figure 6 is a schematic structural diagram of an embodiment of the dual - mode fusion sensor calibration device. The dual - mode fusion sensor calibration device includes the following modules 601 - 604:

[0169] An acquisition module 601, configured to acquire a calibration image file collected by the dual - mode fusion sensor. The calibration image file includes a test voltage value corresponding to an EVS pixel and a test gray - scale value corresponding to an APS pixel;

[0170] An analog module 602, configured to obtain an interpolated gray - scale value of the EVS pixel according to the test gray - scale values of adjacent APS pixels corresponding to the EVS pixel;

[0171] A construction module 603, configured to construct a mapping relationship between the voltage value and the gray - scale value collected by the dual - mode fusion sensor according to the interpolated gray - scale values of multiple EVS pixels and their test voltage values;

[0172] A calculation module 604, configured to calculate a mapped gray - scale value of the EVS pixel according to the mapping relationship and the real - time voltage value collected by the dual - mode fusion sensor.

[0173] In some embodiments of the present application, the acquisition module 601 is further configured to, for each APS pixel, determine a pixel gray - scale continuity type corresponding to the APS pixel according to the neighborhood gray - scale difference between the test gray - scale values of the APS pixel and its neighboring APS pixels;

[0174] Obtain a gray - scale level corresponding to each APS pixel according to a preset gray - scale distribution rule and the test gray - scale values of each APS pixel;

[0175] Use the gray - scale level corresponding to the APS pixel with the pixel gray - scale continuity type of a gray - scale - gradient pixel as a valid gray - scale level, and count the types of valid gray - scale levels based on the same valid gray - scale level;

[0176] If the number of types of valid gray - scale levels is greater than a preset type threshold, obtain an interpolated gray - scale value of the EVS pixel according to the test gray - scale values of adjacent APS pixels corresponding to the EVS pixel.

[0177] In some embodiments of the present application, the analog module 602 is further configured to, if the number of types of valid gray - scale levels is greater than a preset type threshold, obtain a valid pixel position of a target APS pixel among the types of valid gray - scale levels;

[0178] Determine the target EVS pixels adjacent to the effective pixel position according to a preset approximation algorithm;

[0179] Obtain the interpolated gray value of the target EVS pixel according to the test gray value of the corresponding adjacent APS pixel of the target EVS pixel.

[0180] In some embodiments of the present application, the obtaining module 601 is further configured to compare the neighborhood gray difference with a preset gray difference for each APS pixel;

[0181] Count the number of pixels corresponding to the APS pixels in the preset area where the neighborhood gray difference is less than the preset gray difference;

[0182] If the number of pixels is greater than or equal to a preset number of pixels, determine that the APS pixel is a gray-scale gradually changing pixel;

[0183] If the number of pixels is less than the preset number of pixels, determine that the APS pixel is a gray-scale abruptly changing pixel.

[0184] In some embodiments of the present application, the obtaining module 601 is further configured to obtain the temperature change value when the dual-mode fusion sensor collects the calibration image file;

[0185] If the temperature change value is greater than a preset temperature change threshold, re-obtain the calibration image file collected by the dual-mode fusion sensor;

[0186] If the temperature change value is less than or equal to the preset temperature change threshold, obtain the interpolated gray value of the EVS pixel according to the test gray value of the adjacent APS pixel corresponding to the EVS pixel.

[0187] In some embodiments of the present application, the simulation module 602 is further configured to divide pixel regions according to the arrangement of the EVS pixels, and at least one EVS pixel and at least one APS pixel are included in the pixel regions;

[0188] For each of the pixel regions, establish a correspondence relationship between the EVS pixels and the test APS pixels in the pixel regions.

[0189] In some embodiments of the present application, the dual-mode fusion sensor calibration device includes a generation module 605, and the generation module 605 is further configured to obtain the actual gray value of the APS pixel collected by the dual-mode fusion sensor at the same moment when collecting the real-time voltage value;

[0190] Combine and generate and output a target gray image according to the real-time gray value of the APS pixel and the mapped gray value of the EVS pixel.

[0191] In the embodiments of the present application, a calibration image file collected by a dual - mode fusion sensor is obtained. The calibration image file includes a test voltage value corresponding to an EVS pixel and a test gray - scale value corresponding to an APS pixel. According to the test gray - scale value of the adjacent APS pixel corresponding to the EVS pixel, an interpolated gray - scale value of the EVS pixel is obtained. According to the interpolated gray - scale values of multiple EVS pixels and their test voltage values, a mapping relationship between the voltage value and the gray - scale value collected by the dual - mode fusion sensor is constructed. According to the mapping relationship and the real - time voltage value collected by the dual - mode fusion sensor, the mapped gray - scale value of the EVS pixel is calculated. In the technical solution of the embodiments of the present application, during the calibration phase, the test gray - scale value of the APS pixel corresponding to the EVS pixel in the calibration image file collected by the dual - mode sensor is used to interpolate an interpolated gray - scale value simulating the gray - scale value of the EVS pixel, so as to construct a mapping relationship between the voltage value and the gray - scale value. In the application phase, according to the mapping relationship, the real - time voltage value collected by the dual - mode fusion sensor is calculated and converted to obtain the mapped gray - scale value of the EVS pixel. In this way, the mapped gray - scale value is obtained by using the voltage value collected by the EVS pixel in the dual - mode fusion sensor to fill the gray - scale value gap of the EVS pixel in the gray - scale image collected by the dual - mode fusion sensor, so that a gray - scale image with higher resolution, higher frame rate, and wider dynamic range can be combined and generated, improving the quality of the gray - scale image output by the dual - mode fusion sensor.

[0192] An embodiment of the present invention also provides a dual - mode fusion sensor, as Figure 7 shown Figure 7 is a schematic structural diagram of an embodiment of the dual - mode fusion sensor provided in the embodiments of the present application.

[0193] The dual - mode fusion sensor integrates any one of the dual - mode fusion sensor calibration devices provided in the embodiments of the present invention. The dual - mode fusion sensor includes:

[0194] One or more processors;

[0195] A memory; and

[0196] One or more applications, where the one or more applications are stored in the memory and are configured to be executed by the processor to perform the steps in the dual - mode fusion sensor calibration method described in any one of the embodiments of the dual - mode fusion sensor calibration method in the above - mentioned embodiments.

[0197] Specifically, the dual - mode fusion sensor may include a processor 701 with one or more processing cores, a memory 702 with one or more computer - readable storage media, a power supply 703, an input unit 704, and other components. Those skilled in the art can understand, Figure 7The dual - mode fusion sensor structure shown does not limit the dual - mode fusion sensor, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Among them:

[0198] The processor 701 is the control center of the dual - mode fusion sensor. It connects various parts of the entire dual - mode fusion sensor using various interfaces and circuits. By running or executing software programs and / or modules stored in the memory 702, and by calling the data stored in the memory 702, it performs various functions of the dual - mode fusion sensor and processes data, thereby monitoring the dual - mode fusion sensor as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above - mentioned modem processor may not be integrated into the processor 701.

[0199] The memory 702 can be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.); the data storage area can store data created according to the use of the dual - mode fusion sensor. In addition, the memory 702 can include high - speed random - access memory, and can also include non - volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non - volatile solid - state storage devices. Correspondingly, the memory 702 can also include a memory controller to provide the processor 701 with access to the memory 702.

[0200] The dual - mode fusion sensor also includes a power supply 703 that powers each component. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, thereby implementing functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 703 can also include any components such as one or more DC or AC power supplies, a recharge system, a power - failure detection circuit, a power converter or inverter, and a power - status indicator.

[0201] The dual - mode fusion sensor may also include an input unit 704. The input unit 704 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0202] Although not shown, the dual - mode fusion sensor may further include a display unit and the like, which will not be elaborated herein. Specifically, in this embodiment, the processor 701 in the dual - mode fusion sensor will, according to the following instructions, load the executable files corresponding to the processes of one or more application programs into the memory 702, and the processor 701 will run the application programs stored in the memory 702 to implement various functions as follows:

[0203] Obtain the calibration image file collected by the dual - mode fusion sensor, where the calibration image file includes the test voltage value corresponding to the EVS pixel and the test gray - scale value corresponding to the APS pixel;

[0204] Obtain the interpolated gray - scale value of the EVS pixel according to the test gray - scale value of the adjacent APS pixel corresponding to the EVS pixel;

[0205] Construct the mapping relationship between the voltage value and the gray - scale value collected by the dual - mode fusion sensor according to the interpolated gray - scale values of multiple EVS pixels and their test voltage values;

[0206] Calculate the mapped gray - scale value of the EVS pixel according to the mapping relationship and the real - time voltage value collected by the dual - mode fusion sensor.

[0207] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above - mentioned embodiments can be completed by instructions or by controlling relevant hardware through instructions. The instructions can be stored in a computer - readable storage medium and loaded and executed by a processor.

[0208] For this reason, an embodiment of the present invention provides a computer - readable storage medium, which may include: read - only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), a magnetic disk, an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the dual - mode fusion sensor calibration methods provided by the embodiments of the present invention. For example, when the computer program is loaded by a processor, it can execute the following steps:

[0209] Obtain the calibration image file collected by the dual - mode fusion sensor, where the calibration image file includes the test voltage value corresponding to the EVS pixel and the test gray - scale value corresponding to the APS pixel;

[0210] Obtain the interpolated gray - scale value of the EVS pixel according to the test gray - scale value of the adjacent APS pixel corresponding to the EVS pixel;

[0211] Construct a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolated gray values of multiple said EVS pixels and their test voltage values;

[0212] Calculate the mapped gray value of the EVS pixel according to the mapping relationship and the real-time voltage value collected by the dual-mode fusion sensor.

[0213] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the detailed descriptions of other embodiments above, and details will not be repeated here.

[0214] In specific implementation, the above-mentioned units or structures can be implemented as independent entities, or can be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of the above-mentioned units or structures, reference may be made to the method embodiments above, and details will not be repeated here.

[0215] For the specific implementation of the above operations, reference may be made to the previous embodiments, and details will not be repeated here.

[0216] The above has introduced in detail a calibration method for a dual-mode fusion sensor provided by an embodiment of the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A calibration method for a dual - mode fusion sensor, characterized in that, the method includes: Obtain a calibration image file collected by the dual - mode fusion sensor, where the calibration image file includes the test voltage value corresponding to the EVS pixel and the test gray - scale value corresponding to the APS pixel; According to the test gray - scale values of the adjacent APS pixels corresponding to the EVS pixel, obtain the interpolated gray - scale value of the EVS pixel; According to the interpolated gray - scale values of multiple EVS pixels and their test voltage values, construct a mapping relationship between the voltage value and the gray - scale value collected by the dual - mode fusion sensor; According to the mapping relationship and the real - time voltage value collected by the dual - mode fusion sensor, calculate the mapped gray - scale value of the EVS pixel.

2. The calibration method for a dual - mode fusion sensor according to claim 1, characterized in that, after obtaining the calibration image file collected by the dual - mode fusion sensor, the method further includes: For each APS pixel, determine the pixel gray - scale continuity type corresponding to the APS pixel according to the neighborhood gray - scale difference between the test gray - scale values of the APS pixel and its neighboring APS pixels; According to the preset gray - scale distribution rule and the test gray - scale values of each APS pixel, obtain the gray - scale level corresponding to each APS pixel; Take the gray - scale level corresponding to the APS pixel with the pixel gray - scale continuity type of gray - scale gradient pixel as the valid gray - scale level, and count the types of valid gray - scale levels based on the same valid gray - scale level; If the number of types of valid gray - scale levels is greater than the preset type threshold, then according to the test gray - scale values of the adjacent APS pixels corresponding to the EVS pixel, obtain the interpolated gray - scale value of the EVS pixel.

3. The calibration method for a dual - mode fusion sensor according to claim 2, characterized in that, the step of if the number of types of valid gray - scale levels is greater than the preset type threshold, then according to the test gray - scale values of the adjacent APS pixels corresponding to the EVS pixel, obtain the interpolated gray - scale value of the EVS pixel, includes: If the number of types of valid gray - scale levels is greater than the preset type threshold, then obtain the effective pixel position of the target APS pixel in the types of valid gray - scale levels; According to the preset approximation algorithm, determine the target EVS pixels adjacent to the effective pixel position; According to the test gray - scale values of the adjacent APS pixels corresponding to the target EVS pixel, obtain the interpolated gray - scale value of the target EVS pixel.

4. The calibration method for a dual - mode fusion sensor according to claim 2, characterized in that, the step of for each APS pixel, determine the pixel gray - scale continuity type corresponding to the APS pixel according to the neighborhood gray - scale difference between the test gray - scale values of the APS pixel and its neighboring APS pixels, includes: For each APS pixel, compare the neighborhood gray - scale difference with the preset gray - scale difference; Count the number of APS pixels corresponding to the neighborhood gray - scale difference less than the preset gray - scale difference within the preset area where the APS pixel is located; If the number of pixels is greater than or equal to the preset number of pixels, then determine that the APS pixel is a gray - scale gradient pixel. If the number of pixels is less than a preset number of pixels, determine that the APS pixel is a pixel with gray - level mutation.

5. The dual - mode fusion sensor calibration method according to claim 1, characterized in that, before obtaining the interpolated gray - level value of the EVS pixel according to the measured gray - level values of the adjacent APS pixels corresponding to the EVS pixel, the method further includes: obtaining the temperature change value when the dual - mode fusion sensor acquires the calibration image file; if the temperature change value is greater than a preset temperature change threshold, re - obtain the calibration image file acquired by the dual - mode fusion sensor; if the temperature change value is less than or equal to the preset temperature change threshold, obtain the interpolated gray - level value of the EVS pixel according to the measured gray - level values of the adjacent APS pixels corresponding to the EVS pixel.

6. The dual - mode fusion sensor calibration method according to claim 1, characterized in that, after obtaining the calibration image file acquired by the dual - mode fusion sensor, the method further includes: dividing the pixel area according to the arrangement of the EVS pixels, where the pixel area includes at least one EVS pixel and at least one APS pixel; for each of the pixel areas, establishing a correspondence relationship between the EVS pixels and the measured APS pixels in the pixel area.

7. The dual - mode fusion sensor calibration method according to claim 1, characterized in that, after calculating the mapped gray - level value of the EVS pixel according to the mapping relationship and the real - time voltage value acquired by the dual - mode fusion sensor, it further includes: obtaining the actual gray - level value of the APS pixel acquired by the dual - mode fusion sensor at the same moment when acquiring the real - time voltage value; combining and outputting a target gray - level image according to the real - time gray - level value of the APS pixel and the mapped gray - level value of the EVS pixel.

8. A dual - mode fusion sensor calibration device, characterized in that, the dual - mode fusion sensor calibration device includes: an acquisition module, configured to acquire a calibration image file acquired by a dual - mode fusion sensor, where the calibration image file includes the measured voltage value corresponding to the EVS pixel and the measured gray - level value corresponding to the APS pixel; an analog module, configured to obtain the interpolated gray - level value of the EVS pixel according to the measured gray - level values of the adjacent APS pixels corresponding to the EVS pixel; a construction module, configured to construct a mapping relationship between the voltage value and the gray - level value acquired by the dual - mode fusion sensor according to the interpolated gray - level values of multiple EVS pixels and their measured voltage values; a calculation module, configured to calculate the mapped gray - level value of the EVS pixel according to the mapping relationship and the real - time voltage value acquired by the dual - mode fusion sensor.

9. A dual - mode fusion sensor, characterized in that, the dual - mode fusion sensor includes: one or more processors; a memory; and one or more applications, where the one or more applications are stored in the memory and are configured to be executed by the processor to implement the steps of the dual - mode fusion sensor calibration method according to any one of claims 1 to 7.

10. A computer - readable storage medium, characterized in that, A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the dual-modal fusion sensor calibration method according to any one of claims 1 to 7.

Images