Gray mapping calibration method and device based on reference pixel, sensor and medium

Abstract

The invention provides a gray mapping calibration method and device based on a reference pixel, a sensor and a medium, and the method comprises the steps: obtaining a calibration image file collected by a dual-mode fusion sensor, the calibration image file comprises a test voltage value of an EVS pixel in a reference pixel region, and a test gray value of an APS pixel; the reference pixel region comprises a plurality of sub-pixel regions, each sub-pixel region comprises at least one APS pixel and at least one EVS pixel, the sizes of pixel openings of the APS pixels and the EVS pixels in the sub-pixel regions are the same, the sizes of corresponding pixel openings between the sub-pixel regions are different, and for each sub-pixel region, the interpolation gray value of the EVS pixel can be determined, and the interpolation gray value of the EVS pixel can be determined. Therefore, interpolation gray values of a plurality of EVS pixels and test voltage values of the EVS pixels are obtained, a mapping relation between the voltage values and the gray values is constructed, a real-time gray value can be obtained through calculation according to the mapping relation, and the method can be used for improving the quality of gray images collected by a bimodal fusion sensor.

Description

Technical Field

[0001] This application relates to the field of image sensors, and particularly to a gray-scale mapping calibration method, device, sensor, and medium based on reference pixels. Background Art

[0002] The dynamic event vision sensor (EVS) is a new type of sensor that mimics the human retina and generates event signals in response to pixel point pulses of brightness changes caused by motion. Therefore, it can capture the brightness changes of a scene at an extremely high frame rate, record events at specific time points and specific positions in an image, and form an event stream rather than a frame stream. The dynamic event vision sensor solves problems such as high power consumption, low frame rate, and poor dynamic range of APS pixel sensors (APS, Active Pixel Sensor). However, in imaging, it only retains motion edge information and loses rich detail information of objects. In related technologies, a dual-mode fusion sensor has been proposed to fuse the functions of APS pixels and EVS pixels. Currently, when using the dual-mode fusion sensor to output a gray-scale image, only the gray-scale image corresponding to APS pixels can be output, while the dynamic event vision sensor cannot collect gray-scale information and thus does not participate in the gray-scale image output, resulting in the lack of gray-scale values for EVS pixels corresponding to the dynamic event vision sensor and causing the poor quality of the gray-scale image collected by the dual-mode fusion sensor. Summary of the Invention

[0003] This application provides a gray-scale mapping calibration method, device, sensor, and medium based on reference pixels, aiming to solve the technical problem that the lack of gray-scale values for EVS pixels in the imaging pixel area of the dynamic event vision sensor leads to the poor quality of the gray-scale image collected by the dual-mode fusion sensor.

[0004] On the one hand, the method includes:

[0005] Obtain a calibration image file collected by the dual-mode fusion sensor, where the calibration image file includes the test voltage values of EVS pixels in the reference pixel area of the dual-mode fusion sensor and the test gray-scale values of APS pixels. Among them, the reference pixel area includes multiple sub-pixel areas, each sub-pixel area includes at least one APS pixel and at least one EVS pixel, the pixel opening sizes of APS pixels and EVS pixels in the sub-pixel area are the same, and the corresponding pixel opening sizes between sub-pixel areas are different;

[0006] For each of the sub-pixel areas, obtain the interpolated gray-scale value of the EVS pixel in the sub-pixel area according to the test gray-scale value of the APS pixel in the sub-pixel area;

[0007] Construct a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolation gray value of the EVS pixels in multiple sub-pixel regions and their test voltage values;

[0008] According to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the dual-mode fusion sensor, calculate the mapped gray value of the EVS pixels in the imaging pixel region, where the opening sizes of the pixels in the imaging pixel region are the same.

[0009] In some embodiments of the present application, before obtaining the calibration image file collected by the dual-mode fusion sensor, it further includes:

[0010] Irradiate the sensing area of the dual-mode fusion sensor with a preset light source;

[0011] Control the APS pixels and EVS pixels on the sensing area to simultaneously collect optical signals;

[0012] Convert the first optical signal collected by the APS pixels in the reference pixel region into the test gray value of the APS pixels, and convert the second optical signal collected by the EVS pixels in the reference pixel region into the test voltage value of the EVS pixels.

[0013] In some embodiments of the present application, the method further includes:

[0014] Execute the step of controlling the APS pixels and EVS pixels on the sensing area to simultaneously collect optical signals at intervals of a preset time.

[0015] In some embodiments of the present application, the step of constructing a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolation gray value of the EVS pixels in multiple sub-pixel regions and their test voltage values includes:

[0016] Input the interpolation gray values of the EVS pixels in multiple sub-pixel regions and their test voltage values into a preset fitting function, and calculate the fitting parameters in the preset fitting function;

[0017] Substitute the fitting parameters into the preset fitting function to obtain the mapping relationship.

[0018] In some embodiments of the present application, the step of obtaining the interpolation gray value of the EVS pixels in each sub-pixel region according to the test gray value of the APS pixels in the sub-pixel region further includes:

[0019] Obtain the distances between the APS pixels and the EVS pixels in the sub-pixel region;

[0020] Determine the weight value corresponding to each APS pixel according to the distance corresponding to each APS pixel;

[0021] According to the weight value corresponding to each APS pixel and its corresponding test gray value, perform weighted operation to obtain the interpolated gray value.

[0022] In some embodiments of the present application, after calculating the mapped gray value of the EVS pixels in the imaging pixel region of the dual-mode fusion sensor according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region, it further includes:

[0023] Obtain the actual gray value collected by the APS pixels in the imaging pixel region at the same moment when the real-time voltage value is collected;

[0024] Generate and output a target gray image according to the real-time gray value of the APS pixels and the mapped gray value of the EVS pixels.

[0025] On the other hand, the present application provides a gray mapping calibration device based on reference pixels, and the gray mapping calibration device based on reference pixels includes:

[0026] An acquisition module, configured to acquire a calibration image file collected by a dual-mode fusion sensor, where the calibration image file includes the test voltage value of the EVS pixels in the reference pixel region of the dual-mode fusion sensor and the test gray value of the APS pixels. Among them, the reference pixel region includes multiple sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. The pixel aperture sizes of the APS pixels and EVS pixels in the sub-pixel region are the same, and the pixel aperture sizes corresponding to the sub-pixel regions are different;

[0027] An interpolation module, configured to, for each sub-pixel region, obtain the interpolated gray value of the EVS pixels in the sub-pixel region according to the test gray value of the APS pixels in the sub-pixel region;

[0028] A construction module, configured to construct a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolated gray values of the EVS pixels in multiple sub-pixel regions and their test voltage values;

[0029] A calculation module, configured to calculate the mapped gray value of the EVS pixels in the imaging pixel region according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the dual-mode fusion sensor, where the pixel aperture sizes of the pixels in the imaging pixel region are the same.

[0030] On the other hand, the present application also provides a dual-modal fusion sensor, which includes:

[0031] One or more processors;

[0032] A memory; and

[0033] One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the processor to implement the steps of the gray-scale mapping calibration method based on reference pixels.

[0034] On the other hand, the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the gray-scale mapping calibration method based on reference pixels.

[0035] The technical solution of the embodiment of the present application includes: obtaining a calibration image file collected by the dual-modal fusion sensor, where the calibration image file includes the test voltage value of the EVS pixels in the reference pixel area of the dual-modal fusion sensor and the test gray-scale value of the APS pixels. Among them, the reference pixel area includes multiple sub-pixel areas, each sub-pixel area includes at least one APS pixel and at least one EVS pixel, the pixel aperture sizes of the APS pixels and EVS pixels in the sub-pixel area are the same, and the corresponding pixel aperture sizes between the sub-pixel areas are different; for each of the sub-pixel areas, according to the test gray-scale value of the APS pixels in the sub-pixel area, obtain the interpolated gray-scale value of the EVS pixels in the sub-pixel area; according to the interpolated gray-scale values of the EVS pixels in the multiple sub-pixel areas and their test voltage values, construct a mapping relationship between the voltage value and the gray-scale value collected by the dual-modal fusion sensor; according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel area of the dual-modal fusion sensor, calculate the mapped gray-scale value of the EVS pixels in the imaging pixel area, where the pixel aperture sizes of the pixels in the imaging pixel area are consistent.

[0036] In the technical solution of the embodiment of the present application, a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed by using a calibration file image file including the test gray value of the APS pixel and the test voltage value of the EVS pixel in the reference pixel region. Among them, the reference pixel region includes a plurality of sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, their pixel aperture sizes are the same. The interpolation gray value of the EVS pixel can be obtained by interpolating the test gray value of the APS pixel in the same sub-pixel region. In this way, each EVS pixel in each sub-pixel region corresponds to an interpolation gray value and a test voltage value. Since the pixel aperture sizes between the sub-pixel regions are different, the test gray values of the APS pixels and the test voltage values of the EVS pixels are also different between the sub-pixel regions, so that the interpolation gray values and the test voltage values corresponding to the EVS pixels between the sub-pixel regions are different. In this way, multiple groups of sample data of different test voltage values and interpolation gray values can be obtained, and the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor can be accurately constructed. In the application stage, each pixel in the imaging pixel region of the dual-mode fusion sensor is used to collect an image. The imaging pixel region is different from the calibration image region. The aperture sizes of each pixel in the imaging pixel region are the same, and each pixel value has consistency for collecting the gray value and can be used for imaging. The real-time voltage value collected by the EVS pixel in the imaging pixel region can be calculated and converted through the mapping relationship to obtain the real-time gray value of the EVS pixel in the imaging pixel region. This real-time gray value includes dynamic event factors. The real-time gray value can be used to fill the gray value vacancy of the EVS pixel in the gray image collected by the APS pixel corresponding in the imaging pixel region of the dual-mode fusion sensor, so that a gray image with higher resolution, higher frame rate and wider dynamic range can be combined and generated, improving the quality of the gray image output by the dual-mode fusion sensor. Description of the Drawings

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0038] Figure 1 It is a schematic flowchart of an embodiment of the gray mapping calibration method based on reference pixels provided in the embodiment of the present application;

[0039] Figure 2 It is a schematic structural diagram of the sensing area of the dual-mode fusion sensor involved in the embodiment of the present application;

[0040] Figure 3 It is a schematic structural diagram of the sub-pixel region of the dual-modal fusion sensor involved in the embodiment of the present application;

[0041] Figure 4 It is a schematic flowchart of an embodiment for collecting a calibration image file in the gray-scale mapping calibration method based on reference pixels provided in the embodiment of the present application;

[0042] Figure 5 It is a schematic structural diagram of the calibration module of the dual-modal fusion sensor involved in the embodiment of the present application;

[0043] Figure 6 It is a schematic flowchart of an embodiment for constructing the mapping relationship between the voltage value and the gray-scale value collected by the dual-modal fusion sensor in the gray-scale mapping calibration method based on reference pixels provided in the embodiment of the present application;

[0044] Figure 7 It is a schematic structural diagram of an embodiment of the gray-scale mapping calibration device based on reference pixels provided in the embodiment of the present application;

[0045] Figure 8 It is a schematic structural diagram of an embodiment of the dual-modal fusion sensor provided in the embodiment of the present application. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the scope of the present invention.

[0047] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0048] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or demonstration". Any embodiment described as "exemplary" in this application is not necessarily to be construed as more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the present invention. In the following description, details are set forth for purposes of explanation. It should be understood that those of ordinary skill in the art can recognize that the present invention can be implemented without these specific details. In other instances, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in this application.

[0049] In the related art, a dual-mode fusion sensor has been proposed to fuse the functions of APS pixels and EVS pixels. However, when the dual-mode fusion sensor outputs a grayscale image currently, only the grayscale image corresponding to the APS pixels can be output, and the information collected by the EVS pixels does not participate in the output of the grayscale image, resulting in poor quality of the grayscale image output by the dual-mode fusion sensor.

[0050] Based on this, in the embodiments of this application, a grayscale mapping calibration method, device, sensor, and medium based on reference pixels are proposed, and the following will be described in detail respectively.

[0051] The grayscale mapping calibration method based on reference pixels in the embodiments of the present invention is applied to a grayscale mapping calibration device based on reference pixels. The grayscale mapping calibration device based on reference pixels can be set in a dual-mode fusion sensor or in other terminal devices, such as a computer, a mobile phone, etc. The dual-mode fusion sensor is provided with one or more processors, a memory, and one or more application programs, and one or more of the application programs are stored in the memory and configured to be executed by the processor to implement the grayscale mapping calibration method based on reference pixels.

[0052] The dual-mode fusion sensor in the embodiments of the present invention is mainly used for:

[0053] Obtaining a calibration image file collected by the dual-mode fusion sensor, where the calibration image file includes the test voltage values of the EVS pixels in the reference pixel region and the test grayscale values of the APS pixels. Among them, the reference pixel region includes a plurality of sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. The pixel aperture sizes of the APS pixels and EVS pixels in the sub-pixel region are the same, and the pixel aperture sizes corresponding to different sub-pixel regions are different;

[0054] For each of the sub-pixel regions, an interpolated gray value of the EVS pixels in the sub-pixel region is obtained according to the measured gray value of the APS pixels in the sub-pixel region;

[0055] According to the interpolated gray values of the EVS pixels in multiple sub-pixel regions and their measured voltage values, a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed;

[0056] According to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the dual-mode fusion sensor, the mapped gray value of the EVS pixels in the imaging pixel region is calculated, wherein the aperture sizes of the pixels in the imaging pixel region are the same.

[0057] In the application stage, each pixel in the imaging pixel region of the dual-mode fusion sensor is used to collect images. The imaging pixel region is different from the calibration image region. The aperture sizes of the pixels in the imaging pixel region are the same, and each pixel value has consistency for the collected gray value and can be used for imaging. The real-time voltage value collected by the EVS pixels in the imaging pixel region can be calculated and converted through the mapping relationship to obtain the real-time gray value of the EVS pixels in the imaging pixel region. This real-time gray value includes dynamic event factors. The real-time gray value can be used to fill the gray value gap of the EVS pixels in the gray image collected by the APS pixels corresponding to the imaging pixel region of the dual-mode fusion sensor. Thus, a gray image with higher resolution, higher frame rate, and wider dynamic range can be generated by combination, improving the quality of the gray image output by the dual-mode fusion sensor.

[0058] Hereinafter, the content claimed in the claims of the present invention will be explained and illustrated through specific exemplary solutions, so that those skilled in the art can better understand the protection scope of the claims of the present invention. It can be understood that the following exemplary solutions do not limit the protection scope of the present invention and are only used to explain the present invention.

[0059] Exemplarily, as Figure 1 shown, Figure 1 is a schematic flowchart of an embodiment of the gray mapping calibration method based on reference pixels in the embodiment of the present application. The gray mapping calibration method based on reference pixels includes the following steps 101 to 104:

[0060] 101. Obtain the calibration image file collected by the dual-modal fusion sensor. The calibration image file includes the test voltage value of the EVS pixels within the reference pixel region of the dual-modal fusion sensor and the test gray value of the APS pixels. Among them, the reference pixel region includes multiple sub-pixel regions, each sub-pixel region includes at least one APS pixel and at least one EVS pixel, the pixel aperture sizes of the APS pixels and EVS pixels within the sub-pixel region are the same, and the corresponding pixel aperture sizes between the sub-pixel regions are different;

[0061] In this embodiment, the gray mapping calibration method based on the reference pixel can be applied to the gray mapping calibration device based on the reference pixel. The gray mapping calibration device based on the reference pixel can be set in the dual-modal fusion sensor or can be a terminal device connected to the dual-modal fusion sensor.

[0062] The dual-modal fusion sensor can collect a calibration image file for a target scene. The calibration image file can be a file in RAW format. RAW refers to the binary format in which the dual-modal fusion sensor converts the captured light source signal into a data signal.

[0063] The dual-modal fusion sensor includes multiple sensing units. The calibration image file includes the information collected by the sensing units corresponding to the dual-mode fusion sensor. Each sensing unit corresponds to a pixel. According to the different data types collected by the sensing units, the pixels corresponding to the sensing units are also different. The sensing units in the dual-modal fusion sensor can respectively correspond to the APS (EVS, Event-based Vision Sensor dynamic event vision sensing) data mode and the EVS (APS, Active Pixel Sensor active pixel sensing) data mode. The pixel types can be divided into APS pixels and EVS pixels, where the APS data mode corresponds to APS pixels and the EVS data mode corresponds to EVS pixels. Compared with the dual-modal vision sensor composed of multiple discrete sensor modules, the device volume of the sensor is effectively compressed, which is more conducive to the miniaturization of the overall architecture, and eliminates the inspection and image fusion matching problems between different sensors.

[0064] In actual use, when the dual-modal fusion sensor collects the calibration image file, the EVS pixels and APS pixels can be synchronously exposed. There is a photodiode configured in the EVS pixels. The photodiode is integrated with a capacitor that accumulates charge. It generates a photocurrent in response to the incident light intensity, and then generates the voltage value of the corresponding EVS pixel according to the photocurrent.

[0065] Among them, the APS pixels in the dual-modal fusion sensor correspond to the obtained grayscale values, and the EVS pixels correspond to the obtained voltage values. During the calibration phase, it serves as the test voltage value in this embodiment; the grayscale value corresponding to the APS pixels in the dual-modal fusion sensor, during the calibration phase, this grayscale value serves as the test grayscale value in this embodiment. The voltage value of the EVS pixels and the grayscale value corresponding to the adjacent APS pixels together form the above-mentioned calibration image file.

[0066] In this embodiment, please refer to Figure 2 , the dual-modal fusion sensing area is composed of multiple EVS pixels and multiple APS pixels, where A represents APS pixels and E represents EVS pixels, and its sensing area is composed of a reference pixel area and an imaging pixel area. The reference pixel area can be flexibly defined according to the actual application scenario and can be set at the edge position of the sensing area, thereby reducing the impact on the sensing effect of the dual-modal fusion sensor and improving the sensing performance of the dual-modal fusion sensor. Multiple sub-sensing areas can be further divided in the reference pixel area, and each sub-sensing area includes at least one APS pixel and at least one EVS pixel. Within the same sub-pixel area, the pixel opening sizes between each pixel unit are the same, that is, the pixel opening sizes of the APS pixels and the EVS pixels are the same. Since the pixel opening sizes are the same, the light intensity of the incident light received by them is also the same. For the same sub-pixel area, the interpolation grayscale value of the EVS pixels can be obtained more accurately based on the test grayscale value of the APS pixels. Between different sub-pixel areas, the opening sizes of the pixel units are different, that is, the pixel opening sizes of the APS pixels and the EVS pixels are different. Setting different pixel opening sizes can be achieved by setting different opening ratios between different sub-pixel areas, such as using opaque metal tungsten to block part of the pixel area, or different bias voltages can also be set between different sub-pixel areas to change the photosensitivity of the reference pixels, and the reference pixel area is composed of sub-pixel areas with multiple photosensitivities prepared by processing. Since the pixel opening sizes are different, the light intensity of the incident light received by them is also different, and the test grayscale values of the APS pixels and the test voltage values of the EVS pixels between each sub-pixel area are inconsistent. In the imaging pixel area, the pixel opening sizes are the same, so that the light intensity reception rate of the incident light received by them is also the same, and the pixels in the imaging pixel area can be used for imaging.

[0067] It should be understood that Figure 2 the array size in

[0068] is only for illustrative purposes and should not be understood as the only limitation on the array size of the overall pixel array in actual applications. Moreover, the layout and quantity of the two types of pixels in the overall pixel array can be determined according to the actual application scenario, and this embodiment does not make a unique limitation.102. For each of the sub-pixel regions, an interpolated gray value of the EVS pixels in the sub-pixel region is obtained according to the measured gray value of the APS pixels in the sub-pixel region.

[0069] In this embodiment, the calibration image file collected in the calibration stage includes the measured voltage value corresponding to the EVS pixels in the reference pixel region and the measured gray value corresponding to the APS pixels. If we want to calculate the gray value of the EVS pixels using the information collected by the EVS pixels, we need to first obtain the mapping relationship between the voltage value collected by the dual-mode fusion sensor and its corresponding gray value, that is, the mapping relationship between the voltage value and the gray value collected by the EVS pixels. Then, in the calibration stage, we need to obtain the measured voltage value and gray value corresponding to the EVS pixels. In the reference pixel region, there are multiple sub-pixel regions, and the pixel aperture sizes of the APS pixels and EVS pixels in each sub-pixel region are the same. Therefore, for the gray value of the EVS pixels in each sub-pixel region, it can be determined according to the measured gray value of the neighboring APS pixels in the same sub-pixel region. By performing interpolation calculations based on the measured gray value of the APS pixels in the sub-pixel region, the interpolated gray value of the EVS pixels can be obtained.

[0070] Please refer to Figure 3 , in the reference pixel region 100 corresponding to the calibration image file, it includes a sub-pixel region 110. For the sub-pixel 110, the sub-pixel 110 includes 2 * 2 pixels, where there are 3 adjacent APS pixels for the EVS pixel. According to the measured gray values of these 3 adjacent APS pixels, the difference gray value of the EVS pixel can be obtained. Since each APS pixel and the EVS pixel are in the same sub-pixel region and their pixel aperture sizes are the same, it can improve the authenticity of the interpolated gray value obtained by fusing the gray values of the APS pixels and improve the accuracy of the constructed preset mapping relationship.

[0071] Optionally, step 102 further includes:

[0072] Obtain the distances between each APS pixel and the EVS pixel in the sub-pixel region;

[0073] Determine the weight value corresponding to each APS pixel according to the distances corresponding to each APS pixel;

[0074] The interpolated gray value is obtained through weighted operations based on the weight values corresponding to each APS pixel and their corresponding measured gray values.

[0075] In this embodiment, the method for calculating the interpolated gray value of EVS in the sub-pixel region may be to perform weighted average calculation on the measured gray values of each APS pixel in the same sub-pixel region, and use the result of the weighted average calculation as the interpolated gray value of the EVS pixel. Further, in this embodiment, the weight values corresponding to different APS pixels may also be assigned based on the relative distance between the APS pixel and the EVS pixel. First, it is necessary to obtain the relative distance between the EVS pixel and each APS pixel in the same sub-pixel region, and the weight value corresponding to the measured gray value of each APS pixel may be determined according to the ratio between the relative distances corresponding to each APS pixel and the EVS pixel. Among them, the larger the relative distance, the smaller the corresponding weight value. For example, a larger weight value may be assigned to the measured gray value of the APS pixel that is closer to the position of the EVS pixel, while a larger weight value may be assigned to the measured gray value of the APS pixel that is farther from the position of the EVS pixel.

[0076] 103. Construct a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolated gray values and their measured voltage values of the EVS pixels in the multiple sub-pixel regions;

[0077] In this embodiment, the measured voltage value corresponding to the EVS pixel may be obtained based on the dynamic event vision sensor. By using the measured gray values of adjacent APS pixels in the same sub-pixel region, the corresponding interpolated gray value is obtained. For the same sub-pixel region, at least one sample data composed of the measured voltage value corresponding to the EVS pixel and its interpolated gray value can be obtained. For different sub-pixel regions, due to the different pixel aperture sizes of the APS pixels and the EVS pixels, the light intensity of the incident light received by them is also different. The measured gray values of the APS pixels and the measured voltage values of the EVS pixels between different sub-pixel regions are inconsistent. And because the interpolated gray value is calculated based on the measured gray value of the APS pixel in the same sub-pixel region, the interpolated gray values corresponding to the EVS pixels in different sub-pixel regions will also be different. In this way, there will be differences in the sample data corresponding to the EVS pixels in each sub-pixel region. Therefore, a more accurate mapping relationship between the voltage value and the gray value collected by the dynamic event vision sensor can be constructed according to multiple sample data. The mapping relationship may be in the form of a mapping table, a mapping function, etc., and is selected according to specific requirements. The mapping table composed of the measured voltage values corresponding to the EVS pixels in multiple sub-pixel regions and their interpolated gray values may be used as the mapping relationship, or the function constructed according to the measured voltage values corresponding to the EVS pixels in multiple sub-pixel regions and their interpolated gray values may be used as the mapping relationship.

[0078] 104. Calculate the mapped gray value of the EVS pixels in the imaging pixel region according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the bimodal fusion sensor, wherein the aperture sizes of the pixels in the imaging pixel region are the same.

[0079] In this embodiment, in the calibration stage, the mapping relationship between the voltage value and the gray value collected by the bimodal fusion sensor is constructed through the reference pixel region. In the application stage, the bimodal fusion sensor will collect the gray image through the imaging pixel region. The aperture sizes of the pixels in the imaging pixel region are the same. In the application stage, the EVS pixels in the imaging pixel region of the bimodal fusion sensor will collect the real-time voltage value, and this real-time voltage value can map the corresponding mapped gray value of the EVS pixel through the mapping relationship obtained in the calibration stage. This mapped gray value is determined by using the real-time voltage value collected by the EVS pixel that contains the dynamic event factor. This mapped gray value can be used as the gray value collected by the bimodal fusion sensor based on the dynamic event. Based on this mapped gray value, a gray image with higher quality can be obtained for the bimodal fusion sensor.

[0080] Optionally, after step 104, it further includes:

[0081] Obtain the actual gray value collected by the APS pixels in the imaging pixel region at the same moment when the real-time voltage value is collected;

[0082] Generate and output a target gray image according to the real-time gray value of the APS pixels and the mapped gray value of the EVS pixels.

[0083] In this embodiment, the real-time voltage value corresponding to the EVS pixel collected by the dual-modal fusion sensor can be mapped to the mapped gray value corresponding to the EVS pixel by combining the mapping relationship obtained in the calibration stage. In addition, the APS pixels within the imaging pixel region of the dual-modal fusion sensor can be synchronously exposed with the EVS pixels within the imaging pixel region, that is, when the dual-modal fusion sensor collects the real-time voltage value through the EVS pixels within the imaging pixel region, it can collect the actual gray value through the APS pixels within the imaging pixel region at the same time. In this way, the real-time image file collected by the dual-modal sensor during imaging includes the real-time voltage value corresponding to the EVS pixels within the imaging pixel region and the real-time gray value corresponding to the APS pixels within the imaging pixel region. According to the real-time voltage value corresponding to the EVS pixels within the imaging pixel region and the above mapping relationship, the mapped gray value of the EVS pixels within the imaging pixel region can be determined. Since the pixel positions of the EVS pixels and the APS pixels within the imaging pixel region are different, based on the pixel positions of the EVS pixels and the APS pixels within the imaging pixel region, the mapped gray value of the EVS pixels can be combined with the real-time gray value of the APS pixels collected at the same time to generate a target gray image. The pixel positions on the target gray image all correspond to the APS pixels or the EVS pixels. The generation of the target gray image uses both the gray information of the APS pixels and the dynamic voltage information of the EVS pixels, making the details of the target gray image richer and of higher quality.

[0084] Optionally, due to the pixel opening setting of the reference pixel region, after the calibration is completed, methods such as occlusion can be used to prevent imaging through the pixel units within the reference pixel region.

[0085] In the technical solution disclosed in this embodiment, a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed through a calibration file image file including the test gray value of the APS pixel and the test voltage value of the EVS pixel in the reference pixel region. Among them, the reference pixel region includes multiple sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, their pixel aperture sizes are the same. The interpolation gray value of the EVS pixel can be obtained by interpolating the test gray value of the APS pixel in the same sub-pixel region. In this way, the EVS pixels in each sub-pixel region correspond to an interpolation gray value and a test voltage value. Since the pixel aperture sizes between the sub-pixel regions are different, the test gray values of the APS pixels and the test voltage values of the EVS pixels are also different between the sub-pixel regions, so that the interpolation gray values and the test voltage values corresponding to the EVS pixels between the sub-pixel regions are different. In this way, multiple groups of sample data of different test voltage values and interpolation gray values can be obtained, and the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor can be accurately constructed. In the application stage, each pixel in the imaging pixel region of the dual-mode fusion sensor is used to collect images. The imaging pixel region is different from the calibration image region. The aperture sizes of the pixels in the imaging pixel region are the same, and each pixel value has consistency for collecting gray values and can be used for imaging. The real-time voltage value collected by the EVS pixel in the imaging pixel region can be calculated and converted through the mapping relationship to obtain the real-time gray value of the EVS pixel in the imaging pixel region. This real-time gray value includes dynamic event factors, and this real-time gray value can be used to fill the gray value vacancy of the EVS pixel in the gray image collected by the APS pixel in the imaging pixel region of the dual-mode fusion sensor, so that a gray image with higher resolution, higher frame rate, and wider dynamic range can be combined and generated, improving the quality of the gray image output by the dual-mode fusion sensor.

[0086] It can be understood that since the test voltage value and the test gray value of the reference pixel region are used to construct the mapping relationship in the calibration stage of this embodiment, the mapping relationship can be calibrated through a single image, with a high calibration success rate and a faster calibration speed. In the collected calibration image file, the number of gray levels is related to the pixel aperture of the reference pixel region, and the pixel aperture of the reference pixel region can be arbitrarily set according to requirements, with higher flexibility and improved user experience. And when constructing the mapping relationship and determining the mapping gray value, there is no need to program complex image processing algorithms, making the calibration cost lower, improving the efficiency of constructing the mapping relationship in the calibration stage, and being able to more quickly improve the quality of the gray image output by the dual-mode fusion sensor.

[0087] As Figure 4 shown, Figure 4It is a schematic flowchart of an embodiment for acquiring a calibration image file in the gray mapping calibration method based on reference pixels provided in the embodiments of the present application, including steps 401 to 403:

[0088] 401. Irradiate the reference pixel area with a preset light source;

[0089] In this embodiment, when calibrating the dual - mode fusion sensor, the calibration module may include, in addition to the sensing area of the dual - mode fusion sensor, a preset light source, and the preset light source may be an internal light source provided in the dual - mode fusion sensor. In the calibration stage, the preset light source can irradiate an optical signal to the sensing area of the dual - mode fusion sensor, that is, irradiate bright light to the reference pixel area, and the intensity of the bright light irradiated by the preset light source is generally the same.

[0090] Exemplarily, referring to Figure 5 , the calibration module includes an internal light source 501, a sensing area 502, and a lens 503. In the calibration stage, the internal light source irradiates an optical signal to the sensing area, and in the application stage, the reflected light of the target to be captured is captured by the sensing area through the lens to obtain a corresponding image.

[0091] 402. Control the APS pixels and EVS pixels in the reference pixel area to simultaneously collect optical signals;

[0092] In this embodiment, after the preset light source irradiates an optical signal to the sensing area of the dual - mode fusion sensor, control the APS pixels and EVS pixels on the reference pixel area to simultaneously collect the optical signal emitted by the preset light source when the preset light source irradiates the optical signal.

[0093] 403. Convert the first optical signal collected by the APS pixels in the reference pixel area into a test gray value of the APS pixels, and convert the second optical signal collected by the EVS pixels in the reference pixel area into a test voltage value of the EVS pixels.

[0094] In this embodiment, the optical signals collected by the APS pixels and EVS pixels in the reference pixel area can be further processed. The first optical signal collected by the APS pixels in the reference pixel area will be converted by the APS pixels into a test gray value, and the second optical signal collected by the EVS pixels in the reference pixel area will be converted by the EVS pixels into a test voltage value, and then a calibration image file can be obtained.

[0095] It should be noted that the calibration image file includes the measured voltage values of EVS pixels in the reference pixel region and the measured voltage values of APS pixels, and may also include the voltage values of EVS pixels and the gray values of APS pixels in the imaging pixel region. When constructing the mapping relationship between the voltage values and gray values collected by the dual-mode fusion sensor, the measured voltage values of EVS pixels in the reference pixel region and the measured voltage values of APS pixels can be obtained from the calibration image file.

[0096] When the reference pixel region is irradiated by the preset light source, in the same sub-pixel region, the pixel aperture sizes are the same, so that the light intensities received by the pixel units in the same sub-pixel region are the same, and thus the output gray values are also relatively consistent. Between different pixel regions, the pixel aperture sizes are the same, so that the light intensities received by the pixel units in the same sub-pixel region are inconsistent, and thus the output gray values are also inconsistent.

[0097] In the technical solution disclosed in this embodiment, by irradiating the sensing region of the dual-mode fusion sensor with the preset light source, the measured gray values of APS pixels and the measured voltage values of EVS pixels in the reference pixel region are obtained, and then the mapping relationship between the voltage values and gray values collected by the dual-mode fusion sensor can be constructed. In this way, by using the fixed preset light source to irradiate the sensing region to obtain the interpolated gray values and measured voltage values of multiple EVS pixels in the reference pixel region, and by using the internal light source to irradiate the sensor pixel array, the complexity of calibration can be reduced, and the interpolated gray values and measured voltage values of multiple EVS pixels in the reference pixel region can be obtained without relying on external conditions.

[0098] Further, the method further includes:

[0099] Every preset time interval, execute the step of controlling the APS pixels and EVS pixels on the sensing region to simultaneously collect optical signals.

[0100] In this embodiment, when the dual-mode fusion sensor is working, the preset light source and the reference pixel region work in real time. Every preset time interval, for example, every 1 s, or other time intervals, the APS pixels and EVS pixels in the reference pixel region are re-controlled to simultaneously collect the optical signals emitted by the preset light source, and a new calibration image file is obtained to recalculate the mapping parameters once. The mapping parameters updated in a short time can compensate for the influence of temperature changes on the mapping accuracy, thereby improving the accuracy of the mapping relationship.

[0101] It should be noted that the reference pixels can be recalibrated for gray mapping every fixed time (such as 1 s). This frequent recalibration can naturally solve the change in gray mapping caused by temperature changes. That is, the gray mapping relationship is self-calibrated in real time, which can be free from temperature interference and help the imaging pixels obtain an accurate voltage-gray conversion.

[0102] As shown Figure 6 in Figure 6 FIG. X, it is a schematic flowchart of an embodiment for constructing a mapping relationship between voltage values and gray values collected by a dual - mode fusion sensor provided in an embodiment of the present application, including steps 601 - 602:

[0103] 601. Input the interpolated gray values and their test voltage values of the EVS pixels in multiple sub - pixel regions into a preset fitting function, and calculate the fitting parameters in the preset fitting function.

[0104] 602. Substitute the fitting parameters into the preset fitting function to obtain the mapping relationship.

[0105] In this embodiment, the mapping relationship can be a functional relationship between the voltage values and gray values collected by the dual - mode fusion sensor. The determination of this type of mapping relationship can be achieved through a preset fitting function. First, set a preset fitting function between the voltage values and gray values collected by the dual - mode fusion sensor. The fitting function can be in the form of exponential, linear, logarithmic, polynomial, etc. This embodiment does not make a unique limitation on this. Input the sample data composed of the test voltage values corresponding to the EVS pixels in multiple sub - pixel regions within the reference pixel region and their interpolated gray values into the preset fitting function, calculate the fitting parameters in the preset fitting function, and substitute the fitting parameters into the preset fitting function to obtain the globally unified mapping relationship of the dual - mode fusion sensor.

[0106] Specifically, the test voltage values corresponding to the EVS pixels in multiple sub - pixel regions and their interpolated gray values can form multiple sample data. The form of each sample data is (X: test voltage value, Y: interpolated gray value). Input these sample data into the preset fitting function to calculate the fitting parameters in the preset fitting function. The preset fitting function can be expressed as: I = f(V out ) out , where V

[0107] represents the test voltage value corresponding to the EVS pixel, and I represents the interpolated gray value corresponding to the EVS pixel. It should be understood that the mapping relationship of f(·) in the above - mentioned fitting function can be in the form of exponential, linear, logarithmic, polynomial, etc. This embodiment does not make a unique limitation on this.

[0108] In one implementation, if the above - mentioned fitting function is implemented using a linear mapping relationship, the fitting function can be expressed as: out I = a * V

[0109] + b out , where V

[0110] In another implementation, if the above fitting function is implemented using a logarithmic mapping relationship, the fitting function can be expressed as:

[0111] I = t * I 0 * log[(V - V out ) / k]

[0112] where V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, a and b represent fitting parameters, t represents the exposure time of the APS pixel, and I 0 , V, and k represent fitting parameters.

[0113] In yet another implementation, if the above fitting function is implemented using a polynomial mapping relationship, the fitting function can be expressed as:

[0114] I = a + b 1 V out + b 2 V out + b 3 V out

[0115] where V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, and a and b represent fitting parameters.

[0116] In still another implementation, if the above fitting function is implemented using an exponential mapping relationship, the fitting function can be expressed as:

[0117]

[0118] where V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, t represents the exposure time of the APS pixel, and I 0 , V, and k represent fitting parameters.

[0119] Furthermore, the value of m above is the natural constant e, that is, the above fitting function can be further expressed as:

[0120]

[0121] Of course, in practical applications, the values of the above constants such as i, j, and m can be flexibly set according to the application scenario, and this embodiment does not make a unique limitation on this. In this embodiment, a preset fitting function is pre-customized, and the form of this fitting function is not limited to the above form provided in this embodiment, and can also be a Fourier expansion, etc.

[0122] In this embodiment, the test voltage value corresponding to the EVS pixel and its interpolated gray value are used as known quantities and input into the above-mentioned fitting function based on the exponential mapping relationship, etc., to solve the unknown quantities a, b, or I in the fitting function 0 , V, and k, that is, the fitting parameters are obtained. Finally, the obtained solutions are substituted into the fitting function, and a complete fitting function I = f(V out ) is obtained. This fitting function can be used as the mapping relationship between the voltage value and the gray value collected by the dynamic event vision sensor or the dual-modal fusion sensor

[0123] In the technical solution disclosed in this embodiment, multiple sample data of the test voltage values and interpolated gray values corresponding to the EVS pixels in different sub-pixel regions within the reference pixel region are used as known quantities and input into the above-mentioned preset fitting function such as the exponential mapping relationship to solve the fitting parameters that are unknown quantities in the preset fitting function. Finally, the obtained solutions are substituted into the fitting function, and a fitting curve is obtained as the mapping relationship between the voltage value and the gray value collected by the dual-modal fusion sensor. In this way, for the real-time voltage value collected by the EVS pixel within the imaging pixel region of the dual-modal fusion sensor, a corresponding and accurate mapped gray value can be found. This mapped gray value includes dynamic event factors, and a higher-quality gray image can be obtained using this mapped gray value

[0124] To better implement the gray mapping calibration method based on the reference pixel in the embodiment of the present application, on the basis of the gray mapping calibration method based on the reference pixel, an apparatus for gray mapping calibration based on the reference pixel is also provided in the embodiment of the present application, as shown in Figure 7 shown Figure 7 is a schematic structural diagram of an embodiment of the apparatus for gray mapping calibration based on the reference pixel. The apparatus for gray mapping calibration based on the reference pixel includes the following modules 701 to 704

[0125] An acquisition module 701, configured to acquire a calibration image file collected by the dual-modal fusion sensor. The calibration image file includes the test voltage value of the EVS pixel within the reference pixel region of the dual-modal fusion sensor and the test gray value of the APS pixel. Among them, the reference pixel region includes multiple sub-pixel regions, each sub-pixel region includes at least one APS pixel and at least one EVS pixel, the pixel opening sizes of the APS pixel and the EVS pixel within the sub-pixel region are the same, and the pixel opening sizes corresponding to different sub-pixel regions are different

[0126] An interpolation module 702, configured to, for each sub-pixel region, obtain the interpolated gray value of the EVS pixel within the sub-pixel region according to the test gray value of the APS pixel within the sub-pixel region

[0127] A construction module 703, configured to construct a mapping relationship between the voltage value and the grayscale value collected by the dual-modal fusion sensor according to the interpolated grayscale values of the EVS pixels in multiple said sub-pixel regions and their test voltage values;

[0128] A calculation module 704, configured to calculate the mapped grayscale values of the EVS pixels in the imaging pixel region of the dual-modal fusion sensor according to the mapping relationship and the real-time voltage values collected by the EVS pixels in the imaging pixel region of the dual-modal fusion sensor, wherein the aperture sizes of the pixels in the imaging pixel region are the same.

[0129] In some embodiments of the present application, the acquisition module 701 is further configured to irradiate the reference pixel region with a preset light source;

[0130] Control the APS pixels and EVS pixels in the reference pixel region to simultaneously collect optical signals;

[0131] Convert the first optical signal collected by the APS pixels in the reference pixel region into the test grayscale values of the APS pixels, and convert the second optical signal collected by the EVS pixels in the reference pixel region into the test voltage values of the EVS pixels.

[0132] In some embodiments of the present application, the acquisition module 701 is further configured to execute the step of controlling the APS pixels and EVS pixels in the reference pixel region to simultaneously collect optical signals at intervals of a preset time.

[0133] In some embodiments of the present application, the construction module 703 is further configured to input the interpolated grayscale values of the EVS pixels in multiple said sub-pixel regions and their test voltage values into a preset fitting function to calculate the fitting parameters in the preset fitting function;

[0134] Substitute the fitting parameters into the preset fitting function to obtain the mapping relationship.

[0135] In some embodiments of the present application, the interpolation module 702 is further configured to obtain the distances between the APS pixels and the EVS pixels in the sub-pixel region;

[0136] Determine the weight values corresponding to the APS pixels according to the distances corresponding to the APS pixels;

[0137] Perform weighted operation according to the weight values corresponding to the APS pixels and their corresponding test grayscale values to obtain the interpolated grayscale values.

[0138] In some embodiments of the present application, the gray-scale mapping calibration device based on reference pixels further includes a generation module 705, configured to obtain the actual gray-scale value collected by the APS pixels in the imaging pixel region at the same moment when the real-time voltage value is collected;

[0139] Based on the real-time gray-scale value of the APS pixels and the mapped gray-scale value of the EVS pixels, a target gray-scale image is combined, generated, and output.

[0140] In the embodiments of the present application, a mapping relationship between the voltage value and the gray-scale value collected by the dual-mode fusion sensor is constructed through a calibration file image file including the test gray-scale value of the APS pixels and the test voltage value of the EVS pixels in the reference pixel region. Among them, the reference pixel region includes multiple sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, their pixel aperture sizes are the same. The interpolated gray-scale value of the EVS pixel can be obtained by interpolating the test gray-scale value of the APS pixel in the same sub-pixel region. In this way, each EVS pixel in each sub-pixel region corresponds to an interpolated gray-scale value and a test voltage value. Since the pixel aperture sizes between the sub-pixel regions are different, the test gray-scale values of the APS pixels and the test voltage values of the EVS pixels are also different between the sub-pixel regions, so that the interpolated gray-scale values and the test voltage values corresponding to the EVS pixels between the sub-pixel regions are different. In this way, multiple groups of sample data of different test voltage values and interpolated gray-scale values can be obtained, and the mapping relationship between the voltage value and the gray-scale value collected by the dual-mode fusion sensor can be accurately constructed. In the application stage, each pixel in the imaging pixel region of the dual-mode fusion sensor is used to collect images. The imaging pixel region is different from the calibration image region. The pixel aperture sizes of each pixel in the imaging pixel region are the same, and each pixel value has consistency for the collected gray-scale value and can be used for imaging. The real-time voltage value collected by the EVS pixels in the imaging pixel region can be calculated and converted through the mapping relationship to obtain the real-time gray-scale value of the EVS pixels in the imaging pixel region. This real-time gray-scale value includes dynamic event factors. The real-time gray-scale value can be used to fill the gray-scale value vacancy of the EVS pixels in the gray-scale image collected by the APS pixels corresponding in the imaging pixel region of the dual-mode fusion sensor. Thus, a gray-scale image with higher resolution, higher frame rate, and wider dynamic range can be combined and generated, improving the quality of the gray-scale image output by the dual-mode fusion sensor.

[0141] An embodiment of the present invention further provides a dual-mode fusion sensor, as Figure 8 shown, Figure 8 is a schematic structural diagram of an embodiment of the dual-mode fusion sensor provided in the embodiments of the present application.

[0142] The dual - mode fusion sensor integrates any one of the gray - scale mapping calibration devices based on reference pixels provided in the embodiments of the present invention. The dual - mode fusion sensor includes:

[0143] One or more processors;

[0144] A memory; and

[0145] One or more applications, where the one or more applications are stored in the memory and are configured to be executed by the processor to perform the steps in the gray - scale mapping calibration method based on reference pixels in any one of the embodiments of the gray - scale mapping calibration method based on reference pixels described above.

[0146] Specifically, the dual - mode fusion sensor may include components such as a processor 801 with one or more processing cores, a memory 802 of one or more computer - readable storage media, a power supply 803, and an input unit 804. Those skilled in the art can understand that Figure 8 the structure of the dual - mode fusion sensor shown in does not limit the dual - mode fusion sensor, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements. Among them:

[0147] The processor 801 is the control center of the dual - mode fusion sensor. It uses various interfaces and lines to connect all parts of the dual - mode fusion sensor. By running or executing software programs and / or modules stored in the memory 802, and by calling data stored in the memory 802, it executes various functions of the dual - mode fusion sensor and processes data, thereby monitoring the dual - mode fusion sensor as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It can be understood that the above - mentioned modem processor may not be integrated into the processor 801.

[0148] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the dual-modal fusion sensor, etc. In addition, the memory 802 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 802 can also include a memory controller to provide the processor 801 with access to the memory 802.

[0149] The dual-modal fusion sensor further includes a power supply 803 for supplying power to each component. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 803 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0150] The dual-modal fusion sensor may further include an input unit 804, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0151] Although not shown, the dual-modal fusion sensor may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 801 in the dual-modal fusion sensor will load the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 will run the application programs stored in the memory 802 to implement various functions as follows:

[0152] Obtain a calibration image file collected by the dual-modal fusion sensor. The calibration image file includes the test voltage values of EVS pixels and the test gray values of APS pixels within the reference pixel region of the dual-modal fusion sensor. Among them, the reference pixel region includes multiple sub-pixel regions, each sub-pixel region includes at least one APS pixel and at least one EVS pixel, the pixel aperture sizes of the APS pixels and EVS pixels within the sub-pixel region are the same, and the pixel aperture sizes corresponding to different sub-pixel regions are different;

[0153] For each of the sub-pixel regions, an interpolated gray value of the EVS pixels in the sub-pixel region is obtained according to the measured gray value of the APS pixels in the sub-pixel region;

[0154] According to the interpolated gray values of the EVS pixels in multiple sub-pixel regions and their measured voltage values, a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed;

[0155] According to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the dual-mode fusion sensor, the mapped gray value of the EVS pixels in the imaging pixel region is calculated, where the pixel opening sizes of the pixels in the imaging pixel region are the same.

[0156] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0157] Therefore, an embodiment of the present invention provides a computer-readable storage medium, which may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any one of the gray mapping calibration methods based on reference pixels provided by the embodiments of the present invention. For example, when the computer program is loaded by a processor, the following steps may be executed:

[0158] Obtain a calibration image file collected by the dual-mode fusion sensor. The calibration image file includes the measured voltage value of the EVS pixels in the reference pixel region of the dual-mode fusion sensor and the measured gray value of the APS pixels. The reference pixel region includes multiple sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. The pixel opening sizes of the APS pixels and the EVS pixels in the sub-pixel region are the same, and the corresponding pixel opening sizes between the sub-pixel regions are different;

[0159] For each of the sub-pixel regions, an interpolated gray value of the EVS pixels in the sub-pixel region is obtained according to the measured gray value of the APS pixels in the sub-pixel region;

[0160] According to the interpolated gray values of the EVS pixels in multiple sub-pixel regions and their measured voltage values, a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed;

[0161] According to the mapping relationship and the real-time voltage values collected by the EVS pixels within the imaging pixel region of the bimodal fusion sensor, the mapped gray values of the EVS pixels within the imaging pixel region are calculated, where the aperture sizes of the pixels within the imaging pixel region are consistent.

[0162] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not detailed in a certain embodiment, reference may be made to the detailed descriptions of other embodiments above, and details will not be repeated here.

[0163] In specific implementation, the above units or structures can be implemented as independent entities, or can be combined arbitrarily to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the method embodiments above, and details will not be repeated here.

[0164] The specific implementation of the above operations can refer to the previous embodiments, and details will not be repeated here.

[0165] The above provides a detailed introduction to a gray mapping calibration method based on reference pixels provided by the embodiments of the present application. Specific examples are used herein to elaborate on the principle and implementation manner of the present invention. The descriptions of the above embodiments are only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A gray-scale mapping calibration method based on reference pixels, characterized in that, the method includes: Obtain a calibration image file collected by a dual-modal fusion sensor, where the calibration image file includes the test voltage value of the EVS pixels in the reference pixel area of the dual-modal fusion sensor and the test gray-scale value of the APS pixels. Among them, the reference pixel area includes multiple sub-pixel areas, each sub-pixel area includes at least one APS pixel and at least one EVS pixel, the pixel aperture sizes of the APS pixels and EVS pixels in the sub-pixel area are the same, and the corresponding pixel aperture sizes between the sub-pixel areas are different; For each of the sub-pixel areas, obtain the interpolated gray-scale value of the EVS pixels in the sub-pixel area according to the test gray-scale value of the APS pixels in the sub-pixel area; Construct a mapping relationship between the voltage value and the gray-scale value collected by the dual-modal fusion sensor according to the interpolated gray-scale values and the test voltage values of the EVS pixels in multiple sub-pixel areas; Calculate the mapped gray-scale value of the EVS pixels in the imaging pixel area according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel area of the dual-modal fusion sensor, where the pixel aperture sizes of the pixels in the imaging pixel area are the same.

2. The gray-scale mapping calibration method based on reference pixels according to claim 1, characterized in that, before obtaining the calibration image file collected by the dual-modal fusion sensor, it further includes: Irradiate the reference pixel area with a preset light source; Control the APS pixels and EVS pixels in the reference pixel area to collect optical signals simultaneously; According to the first optical signal collected by the APS pixels in the reference pixel area, convert it into the test gray-scale value of the APS pixels, and according to the second optical signal collected by the EVS pixels in the reference pixel area, convert it into the test voltage value of the EVS pixels.

3. The gray-scale mapping calibration method based on reference pixels according to claim 2, characterized in that, the method further includes: Execute the step of controlling the APS pixels and EVS pixels in the reference pixel area to collect optical signals simultaneously at intervals of a preset time.

4. The gray-scale mapping calibration method based on reference pixels according to claim 1, characterized in that, the step of constructing a mapping relationship between the voltage value and the gray-scale value collected by the dual-modal fusion sensor according to the interpolated gray-scale values and the test voltage values of the EVS pixels in multiple sub-pixel areas includes: Input the interpolated gray-scale values and the test voltage values of the EVS pixels in multiple sub-pixel areas into a preset fitting function, and calculate the fitting parameters in the preset fitting function; Substitute the fitting parameters into the preset fitting function to obtain the mapping relationship.

5. The gray-scale mapping calibration method based on reference pixels according to claim 1, characterized in that, For each of the sub-pixel regions, the step of obtaining the interpolated gray value of the EVS pixels in the sub-pixel region according to the measured gray value of the APS pixels in the sub-pixel region further includes: Obtaining the distances between the APS pixels and the EVS pixels in the sub-pixel region; Determining the weight value corresponding to each APS pixel according to the distances corresponding to the APS pixels; Performing weighted calculation based on the weight value corresponding to each APS pixel and its corresponding measured gray value to obtain the interpolated gray value.

6. The gray-scale mapping calibration method based on a reference pixel according to claim 1, wherein, after calculating the mapped gray value of the EVS pixels in the imaging pixel region of the dual-mode fusion sensor according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the dual-mode fusion sensor, it further includes: Obtaining the actual gray value collected by the APS pixels in the imaging pixel region at the same moment when the real-time voltage value is collected; Combining and outputting a target gray image according to the real-time gray value of the APS pixels and the mapped gray value of the EVS pixels.

7. A gray-scale mapping calibration device based on a reference pixel, wherein, the gray-scale mapping calibration device based on a reference pixel includes: An acquisition module, configured to acquire a calibration image file collected by a dual-mode fusion sensor, where the calibration image file includes the measured voltage value of the EVS pixels in the reference pixel region of the dual-mode fusion sensor and the measured gray value of the APS pixels. Among them, the reference pixel region includes multiple sub-pixel regions, each sub-pixel region includes at least one APS pixel and at least one EVS pixel, the pixel aperture sizes of the APS pixels and the EVS pixels in the sub-pixel region are the same, and the pixel aperture sizes corresponding to different sub-pixel regions are different; An interpolation module, configured to, for each of the sub-pixel regions, obtain the interpolated gray value of the EVS pixels in the sub-pixel region according to the measured gray value of the APS pixels in the sub-pixel region; A construction module, configured to construct a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolated gray values of the EVS pixels in multiple sub-pixel regions and their measured voltage values; A calculation module, configured to calculate the mapped gray value of the EVS pixels in the imaging pixel region of the dual-mode fusion sensor according to the mapping relationship and the real-time voltage value collected by the EVS pixels in the imaging pixel region of the dual-mode fusion sensor, where the pixel aperture sizes of the pixels in the imaging pixel region are consistent.

8. A dual-mode fusion sensor, wherein, the dual-mode fusion sensor includes: One or more processors; A memory; and One or more applications, where the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the gray-scale mapping calibration method according to any one of claims 1 to 7.

9. A computer-readable storage medium, wherein, A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the method for calibrating gray-scale mapping based on a reference pixel according to any one of claims 1 to 7.

Images