Reference pixel-based grayscale mapping calibration method and device, sensor and medium

Abstract

The application provides a reference pixel-based gray mapping calibration method and device, a sensor and a medium. The method comprises the following steps: acquiring a calibration image file collected by a dual-mode fusion sensor; the calibration image file comprises test voltage values of EVS pixels in a reference pixel area and test gray values of APS pixels; the reference pixel area comprises a plurality of sub-pixel areas; each sub-pixel area comprises at least one APS pixel and at least one EVS pixel; the pixel opening sizes of the APS pixels and the EVS pixels in each sub-pixel area are the same; the corresponding pixel opening sizes between the sub-pixel areas are different; for each sub-pixel area, an interpolated gray value of the EVS pixel can be determined, thereby obtaining the interpolated gray values of a plurality of EVS pixels and the test voltage values thereof; a mapping relationship between the voltage values and the gray values is constructed; and real-time gray values can be calculated according to the mapping relationship, which can be used to improve the quality of a gray image collected by the dual-mode fusion sensor.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image sensors, in particular to a reference pixel-based gray scale mapping calibration method and device, a sensor and a medium. BACKGROUND

[0002] The dynamic event vision sensor (EVS, Event-based Vision Sensor) is a new type of sensor which simulates the human retina and generates event signals in response to the brightness change of motion, so it can capture the brightness change of the scene at a very high frame rate, record the events at a specific time point and a specific position in the image, and form an event stream instead of a frame stream. The dynamic event vision sensor solves the problems of high power consumption, low frame rate and poor dynamic range of the APS pixel sensor (APS, Active Pixel Sensor), but it only retains the motion edge information in imaging and loses the rich detail information of the object. In the related technology, a dual-mode fusion sensor is proposed to fuse the functions of APS pixels and EVS pixels. However, when the dual-mode fusion sensor outputs a gray scale image, it can only output the gray scale image corresponding to the APS pixels, and the dynamic event vision sensor cannot collect gray scale information, so it does not participate in the output of the gray scale image, resulting in the lack of gray scale values of the EVS pixels corresponding to the dynamic event vision sensor, which leads to low quality of the gray scale image collected by the dual-mode fusion sensor. SUMMARY

[0003] The present application provides a reference pixel-based gray scale mapping calibration method, device, sensor and medium, which aims to solve the technical problem of the lack of gray scale values of the EVS pixels in the imaging pixel area of the dynamic event vision sensor, resulting in low quality of the gray scale image collected by the dual-mode fusion sensor.

[0004] In one aspect, the method comprises:

[0005] obtaining a calibration image file collected by a dual-mode fusion sensor, the calibration image file comprising test voltage values of EVS pixels in a reference pixel area of the dual-mode fusion sensor, and test gray scale values of APS pixels, wherein the reference pixel area comprises a plurality of sub-pixel areas, each sub-pixel area comprising at least one APS pixel and at least one EVS pixel, the APS pixels and the EVS pixels in each sub-pixel area having the same pixel opening size, and the corresponding pixel opening sizes between the sub-pixel areas being different;

[0006] for each sub-pixel area, obtaining an interpolated gray scale value of the EVS pixel in the sub-pixel area according to the test gray scale value of the APS pixel in the sub-pixel area;

[0007] According to the interpolation gray value of the EVS pixel in each of the plurality of sub-pixel regions and the test voltage value of the EVS pixel in each of the plurality of sub-pixel regions, a mapping relationship between the voltage value and the gray value collected by the dual-modal fusion sensor is constructed.

[0008] According to the mapping relationship and the real-time voltage value of the EVS pixel in the imaging pixel region of the dual-modal fusion sensor, a mapping gray value of the EVS pixel in the imaging pixel region is calculated, wherein the opening size of each pixel in the imaging pixel region is consistent.

[0009] In some embodiments of the present application, before the step of obtaining the calibration image file collected by the dual-modal fusion sensor, the method further comprises:

[0010] Illuminating a sensing region of the dual-modal fusion sensor by a preset light source;

[0011] Controlling the APS pixel and the EVS pixel on the sensing region to simultaneously collect optical signals;

[0012] According to the first optical signal collected by the APS pixel in the reference pixel region, a test gray value of the APS pixel is converted, and according to the second optical signal collected by the EVS pixel in the reference pixel region, a test voltage value of the EVS pixel is converted.

[0013] In some embodiments of the present application, the method further comprises:

[0014] Every interval of a preset time, the step of controlling the APS pixel and the EVS pixel on the sensing region to simultaneously collect optical signals is performed.

[0015] In some embodiments of the present application, the step of constructing the mapping relationship between the voltage value and the gray value collected by the dual-modal fusion sensor according to the interpolation gray value of the EVS pixel in each of the plurality of sub-pixel regions and the test voltage value of the EVS pixel in each of the plurality of sub-pixel regions comprises:

[0016] The interpolation gray value of the EVS pixel in each of the plurality of sub-pixel regions and the test voltage value of the EVS pixel in each of the plurality of sub-pixel regions are input into a preset fitting function, and a fitting parameter in the preset fitting function is calculated;

[0017] The fitting parameter is substituted into the preset fitting function, and the mapping relationship is obtained.

[0018] In some embodiments of the present application, the step of obtaining, for each of the plurality of sub-pixel regions, the interpolation gray value of the EVS pixel in the sub-pixel region according to the test gray value of the APS pixel in the sub-pixel region further comprises:

[0019] The distance between each APS pixel and each EVS pixel in the sub-pixel region is obtained.

[0020] determining a weight value corresponding to each of the APS pixels according to the distance corresponding to each of the APS pixels;

[0021] obtaining the interpolation gray value according to the weight value corresponding to each of the APS pixels and the test gray value corresponding thereto.

[0022] In some embodiments of the present application, after the mapping gray value of the EVS pixel in the imaging pixel region is calculated according to the mapping relationship and the real-time voltage value collected by the EVS pixel in the imaging pixel region of the dual-mode fusion sensor, the method further comprises:

[0023] obtaining the actual gray value of the APS pixel in the imaging pixel region collected at the same time when the real-time voltage value is collected;

[0024] combining and outputting a target gray image according to the real-time gray value of the APS pixel and the mapping gray value of the EVS pixel.

[0025] On the other hand, the present application provides a gray mapping calibration device based on reference pixels, comprising:

[0026] an acquisition module, configured to acquire a calibration image file collected by a dual-mode fusion sensor, the calibration image file comprising a test voltage value of an EVS pixel in a reference pixel region of the dual-mode fusion sensor and a test gray value of an APS pixel, wherein the reference pixel region comprises a plurality of sub-pixel regions, each of the sub-pixel regions comprises at least one APS pixel and at least one EVS pixel, the pixel opening size of the APS pixel and the EVS pixel in each of the sub-pixel regions is the same, and the pixel opening size corresponding to each of the sub-pixel regions is different;

[0027] an interpolation module, configured to obtain, for each of the sub-pixel regions, an interpolation gray value of the EVS pixel in the sub-pixel region according to the test gray value of the APS pixel in the sub-pixel region;

[0028] a construction module, configured to construct a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolation gray value of the EVS pixel in each of the sub-pixel regions and the test voltage value thereof;

[0029] a calculation module, configured to calculate a mapping gray value of an EVS pixel in an imaging pixel region according to the mapping relationship and a real-time voltage value collected by the EVS pixel in the imaging pixel region of the dual-mode fusion sensor, wherein the pixel opening size in the imaging pixel region is consistent.

[0030] In another aspect, the application also provides a dual-mode fusion sensor, comprising:

[0031] one or more processors;

[0032] a memory; and

[0033] one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps of the reference pixel-based gray scale mapping calibration method.

[0034] In another aspect, the application also provides a computer readable storage medium having stored thereon a computer program, which is loaded by a processor to execute the steps of the reference pixel-based gray scale mapping calibration method.

[0035] The technical scheme of the embodiments of the application comprises: obtaining a calibration image file collected by a dual-mode fusion sensor, the calibration image file comprising test voltage values of EVS pixels in a reference pixel region of the dual-mode fusion sensor and test gray scale values of APS pixels, wherein the reference pixel region comprises a plurality of sub-pixel regions, each of the sub-pixel regions comprises at least one APS pixel and at least one EVS pixel, the pixel opening sizes of the APS pixels and the EVS pixels in each of the sub-pixel regions are the same, and the pixel opening sizes corresponding to the sub-pixel regions are different; for each of the sub-pixel regions, obtaining an interpolated gray scale value of the EVS pixel in the sub-pixel region according to the test gray scale value of the APS pixel in the sub-pixel region; constructing a mapping relationship between voltage values and gray scale values collected by the dual-mode fusion sensor according to the interpolated gray scale values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values thereof; and calculating a mapping gray scale value of an EVS pixel in an imaging pixel region of the dual-mode fusion sensor according to the mapping relationship and a real-time voltage value collected by the EVS pixel in the imaging pixel region, wherein the pixel opening sizes in the imaging pixel region are consistent.

[0036] In the technical scheme of the embodiment of the present application, the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed by using a calibration file image file containing the test gray value of the APS pixel and the test voltage value of the EVS pixel in the reference pixel region. The reference pixel region includes a plurality of sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, the pixel opening sizes are the same, and the test gray value of the APS pixel in the same sub-pixel region can be used to obtain the interpolation gray value of the EVS pixel by interpolation, so that the EVS pixel in each sub-pixel region corresponds to the interpolation gray value and the test voltage value. Because the pixel opening sizes of the sub-pixel regions are different, the test gray value of the APS pixel and the test voltage value of the EVS pixel are different between the sub-pixel regions, so that the interpolation gray value and the test voltage value of the EVS pixel are different between the sub-pixel regions. Thus, a plurality of different sample data of the test voltage value and the interpolation gray value can be obtained, and the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor can be accurately constructed. In the application stage, the pixels in the imaging pixel region of the dual-mode fusion sensor are used to collect images, the imaging pixel region is different from the calibration image region, the opening sizes of the pixels in the imaging pixel region are the same, the pixel values have consistency in collecting the gray value, and can be used for imaging. The real-time voltage value collected by the EVS pixel in the imaging pixel region can be converted into the real-time gray value of the EVS pixel in the imaging pixel region by using the mapping relationship, the real-time gray value contains the dynamic event factor, and the real-time gray value can be used to fill the gray value vacancy of the EVS pixel in the gray image collected by the APS pixel in the imaging pixel region of the dual-mode fusion sensor, so that a gray image with higher resolution, higher frame rate and wider dynamic range can be generated, and the quality of the gray image output by the dual-mode fusion sensor is improved. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 is an embodiment flow diagram of the gray mapping calibration method based on reference pixels provided in the embodiments of the present application;

[0039] Figure 2 is a structural diagram of the sensing region of the dual-mode fusion sensor involved in the embodiments of the present application;

[0040] Figure 3 is a structural schematic view of a sub-pixel region of a dual-modal fusion sensor according to an embodiment of the present application;

[0041] Figure 4 is an embodiment flowchart of a method for calibrating a gray scale mapping based on reference pixels according to an embodiment of the present application;

[0042] Figure 5 is a structural schematic view of a calibration module of a dual-modal fusion sensor according to an embodiment of the present application;

[0043] Figure 6 is an embodiment flowchart of a method for constructing a mapping relationship between voltage values and gray scale values collected by a dual-modal fusion sensor according to an embodiment of the present application;

[0044] Figure 7 is an embodiment structural schematic view of a device for calibrating a gray scale mapping based on reference pixels according to an embodiment of the present application;

[0045] Figure 8 is an embodiment structural schematic view of a dual-modal fusion sensor according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0047] In the description of the present application, it should be understood that the terms “center”, “longitudinal”, “transverse”, “length”, “width”, “thickness”, “upper”, “lower”, “front”, “rear”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, “outer” and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application. In addition, the terms “first”, “second” are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features limited by “first”, “second” can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of “multiple” is two or more, unless otherwise specifically limited.

[0048] In this application, the word "exemplary" is used to mean "serving as an example, instance, or illustration." Any implementation described as "exemplary" in this application is not necessarily to be construed as preferred or advantageous over other implementations. The following description is presented to enable any person skilled in the art to make and use the application. In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of the present application. It will be apparent to one skilled in the art, however, that the present application can be practiced without using these specific details. In other instances, well-known structures and processes are not elaborated in order not to obscure the description of the present application with unnecessary details. Thus, the present application is not intended to be limited by the embodiments shown, but is to be accorded with the widest scope consistent with the principles and features disclosed herein.

[0049] A dual-mode fusion sensor is proposed in the related art to fuse the functions of APS pixels and EVS pixels, but when a gray scale image is output using the dual-mode fusion sensor, only the gray scale image corresponding to the APS pixels can be output, and the information collected by the EVS pixels does not participate in the output of the gray scale image, resulting in poor quality of the gray scale image output by the dual-mode fusion sensor.

[0050] Based on this, a reference pixel-based gray scale mapping calibration method, device, sensor and medium are proposed in the embodiments of the present application, which are described in detail below.

[0051] The reference pixel-based gray scale mapping calibration method in the embodiments of the present application is applied to a reference pixel-based gray scale mapping calibration device, which can be arranged in a dual-mode fusion sensor or other terminal equipment such as a computer or a mobile phone. The dual-mode fusion sensor is provided with one or more processors, a memory, and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the reference pixel-based gray scale mapping calibration method.

[0052] The dual-mode fusion sensor in the embodiments of the present application is mainly used for:

[0053] The calibration image file collected by the dual-mode fusion sensor is acquired, the calibration image file includes the test voltage value of the EVS pixel in the reference pixel region and the test gray scale value of the APS pixel, wherein the reference pixel region includes a plurality of sub-pixel regions, the sub-pixel region includes at least one APS pixel and at least one EVS pixel, the pixel opening size of the APS pixel and the EVS pixel in the sub-pixel region is the same, and the corresponding pixel opening size between the sub-pixel regions is not the same;

[0054] According to a test gray value of the APS pixel in the sub-pixel region, an interpolation gray value of the EVS pixel in the sub-pixel region is obtained;

[0055] According to the interpolation gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values thereof, a mapping relationship between voltage values and gray values collected by the dual-mode fusion sensor is constructed;

[0056] According to the mapping relationship and real-time voltage values of the EVS pixels in an imaging pixel region of the dual-mode fusion sensor, a mapping gray value of the EVS pixel in the imaging pixel region is calculated, wherein the opening sizes of the pixels in the imaging pixel region are consistent.

[0057] In this way, in the application stage, the pixels in the imaging pixel region of the dual-mode fusion sensor are used to collect images, the imaging pixel region is different from the calibration image region, the opening sizes of the pixels in the imaging pixel region are the same, the pixel values have consistency in the collection of gray values, and can be used for imaging. The real-time voltage values of the EVS pixels in the imaging pixel region can be converted into real-time gray values of the EVS pixels in the imaging pixel region through the mapping relationship, the real-time gray values contain dynamic event factors, the real-time gray values can be used to fill in the gray value vacancy of the EVS pixels in the gray image collected by the APS pixels in the imaging pixel region of the dual-mode fusion sensor, and therefore a gray image with higher resolution, higher frame rate and wider dynamic range can be generated by combination, and the quality of the gray image output by the dual-mode fusion sensor is improved.

[0058] Hereinafter, the content claimed in the claims of the present application is explained and described through specific exemplary schemes, so that those skilled in the art can better understand the protection scope of the claims of the present application. It can be understood that the following exemplary schemes do not limit the protection scope of the present application, but are only used to explain the present application.

[0059] Exemplarily, as shown in FIG. 1, Figure 1 Figure 1 FIG. 1 is a flowchart of an embodiment of a gray mapping calibration method based on reference pixels in the embodiments of the present application. The gray mapping calibration method based on reference pixels includes the following steps 101-104.

[0060] ​101. Obtain a calibration image file collected by the dual-mode fusion sensor, the calibration image file comprising test voltage values of EVS pixels in a reference pixel region of the dual-mode fusion sensor, and test gray scale values of APS pixels, wherein the reference pixel region comprises a plurality of sub-pixel regions, each of the sub-pixel regions comprising at least one APS pixel and at least one EVS pixel, the APS pixels and the EVS pixels in each of the sub-pixel regions having the same pixel opening size, and the corresponding pixel opening sizes of the sub-pixel regions being different from each other;

[0061] In the embodiment, the gray scale mapping calibration method based on reference pixels can be applied to a gray scale mapping calibration device based on reference pixels, which can be arranged in the dual-mode fusion sensor or connected to a terminal device.

[0062] The dual-mode fusion sensor can collect a calibration image file for a target scene. The calibration image file can be in RAW format, where RAW refers to a binary format in which the dual-mode fusion sensor converts a captured light signal into a data signal.

[0063] The dual-mode fusion sensor comprises a plurality of sensing units, and the calibration image file comprises information collected by the sensing units of the dual-mode fusion sensor. Each sensing unit corresponds to one pixel. According to different types of data collected by the sensing units, the corresponding pixels are also different. The sensing units in the dual-mode fusion sensor can correspond to APS (EVS, Event-based Vision Sensor) data mode and EVS (APS, ActivePixel Sensor) data mode, respectively. The pixel types can be divided into APS pixels and EVS pixels, wherein the APS data mode corresponds to the APS pixels, and the EVS data mode corresponds to the EVS pixels. Compared with a dual-mode vision sensor composed of a plurality of sensor modules arranged separately, the device volume of the sensor is effectively compressed, which is more conducive to the miniaturization of the overall architecture and eliminates the problem of image fusion matching between different sensors.

[0064] In actual application, the EVS pixels and the APS pixels can be exposed synchronously when the dual-mode fusion sensor collects the calibration image file. The EVS pixels are configured with a photodiode integrated with a capacitor for collecting electric charges. The photodiode generates a photocurrent in response to the intensity of incident light, and then generates a voltage value of the corresponding EVS pixel according to the photocurrent.

[0065] In the dual-mode fusion sensor, the APS pixels correspond to a gray value, and the EVS pixels correspond to a voltage value. In the calibration stage, the voltage value is used as the test voltage value in this embodiment, and the gray value of the APS pixels in the dual-mode fusion sensor is used as the test gray value in this embodiment. The voltage value of the EVS pixel and the gray value of the adjacent APS pixel jointly constitute the above-mentioned calibration image file.

[0066] In this embodiment, referring to Figure 2 , the dual-mode fusion sensor region is composed of a plurality of EVS pixels and a plurality of APS pixels, wherein A represents an APS pixel, and E represents an EVS pixel. The sensing region is composed of a reference pixel region and an imaging pixel region. The reference pixel region can be flexibly divided according to the actual application scene, and can be arranged at the edge position of the sensing region, so as to reduce the influence on the sensing effect of the dual-mode fusion sensor and improve the sensing performance of the dual-mode fusion sensor. The reference pixel region can be further divided into a plurality of sub-sensing regions, each of which includes at least one APS pixel and at least one EVS pixel. In the same sub-pixel region, the pixel opening sizes of the pixel units are consistent, that is, the pixel opening sizes of the APS pixels and the EVS pixels are the same. Since the pixel opening sizes are the same, the light intensity of the incident light received by them is also the same. For the same sub-pixel region, the interpolation gray value of the EVS pixel can be more accurately obtained based on the test gray value of the APS pixel. The opening sizes of the pixel units in different sub-pixel regions are not the same, that is, the pixel opening sizes of the APS pixels and the EVS pixels are not the same. The pixel opening sizes can be set to be different by setting different opening rates between different sub-pixel regions, for example, using opaque tungsten to shield part of the pixel region, or different bias voltages can be set between different sub-pixel regions to change the photosensitive sensitivity of the reference pixels. The reference pixel region is composed of sub-pixel regions with different photosensitive sensitivities. Since the pixel opening sizes are not the same, the light intensity of the incident light received by them is also not the same. The test gray value of the APS pixel and the test voltage value of the EVS pixel in each sub-pixel region are not the same. In the imaging pixel region, the pixel opening sizes are the same, so the light intensity of the incident light received by them is also the same, and imaging can be performed by using the pixels in the imaging pixel region.

[0067] It should be understood that, Figure 2 The array size in the array is only used for illustrative purposes and should not be understood as the only limitation of the array size of the overall pixel array in actual applications, and the layout and number of the two types of pixels in the overall pixel array can be determined according to the actual application scene, and the embodiment is not limited.

[0068] 102. For each of the sub-pixel regions, according to the test gray value of the APS pixel in the sub-pixel region, an interpolation gray value of the EVS pixel in the sub-pixel region is obtained;

[0069] In the embodiment, the calibration image file collected in the calibration stage includes the test voltage value corresponding to the EVS pixel of the reference pixel region and the test gray value corresponding to the APS pixel. If the information collected by the EVS pixel is used to calculate the gray value of the EVS pixel, the mapping relationship between the voltage value collected by the dual-mode fusion sensor and the corresponding gray value, that is, the mapping relationship between the voltage value and the gray value collected by the EVS pixel, needs to be obtained. Therefore, the test voltage value and the gray value corresponding to the EVS pixel need to be obtained in the calibration stage. In the reference pixel region, there are multiple sub-pixel regions, and the pixel opening size of the APS pixel and the EVS pixel in each sub-pixel region is the same. Therefore, for the gray value of the EVS pixel in each sub-pixel region, the test gray value of the neighborhood APS pixel in the same sub-pixel region can be used to determine the interpolation gray value of the EVS pixel. According to the test gray value of the APS pixel in the sub-pixel region, the interpolation gray value of the EVS pixel can be obtained by interpolation calculation.

[0070] Please refer to Figure 3 In the reference pixel region 100 corresponding to the calibration image file, the sub-pixel region 110 is included. For the sub-pixel region 110, the sub-pixel region 110 includes 2*2 pixels, and there are three adjacent APS pixels corresponding to the EVS pixel. According to the test gray value of the three adjacent APS pixels, the difference gray value of the EVS pixel can be obtained. Since each APS pixel and the EVS pixel are in the same sub-pixel region and have the same pixel opening size, the authenticity of the interpolation gray value obtained by fusing the gray value of the APS pixel can be improved, and the accuracy of the preset mapping relationship constructed can be improved.

[0071] Optionally, the step 102 further includes:

[0072] The distance between each APS pixel and the EVS pixel in the sub-pixel region is obtained.

[0073] According to the distance corresponding to each APS pixel, a weight value corresponding to each APS pixel is determined.

[0074] According to the weight value corresponding to each APS pixel and the corresponding test gray value, a weighted operation is performed to obtain the interpolation gray value.

[0075] In the embodiment, the way of calculating the interpolation gray value of the EVS in the sub-pixel region can be to perform weighted average calculation on the test gray values of each APS pixel in the same sub-pixel region, and the result of the weighted average calculation is taken as the interpolation gray value of the EVS pixel. Further, the embodiment can also assign corresponding weight values to different APS pixels based on the relative distance between the APS pixels and the EVS pixel. First, the relative distance between the EVS pixel and each APS pixel in the same sub-pixel region needs to be obtained, and the weight value of the corresponding test gray value of each APS pixel can be determined according to the ratio between the relative distances corresponding to the EVS pixel and each APS pixel. The greater the relative distance, the smaller the corresponding weight value can be. For example, a test gray value corresponding to an APS pixel closer to the position of the EVS pixel can be assigned a larger weight value, while a test gray value corresponding to an APS pixel farther from the position of the EVS pixel can be assigned a larger weight value.

[0076] 103. Constructing a mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor according to the interpolation gray value of the EVS pixel of each of the plurality of sub-pixel regions and the test voltage value thereof;

[0077] In the embodiment, the test voltage value corresponding to the EVS pixel can be obtained based on the dynamic event vision sensor, by obtaining the interpolation gray value from the test gray values of the adjacent APS pixels in the same sub-pixel region. For the same sub-pixel region, at least one test voltage value corresponding to the EVS pixel and the sample data composed of the interpolation gray value thereof can be obtained. For different sub-pixel regions, because the pixel opening sizes of the APS pixels and the EVS pixels are different, the light intensities of the incident light received by the APS pixels and the EVS pixels are also different. The test gray values of the APS pixels and the test voltage values of the EVS pixels in each sub-pixel region are inconsistent. Because the interpolation gray value is calculated from the test gray values of the APS pixels in the same sub-pixel region, the interpolation gray values corresponding to the EVS pixels in different sub-pixel regions are also different. Thus, more accurate mapping relationship between the voltage value and the gray value collected by the dynamic event vision sensor can be constructed according to the plurality of sample data. The mapping relationship can be in the form of a mapping table or a mapping function, which can be selected according to specific requirements. The mapping table composed of the test voltage values corresponding to the EVS pixels of the plurality of sub-pixel regions and the interpolation gray values thereof can be taken as the mapping relationship, or the function constructed from the test voltage values corresponding to the EVS pixels of the plurality of sub-pixel regions and the interpolation gray values thereof can be taken as the mapping relationship.

[0078] 104. According to the mapping relationship and the real-time voltage value collected by the EVS pixel in the imaging pixel region of the dual-modal fusion sensor, a mapping gray value of the EVS pixel in the imaging pixel region is calculated, wherein the opening size of each pixel in the imaging pixel region is consistent.

[0079] In the present embodiment, in the calibration stage, the mapping relationship between the voltage value and the gray value collected by the dual-modal fusion sensor is constructed by referring to the pixel region, and in the application stage, the dual-modal fusion sensor collects a gray image through the imaging pixel region. The opening size of each pixel in the imaging pixel region is consistent. In the application stage, the EVS pixel in the imaging pixel region of the dual-modal fusion sensor collects a real-time voltage value, and the mapping relationship obtained in the calibration stage can be used to map the mapping gray value corresponding to the EVS pixel. The mapping gray value is determined by the real-time voltage value collected by the EVS pixel and contains a dynamic event factor. The mapping gray value can be used as the gray value collected by the dual-modal fusion sensor based on the dynamic event. Based on the mapping gray value, a higher-quality gray image of the dual-modal fusion sensor can be obtained.

[0080] Optionally, after step 104, the method further comprises:

[0081] acquiring an actual gray value of the APS pixel in the imaging pixel region collected at the same time when the real-time voltage value is collected;

[0082] combining and outputting a target gray image according to the real-time gray value of the APS pixel and the mapping gray value of the EVS pixel.

[0083] In the embodiment, the real-time voltage value corresponding to the EVS pixel collected by the dual-mode fusion sensor can be mapped to the mapping gray value corresponding to the EVS pixel in combination with the mapping relationship obtained in the calibration stage. In addition, the APS pixel in the imaging pixel region of the dual-mode fusion sensor can also be synchronously exposed with the EVS pixel in the imaging pixel region, that is, when the dual-mode fusion sensor collects the real-time voltage value through the EVS pixel in the imaging pixel region, the actual gray value can be collected through the APS pixel in the imaging pixel region at the same time. In this way, the real-time image file collected by the dual-mode sensor during imaging includes the real-time voltage value corresponding to the EVS pixel in the imaging pixel region and the real-time gray value corresponding to the APS pixel in the imaging pixel region. According to the real-time voltage value corresponding to the EVS pixel in the imaging pixel region and the mapping relationship, the mapping gray value of the EVS pixel in the imaging pixel region can be determined. Since the pixel positions of the EVS pixel and the APS pixel in the imaging pixel region are different, based on the pixel positions of the EVS pixel and the APS pixel in the imaging pixel region, the mapping gray value of the EVS pixel can be combined with the real-time gray value of the APS pixel collected at the same time to generate a target gray image. The target gray image has pixel positions corresponding to the APS pixel or the EVS pixel. The generation of the target gray image uses both the gray information of the APS pixel and the dynamic voltage information of the EVS pixel, so that the details of the target gray image are more abundant and the quality is higher.

[0084] Optionally, due to the pixel opening setting of the reference pixel region, after the calibration is completed, the pixel unit in the reference pixel region can be imaged by shielding or the like.

[0085] In the technical solution disclosed in the embodiment, the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed by using the calibration file image file containing the test gray value of the APS pixel and the test voltage value of the EVS pixel in the reference pixel region. The reference pixel region includes a plurality of sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, the pixel opening sizes are the same, and the test gray value of the APS pixel in the same sub-pixel region can be used to obtain the interpolation gray value of the EVS pixel by interpolation. Thus, the EVS pixels in each sub-pixel region correspond to the interpolation gray value and the test voltage value. Because the pixel opening sizes of the sub-pixel regions are different, the test gray value of the APS pixel and the test voltage value of the EVS pixel are different between the sub-pixel regions, and the EVS pixels in the sub-pixel regions correspond to different interpolation gray values and test voltage values. Thus, a plurality of different sample data of the test voltage value and the interpolation gray value can be obtained, and the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor can be accurately constructed. In the application stage, the pixels in the imaging pixel region of the dual-mode fusion sensor are used to collect images. The imaging pixel region is different from the calibration image region, the opening sizes of the pixels in the imaging pixel region are the same, the pixel values have consistency in collecting the gray value, and the pixels can be used for imaging. The real-time voltage value collected by the EVS pixel in the imaging pixel region can be converted into the real-time gray value of the EVS pixel in the imaging pixel region by using the mapping relationship. The real-time gray value contains the dynamic event factor, and the real-time gray value can be used to fill the gray value vacancy of the EVS pixel in the gray image collected by the APS pixel in the imaging pixel region of the dual-mode fusion sensor. Thus, a gray image with higher resolution, higher frame rate, and wider dynamic range can be generated, and the quality of the gray image output by the dual-mode fusion sensor is improved.

[0086] It can be understood that, because the test voltage value and the test gray value of the reference pixel region are used to construct the mapping relationship in the calibration stage in the embodiment, the mapping relationship can be calibrated by using a single image, the success rate of calibration is high, and the calibration speed is faster. The number of gray levels in the collected calibration image file is related to the pixel opening of the reference pixel region, and the pixel opening of the reference pixel region can be set arbitrarily according to the requirement, the flexibility is higher, and the user experience is improved. When the mapping relationship is constructed and the mapping gray value is determined, a complex image processing algorithm does not need to be programmed, the calibration cost is lower, the efficiency of constructing the mapping relationship in the calibration stage is improved, and the quality of the gray image output by the dual-mode fusion sensor can be improved faster.

[0087] As shown in Figure 4 , the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor is constructed by using the calibration file image file containing the test gray value of the APS pixel and the test voltage value of the EVS pixel in the reference pixel region. The reference pixel region includes a plurality of sub-pixel regions, and each sub-pixel region includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, the pixel opening sizes are the same, and the test gray value of the APS pixel in the same sub-pixel region can be used to obtain the interpolation gray value of the EVS pixel by interpolation. Thus, the EVS pixels in each sub-pixel region correspond to the interpolation gray value and the test voltage value. Because the pixel opening sizes of the sub-pixel regions are different, the test gray value of the APS pixel and the test voltage value of the EVS pixel are different between the sub-pixel regions, and the EVS pixels in the sub-pixel regions correspond to different interpolation gray values and test voltage values. Thus, a plurality of different sample data of the test voltage value and the interpolation gray value can be obtained, and the mapping relationship between the voltage value and the gray value collected by the dual-mode fusion sensor can be accurately constructed. In the application stage, the pixels in the imaging pixel region of the dual-mode fusion sensor are used to collect images. The imaging pixel region is different from the calibration image region, the opening sizes of the pixels in the imaging pixel region are the same, the pixel values have consistency in collecting the gray value, and the pixels can be used for imaging. The real-time voltage value collected by the EVS pixel in the imaging pixel region can be converted into the real-time gray value of the EVS pixel in the imaging pixel region by using the mapping relationship. The real-time gray value contains the dynamic event factor, and the real-time gray value can be used to fill the gray value vacancy of the EVS pixel in the gray image collected by the APS pixel in the imaging pixel region of the dual-mode fusion sensor. Thus, a gray image with higher resolution, higher frame rate, and wider dynamic range can be generated, and the quality of the gray image output by the dual-mode fusion sensor is improved. Figure 4An embodiment flowchart for acquiring a calibration image file in the reference pixel-based grayscale mapping calibration method provided in the embodiments of the present application is shown in FIG. 4, which includes steps 401-403.

[0088] 401. Irradiate the reference pixel region with a preset light source.

[0089] In the present embodiment, when calibrating the dual-modal fusion sensor, the calibration module includes not only the sensing region of the dual-modal fusion sensor, but also a preset light source, which can be an internal light source of the dual-modal fusion sensor. In the calibration stage, the preset light source irradiates the sensing region of the dual-modal fusion sensor with optical signals, i.e., irradiates the reference pixel region with light, and the light intensity irradiated by the preset light source is generally the same.

[0090] For example, referring to FIG. 5, Figure 5 the calibration module includes an internal light source 501, a sensing region 502, and a lens 503. In the calibration stage, the internal light source irradiates the sensing region with optical signals, and in the application stage, the reflected light of the target to be captured is captured by the sensing region through the lens to obtain the corresponding image.

[0091] 402. Control the APS pixels and EVS pixels in the reference pixel region to simultaneously acquire optical signals.

[0092] In the present embodiment, after the preset light source irradiates the sensing region of the dual-modal fusion sensor with optical signals, the APS pixels and EVS pixels on the reference pixel region are controlled to simultaneously acquire the optical signals emitted by the preset light source when the preset light source irradiates the optical signals.

[0093] 403. Convert the first optical signal acquired by the APS pixel in the reference pixel region into a test grayscale value of the APS pixel, and convert the second optical signal acquired by the EVS pixel in the reference pixel region into a test voltage value of the EVS pixel.

[0094] In the present embodiment, the optical signals acquired by the APS pixels and EVS pixels in the reference pixel region can be further processed. The first optical signal acquired by the APS pixel in the reference pixel region is converted into a test grayscale value by the APS pixel, and the second optical signal acquired by the EVS pixel in the reference pixel region is converted into a test voltage value by the EVS pixel, thereby obtaining a calibration image file.

[0095] It should be noted that the calibration image file includes the test voltage values of the EVS pixels in the reference pixel region and the test voltage values of the APS pixels, and can also include the voltage values of the EVS pixels and the gray values of the APS pixels in the imaging pixel region, and when the mapping relationship between the voltage values and the gray values collected by the dual-mode fusion sensor is constructed, the test voltage values of the EVS pixels in the reference pixel region and the test voltage values of the APS pixels can be obtained from the calibration image file.

[0096] In the preset light source irradiates the reference pixel region, the pixel opening size in the same sub-pixel region is consistent, so that the light intensity received by the pixel unit in the same sub-pixel region is consistent, thereby the gray value output is also relatively consistent, and the pixel opening size between different pixel regions is consistent, so that the light intensity received by the pixel unit in the same sub-pixel region is inconsistent, thereby the gray value output is also inconsistent.

[0097] In the technical scheme disclosed in the embodiment, the test gray values of the APS pixels and the test voltage values of the EVS pixels in the reference pixel region are obtained by irradiating the sensing region of the dual-mode fusion sensor by the preset light source, and then the mapping relationship between the voltage values and the gray values collected by the dual-mode fusion sensor can be constructed. In this way, the sensing region is irradiated by the fixed preset light source to obtain the interpolation gray values and the test voltage values of the multiple EVS pixels in the reference pixel region, and the sensor pixel array is irradiated by the internal light source, which can reduce the complexity of calibration and obtain the interpolation gray values and the test voltage values of the multiple EVS pixels in the reference pixel region without the aid of external conditions.

[0098] Further, the method further comprises:

[0099] Every interval preset time, the step of controlling the APS pixels and the EVS pixels in the sensing region to collect optical signals at the same time is performed.

[0100] In the embodiment, the preset light source and the reference pixel region are real-time during the working of the dual-mode fusion sensor. Every interval preset time, for example, every interval 1s, which can also be other time intervals, the APS pixels and the EVS pixels in the reference pixel region are controlled to collect the optical signals emitted by the preset light source at the same time, the calibration image file is obtained again, and the mapping parameters are calculated again. The mapping parameters updated in a short time can compensate for the influence of temperature change on the mapping accuracy, thereby improving the accuracy of the mapping relationship.

[0101] It should be noted that the reference pixel can be recalibrated gray mapping every fixed time (for example, 1s). This frequent recalibration can naturally solve the change of gray mapping caused by temperature change. That is, the gray mapping relationship is real-time self-calibrated and is not disturbed by temperature, which helps the imaging pixel to obtain accurate voltage-gray conversion.

[0102] As Figure 6 shown, Figure 6 is a flowchart of an embodiment of constructing a mapping relationship between voltage values and gray values collected by a dual-modal fusion sensor provided in the embodiments of the present application, comprising steps 601-602:

[0103] 601, input the interpolation gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values thereof to a preset fitting function, and calculate fitting parameters in the preset fitting function.

[0104] 602, substitute the fitting parameters into the preset fitting function to obtain the mapping relationship.

[0105] In the embodiment, the mapping relationship can be a functional relationship between voltage values and gray values collected by the dual-modal fusion sensor, and the determination of the mapping relationship can be determined by a preset fitting function. The preset fitting function between the voltage values and the gray values collected by the dual-modal fusion sensor is first set, and the fitting function can be in the form of exponential, linear, logarithmic, polynomial, etc., which is not uniquely limited in the embodiment. The sample data composed of the test voltage values corresponding to the EVS pixels in the plurality of sub-pixel regions in the reference pixel region and the interpolation gray values thereof are input to the preset fitting function, the fitting parameters in the preset fitting function are calculated, the fitting parameters are substituted into the preset fitting function, and the global unified mapping relationship of the dual-modal fusion sensor is obtained.

[0106] Specifically, the test voltage values corresponding to the EVS pixels in the plurality of sub-pixel regions and the interpolation gray values thereof can compose a plurality of sample data, each sample data is in the form of (X: test voltage value, Y: interpolation gray value), the sample data is input to the preset fitting function, and the fitting parameters in the preset fitting function are calculated. The preset fitting function can be expressed as: I=f(V out ), wherein V out represents the test voltage value corresponding to the EVS pixel, and I represents the interpolation gray value corresponding to the EVS pixel. It should be understood that the mapping relationship of f(·) in the above fitting function can be in the form of exponential, linear, logarithmic, polynomial, etc., which is not uniquely limited in the embodiment.

[0107] In one implementation mode, if the above fitting function is implemented by using a linear mapping relationship, the fitting function can be expressed as:

[0108] I=a*V out +b

[0109] wherein V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolation gray value corresponding to the EVS pixel, and a and b represent fitting parameters.

[0110] In another implementation, if the fitting function is implemented by using a logarithmic mapping relationship, the fitting function can be expressed as:

[0111] I = t * I0 * log[(V - V out ) / k]

[0112] wherein V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, a and b represent fitting parameters, t represents the exposure time of the APS pixel, and I0, V and k represent fitting parameters.

[0113] In yet another implementation, if the fitting function is implemented by using a polynomial mapping relationship, the fitting function can be expressed as:

[0114] I = a + b1V out + b2V out + b3V out

[0115] wherein V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, and a and b represent fitting parameters.

[0116] In still another implementation, if the fitting function is implemented by using an exponential mapping relationship, the fitting function can be expressed as:

[0117]

[0118] wherein V out represents the test voltage value corresponding to the EVS pixel, I represents the interpolated gray value corresponding to the EVS pixel, t represents the exposure time of the APS pixel, and I0, V and k represent fitting parameters.

[0119] Further, the value of m is the natural constant e, that is, the fitting function can be further expressed as:

[0120]

[0121] Of course, in actual applications, the values of the constants such as i, j and m can be flexibly set according to application scenarios, and the present embodiment does not make a unique limitation in this regard. In the present embodiment, a preset fitting function is predefined, and the form of the fitting function is not limited to the above forms provided in the present embodiment, but can also be a Fourier expansion, etc.

[0122] In this embodiment, the test voltage value corresponding to the EVS pixel and its interpolated grayscale value are input as known quantities into the aforementioned fitting function based on exponential mapping relationship, etc. The unknown quantities a, b or I0, V, and k in the fitting function are solved to obtain the fitting parameters. Finally, the obtained solutions are substituted into the fitting function to obtain a complete fitting function I = f(V). out The fitting function can be used as a mapping relationship between voltage values ​​and grayscale values ​​acquired by dynamic event visual sensors or dual-modal fusion sensors.

[0123] In the technical solution disclosed in this embodiment, multiple sample data of test voltage values ​​and interpolated grayscale values ​​corresponding to EVS pixels in different sub-pixel regions within the reference pixel region are taken as known quantities and input into the preset fitting function with the above-mentioned exponential mapping relationship. The fitting parameters, which are unknown quantities in the preset fitting function, are solved. Finally, the obtained solution is substituted into the fitting function to obtain a fitting curve, which serves as the mapping relationship between the voltage values ​​and grayscale values ​​acquired by the dual-modal fusion sensor. In this way, for the real-time voltage values ​​acquired by EVS pixels within the imaging pixel region of the dual-modal fusion sensor, a corresponding and accurate mapped grayscale value can be found. This mapped grayscale value includes dynamic event factors, and a higher quality grayscale image can be obtained using this mapped grayscale value.

[0124] To better implement the grayscale mapping calibration method based on reference pixels in the embodiments of this application, this application also provides a grayscale mapping calibration device based on reference pixels, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of an embodiment of a grayscale mapping calibration device based on reference pixels. The grayscale mapping calibration device based on reference pixels includes the following modules 701 to 704:

[0125] The acquisition module 701 is used to acquire a calibration image file collected by the dual-modal fusion sensor. The calibration image file includes the test voltage value of the EVS pixel and the test grayscale value of the APS pixel in the reference pixel area of ​​the dual-modal fusion sensor. The reference pixel area includes multiple sub-pixel areas. Each sub-pixel area includes at least one APS pixel and at least one EVS pixel. The pixel aperture size of the APS pixel and the EVS pixel in the sub-pixel area is the same, and the pixel aperture size of the corresponding sub-pixel areas is different.

[0126] Interpolation module 702 is used to obtain the interpolated gray value of the EVS pixel in each sub-pixel region based on the test gray value of the APS pixel in the sub-pixel region.

[0127] constructing, by a constructing module 703, a mapping relationship between voltage values and gray values according to the interpolation gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values of the EVS pixels in the plurality of sub-pixel regions;

[0128] calculating, by a calculating module 704, a mapping gray value of an EVS pixel in an imaging pixel region of the dual-modal fusion sensor according to the mapping relationship and a real-time voltage value of the EVS pixel in the imaging pixel region, wherein each pixel in the imaging pixel region has a same opening size.

[0129] In some embodiments of the present application, the obtaining module 701 is further configured to irradiate the reference pixel region by a preset light source.

[0130] controlling the APS pixels and the EVS pixels in the reference pixel region to simultaneously collect optical signals;

[0131] converting a first optical signal collected by the APS pixels in the reference pixel region into a test gray value of the APS pixels, and converting a second optical signal collected by the EVS pixels in the reference pixel region into a test voltage value of the EVS pixels.

[0132] In some embodiments of the present application, the obtaining module 701 is further configured to execute the step of controlling the APS pixels and the EVS pixels in the reference pixel region to simultaneously collect optical signals every interval of a preset time.

[0133] In some embodiments of the present application, the constructing module 703 is further configured to input the interpolation gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values of the EVS pixels in the plurality of sub-pixel regions into a preset fitting function, and calculate fitting parameters in the preset fitting function.

[0134] substituting the fitting parameters into the preset fitting function to obtain the mapping relationship.

[0135] In some embodiments of the present application, the interpolation module 702 is further configured to obtain distances between each APS pixel and each EVS pixel in the sub-pixel region.

[0136] determining a weight value corresponding to each APS pixel according to the distance corresponding to the APS pixel;

[0137] weighting the interpolation gray value according to the weight value corresponding to each APS pixel and the test gray value corresponding to the APS pixel.

[0138] In some embodiments of the present application, the reference-pixel-based grayscale mapping calibration device further comprises a generation module 705 configured to obtain an actual grayscale value of the APS pixel in the imaging pixel region acquired at the same time when the real-time voltage value is acquired;

[0139] According to the real-time grayscale value of the APS pixel and the mapped grayscale value of the EVS pixel, a target grayscale image is generated and output.

[0140] In the embodiments of the present application, the mapping relationship between the voltage value and the grayscale value acquired by the dual-modal fusion sensor is constructed by using a calibration file image file containing the test grayscale value of the APS pixel and the test voltage value of the EVS pixel in the reference pixel region. The reference pixel region includes a plurality of sub-pixel regions, each of which includes at least one APS pixel and at least one EVS pixel. For the same sub-pixel region, the pixel opening sizes are the same, and the test grayscale value of the APS pixel in the same sub-pixel region can be used to interpolate the interpolated grayscale value of the EVS pixel. Thus, the EVS pixels in each sub-pixel region correspond to the interpolated grayscale value and the test voltage value. Because the pixel opening sizes of the sub-pixel regions are different, the test grayscale value of the APS pixel and the test voltage value of the EVS pixel are different between the sub-pixel regions, and the EVS pixels in the sub-pixel regions correspond to different interpolated grayscale values and test voltage values. Thus, a plurality of different sample data of the test voltage value and the interpolated grayscale value can be obtained, and the mapping relationship between the voltage value and the grayscale value acquired by the dual-modal fusion sensor can be accurately constructed. In the application stage, the pixels in the imaging pixel region of the dual-modal fusion sensor are used to acquire images. The imaging pixel region is different from the calibration image region, the pixel opening sizes in the imaging pixel region are the same, and the pixel values have consistency in acquiring grayscale values, which can be used for imaging. The real-time voltage value acquired by the EVS pixel in the imaging pixel region can be converted to the real-time grayscale value of the EVS pixel in the imaging pixel region by using the mapping relationship. The real-time grayscale value contains the dynamic event factor, which can be used to fill the grayscale value vacancy of the EVS pixel in the grayscale image acquired by the APS pixel in the imaging pixel region of the dual-modal fusion sensor. Thus, a grayscale image with higher resolution, higher frame rate, and wider dynamic range can be generated, and the quality of the grayscale image output by the dual-modal fusion sensor is improved.

[0141] The present application also provides a dual-modal fusion sensor, as shown in Figure 8 Figure 8 is a schematic structural diagram of an embodiment of the dual-modal fusion sensor provided in the present application.

[0142] ​The dual-mode fusion sensor integrates any one of the reference pixel-based gray mapping calibration apparatuses provided by the embodiments of the present application, and the dual-mode fusion sensor comprises:

[0143] one or more processors;

[0144] a memory; and

[0145] one or more application programs, wherein the one or more application programs are stored in the memory and are configured to perform the steps of the reference pixel-based gray mapping calibration method in any one of the reference pixel-based gray mapping calibration method embodiments provided by the embodiments of the present application by the processor.

[0146] Specifically, the dual-mode fusion sensor can include a processor 801 with one or more processing cores, a memory 802 with one or more computer readable storage media, a power supply 803, an input unit 804, and the like. Those skilled in the art can understand that the dual-mode fusion sensor can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements. Among them: Figure 8 The dual-mode fusion sensor structure shown in the figure does not constitute a limitation on the dual-mode fusion sensor, and can include more or fewer components than those shown in the figure, or combine certain components, or different component arrangements. Among them:

[0147] The processor 801 is the control center of the dual-mode fusion sensor, connects various parts of the dual-mode fusion sensor through various interfaces and lines, executes various functions of the dual-mode fusion sensor and processes data by running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, thereby overall monitoring the dual-mode fusion sensor. Optionally, the processor 801 can include one or more processing cores; preferably, the processor 801 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 801.

[0148] The memory 802 can be used to store software programs and modules, and the processor 801 can execute various functions and data processing by running the software programs and modules stored in the memory 802. The memory 802 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the dual-mode fusion sensor, etc. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 802 can also include a memory controller to provide the processor 801 with access to the memory 802.

[0149] The dual-mode fusion sensor further includes a power supply 803 for powering various components. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so as to realize functions such as charge management, discharge management, and power consumption management through the power management system. The power supply 803 can also include one or more direct current or alternating current power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

[0150] The dual-mode fusion sensor can further include an input unit 804, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0151] Although not shown, the dual-mode fusion sensor can also include a display unit, etc., which will not be described here. Specifically, in the present embodiment, the processor 801 in the dual-mode fusion sensor will load executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and run the application programs stored in the memory 802 by the processor 801, so as to realize various functions, as follows:

[0152] An image file collected by the dual-mode fusion sensor is obtained, the image file includes test voltage values of EVS pixels in a reference pixel area of the dual-mode fusion sensor, and test gray values of APS pixels, wherein the reference pixel area includes a plurality of sub-pixel areas, the sub-pixel area includes at least one APS pixel and at least one EVS pixel, the pixel opening size of the APS pixel and the EVS pixel in the sub-pixel area is the same, and the corresponding pixel opening size between the sub-pixel areas is not the same;

[0153] For each of the sub-pixel regions, an interpolation gray value of the EVS pixel in the sub-pixel region is obtained according to a test gray value of the APS pixel in the sub-pixel region;

[0154] According to the interpolation gray values of the EVS pixels of the plurality of sub-pixel regions and test voltage values thereof, a mapping relationship between voltage values and gray values collected by the dual-mode fusion sensor is constructed;

[0155] According to the mapping relationship and real-time voltage values of the EVS pixels in an imaging pixel region of the dual-mode fusion sensor, a mapping gray value of the EVS pixel in the imaging pixel region is calculated, wherein the opening sizes of the pixels in the imaging pixel region are consistent.

[0156] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by related hardware controlled by the instructions, which can be stored in a computer readable storage medium and loaded and executed by a processor.

[0157] To this end, the embodiment of the present application provides a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc. A computer program is stored on the computer readable storage medium, and the computer program is loaded by a processor to execute the steps in any of the gray mapping calibration methods based on reference pixels provided by the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0158] An image file collected by a dual-mode fusion sensor is obtained, the image file includes test voltage values of EVS pixels in a reference pixel region of the dual-mode fusion sensor, and test gray values of APS pixels, wherein the reference pixel region includes a plurality of sub-pixel regions, the sub-pixel regions include at least one APS pixel and at least one EVS pixel, the pixel opening sizes of the APS pixels and the EVS pixels in the sub-pixel regions are the same, and the corresponding pixel opening sizes between the sub-pixel regions are different;

[0159] For each of the sub-pixel regions, an interpolation gray value of the EVS pixel in the sub-pixel region is obtained according to a test gray value of the APS pixel in the sub-pixel region;

[0160] According to the interpolation gray values of the EVS pixels of the plurality of sub-pixel regions and test voltage values thereof, a mapping relationship between voltage values and gray values collected by the dual-mode fusion sensor is constructed;

[0161] According to the mapping relationship and the real-time voltage value collected by the EVS pixel in the imaging pixel region of the dual-mode fusion sensor, a mapping gray value of the EVS pixel in the imaging pixel region is calculated, wherein each pixel in the imaging pixel region has a consistent opening size.

[0162] In the above embodiments, the description of each embodiment has its own focus. The parts not described in detail in a certain embodiment can be referred to the detailed description of other embodiments above, which will not be repeated here.

[0163] In the implementation, each unit or structure above can be implemented as an independent entity, or can be combined as the same or several entities. The specific implementation of each unit or structure can be referred to the method embodiments above, which will not be repeated here.

[0164] The specific implementation of each operation above can be referred to the embodiments above, which will not be repeated here.

[0165] The above describes in detail a gray mapping calibration method based on reference pixels provided by the embodiments of the present application. The principle and implementation manner of the present application are described by applying specific examples. The above embodiment is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, the specific implementation manner and application range of the present application will be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as the limitation of the present application.

Claims

1. A reference pixel based gray scale mapping calibration method, characterized by, The method comprises: acquiring a calibration image file collected by a dual-mode fusion sensor, the calibration image file comprising test voltage values of EVS pixels in a reference pixel region of the dual-mode fusion sensor and test gray values of APS pixels, wherein the reference pixel region comprises a plurality of sub-pixel regions, each sub-pixel region comprising at least one APS pixel and at least one EVS pixel, the pixel opening sizes of the APS pixels and the EVS pixels in each sub-pixel region being the same, and the corresponding pixel opening sizes between the sub-pixel regions being different; for each sub-pixel region, obtaining an interpolated gray value of the EVS pixels in the sub-pixel region according to the test gray values of the APS pixels in the sub-pixel region; constructing a mapping relationship between voltage values and gray values collected by the dual-mode fusion sensor according to the interpolated gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values thereof; calculating a mapped gray value of the EVS pixels in an imaging pixel region of the dual-mode fusion sensor according to the mapping relationship and real-time voltage values of the EVS pixels collected in the imaging pixel region, wherein the pixel opening sizes in the imaging pixel region are consistent.

2. The reference pixel based gray scale mapping calibration method of claim 1, wherein, Before the step of acquiring the calibration image file collected by the dual-mode fusion sensor, the method further comprises: irradiating the reference pixel region by a preset light source; controlling the APS pixels and the EVS pixels in the reference pixel region to simultaneously collect optical signals; converting a first optical signal collected by the APS pixels in the reference pixel region into a test gray value of the APS pixels and converting a second optical signal collected by the EVS pixels in the reference pixel region into a test voltage value of the EVS pixels.

3. The reference pixel based gray scale mapping calibration method of claim 2, wherein, The method further comprises: performing the step of controlling the APS pixels and the EVS pixels in the reference pixel region to simultaneously collect optical signals every preset time interval.

4. The reference pixel based gray scale mapping calibration method of claim 1, wherein, The step of constructing the mapping relationship between voltage values and gray values collected by the dual-mode fusion sensor according to the interpolated gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values thereof comprises: inputting the interpolated gray values of the EVS pixels in the plurality of sub-pixel regions and the test voltage values thereof into a preset fitting function to calculate fitting parameters in the preset fitting function; substituting the fitting parameters into the preset fitting function to obtain the mapping relationship.

5. The reference pixel based gray scale mapping calibration method of claim 1, wherein, The step of obtaining, for each sub-pixel region, an interpolated gray value of the EVS pixels in the sub-pixel region according to test gray values of the APS pixels in the sub-pixel region further comprises: acquiring distances between the APS pixels and the EVS pixels in the sub-pixel region; determining weight values corresponding to the APS pixels according to the distances corresponding to the APS pixels; performing weighted operation to obtain the interpolated gray value according to the weight values corresponding to the APS pixels and the test gray values corresponding to the APS pixels.

6. The reference pixel based gray scale mapping calibration method of claim 1, wherein, The method further comprises: acquiring actual gray scale values of the APS pixels in the imaging pixel region at the same time when the real-time voltage values are acquired; generating and outputting a target gray scale image according to the real-time gray scale values of the APS pixels and the mapped gray scale values of the EVS pixels.

7. A reference pixel-based gray scale mapping calibration apparatus, characterized by, The gray scale mapping calibration device based on reference pixels comprises: an acquisition module configured to acquire a calibration image file acquired by a dual-modal fusion sensor, the calibration image file comprising test voltage values of EVS pixels in a reference pixel region of the dual-modal fusion sensor and test gray scale values of APS pixels, wherein the reference pixel region comprises a plurality of sub-pixel regions, each of the sub-pixel regions comprises at least one APS pixel and at least one EVS pixel, the APS pixels and the EVS pixels in each of the sub-pixel regions have the same pixel opening size, and the corresponding pixel opening sizes of the sub-pixel regions are different from each other; an interpolation module configured to obtain, for each of the sub-pixel regions, an interpolated gray scale value of the EVS pixels in the sub-pixel region according to the test gray scale values of the APS pixels in the sub-pixel region; a construction module configured to construct a mapping relationship between voltage values and gray scale values acquired by the dual-modal fusion sensor according to the interpolated gray scale values of the EVS pixels and the test voltage values of the plurality of sub-pixel regions; a calculation module configured to calculate mapped gray scale values of EVS pixels in an imaging pixel region of the dual-modal fusion sensor according to the mapping relationship and real-time voltage values of the EVS pixels in the imaging pixel region.

8. A dual modality fusion sensor characterized by, The dual-modal fusion sensor comprises: one or more processors; a memory; and one or more application programs stored in the memory and configured to be executed by the processors to implement the steps of the gray scale mapping calibration method based on reference pixels according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer program, and the computer program is loaded by a processor to execute the steps of the gray scale mapping calibration method based on reference pixels according to any one of claims 1 to 7.

Images