Bimodal fusion sensor calibration method, device, equipment and medium

Abstract

The invention provides a bimodal fusion sensor calibration method and device, equipment and a medium, and the specific implementation scheme is as follows: obtaining a single original image file output by a bimodal fusion sensor for photographing a to-be-photographed calibration board, the first gray values of the APS pixels and the readout voltage values of the EVS pixels corresponding to different pixel array areas in the original image file are different; for each pixel array area, calculating a second gray value corresponding to the EVS pixel based on the first gray value; constructing a gray value mapping model based on a plurality of sample points formed by different second gray values and corresponding read voltage values; and all EVS pixels are calibrated based on the gray value mapping model. Based on this, the calibrated bimodal fusion sensor can calculate the simulation gray value of the EVS pixel, and the resolution, the frame rate and the photosensitive dynamic range of the gray image are improved.

Description

Technical Field

[0001] The present application relates to the technical field of image sensors, and particularly to the technical field of sensor calibration, and can be applied to the scenario of dual-modal fusion sensor calibration. More specifically, the present application discloses a method, device, equipment and medium for dual-modal fusion sensor calibration. Background Art

[0002] An event monitoring vision sensor (EVS) is a new type of sensor that mimics the human retina and generates event signals in response to pixel point pulses of brightness changes caused by motion. Therefore, it can capture the brightness changes of a scene at an extremely high frame rate, record events at specific time points and specific positions in an image, and form an event stream instead of a frame stream, thus solving problems such as information redundancy, large data storage, and real-time processing of traditional image sensors, and having a relatively low working power consumption.

[0003] Currently, although the event monitoring vision sensor has solved problems such as high power consumption, low frame rate, and poor dynamic range of active pixel sensors (APS), it only retains motion edge information in imaging and loses rich detail information of objects. Based on this, a dual-modal fusion sensor has been proposed in the related art to integrate the functions of APS pixels and EVS pixels. However, when using the dual-modal fusion sensor to output a grayscale image currently, only the grayscale image corresponding to the APS pixels can be output, and the information collected by the EVS pixels does not participate in the output of the grayscale image, resulting in a relatively low resolution of the grayscale image output by the dual-modal fusion sensor.

[0004] It should be noted that the technologies described in this part are not necessarily technologies that have been previously envisioned or adopted. Unless otherwise specified, any technology described in this part should not be considered as prior art solely because it is included in this part. Similarly, unless otherwise specified, the problems mentioned in this part should not be considered as having been recognized in any prior art. Summary of the Invention

[0005] The present application provides a method, device, equipment and medium for dual-modal fusion sensor calibration, which can at least solve the problems of relatively low resolution, narrow photosensitive dynamic range, and low frame rate of the grayscale image caused by the fact that the dual-modal fusion sensor provided in the related art can only output a grayscale image based on the information collected by the APS pixels.

[0006] The first aspect of the present application provides a calibration method for a dual-modal fusion sensor. The overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels. The calibration method for the dual-modal fusion sensor includes: obtaining a single raw image file output by the dual-modal fusion sensor when photographing a calibration board to be photographed at a test working moment; wherein, the raw image file includes a first gray value corresponding to the APS pixels and a readout voltage value corresponding to the EVS pixels, and the first gray value and the readout voltage value corresponding to different pixel array regions in the raw image file are different; for each of the pixel array regions, respectively calculating a second gray value corresponding to the EVS pixels based on the first gray value of the APS pixels; constructing a gray value mapping model based on a plurality of sample points composed of different second gray values and the corresponding readout voltage values; and calibrating all the EVS pixels in the dual-modal fusion sensor based on the gray value mapping model.

[0007] The second aspect of the present application provides a calibration device for a dual-modal fusion sensor, including: a file acquisition module, configured to obtain a single raw image file output by the dual-modal fusion sensor when photographing a calibration board to be photographed at a test working moment; wherein, the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, the raw image file includes a first gray value corresponding to the APS pixels and a readout voltage value corresponding to the EVS pixels, and the first gray value and the readout voltage value corresponding to different pixel array regions in the raw image file are different; a gray value calculation module, configured to, for each of the pixel array regions, respectively calculate a second gray value corresponding to the EVS pixels based on the first gray value of the APS pixels; a model construction module, configured to construct a gray value mapping model based on a plurality of sample points composed of different second gray values and the corresponding readout voltage values; and a sensor calibration module, configured to calibrate all the EVS pixels in the dual-modal fusion sensor based on the gray value mapping model.

[0008] The third aspect of the present application provides an electronic device, including: a memory and a processor. The processor is configured to execute a computer program stored in the memory. When the processor executes the computer program, it implements each step in the calibration method for the dual-modal fusion sensor provided in the first aspect of the present application.

[0009] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements each step in the calibration method for the dual-modal fusion sensor provided in the first aspect of the embodiments of the present application.

[0010] As can be seen from the above, according to the dual-modal fusion sensor calibration method, device, equipment and medium provided by the solution of the present application, a single original image file output by the dual-modal fusion sensor when photographing a calibration board to be photographed at the test working moment is obtained. Among them, the original image file includes the first gray value corresponding to the APS pixel and the readout voltage value corresponding to the EVS pixel, and the first gray value and the readout voltage value corresponding to different pixel array regions in the original image file are different; for each pixel array region, the second gray value corresponding to the EVS pixel is calculated respectively based on the first gray value of the APS pixel; a gray value mapping model is constructed based on a plurality of sample points composed of different second gray values and the corresponding readout voltage values; all EVS pixels in the dual-modal fusion sensor are calibrated based on the gray value mapping model. Through the implementation of the solution of the present application, a calibration file is obtained based on the gray value of the APS pixel and the readout voltage value of the EVS pixel, so that the calibrated dual-modal fusion sensor can calculate the analog gray value of the EVS pixel, and then a gray image can be generated by combining the gray values of all pixels, effectively improving the resolution, frame rate and photosensitive dynamic range of the gray image.

[0011] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The drawings exemplarily show embodiments and constitute a part of the specification, and are used together with the written description of the specification to explain the exemplary embodiments. The shown drawings are only for exemplary purposes and do not limit the scope of the claims. In all the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0013] Figure 1 It is a schematic diagram of the array of the overall pixel array of the dual-modal fusion sensor provided by an embodiment of the present application;

[0014] Figure 2 It is a schematic diagram of the basic process of the dual-modal fusion sensor calibration method provided by an embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of the partition of a calibration board to be photographed provided by an embodiment of the present application;

[0016] Figure 4 It is a schematic diagram of a fitting curve provided by an embodiment of the present application;

[0017] Figure 5 It is a schematic diagram of the refined process of the dual-modal fusion sensor calibration method provided by an embodiment of the present application;

[0018] Figure 6 Schematic diagram of program modules of a dual - mode fusion sensor calibration device provided by an embodiment of the present application;

[0019] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0020] To make the invention objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present application.

[0021] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present application, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0022] A dual - mode fusion sensor calibration method, device, equipment, and medium in an embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.

[0023] To solve the problem that the grayscale image output by the dual - mode fusion sensor provided in the related technology based only on the information collected by APS pixels has a low resolution, an embodiment of the present application provides a dual - mode fusion sensor calibration method. The overall pixel array of the dual - mode fusion sensor includes APS pixels and EVS pixels. As Figure 1 shown is the array schematic diagram of the overall pixel array of the dual - mode fusion sensor provided in this embodiment. Figure 1 In it, A represents APS pixels, and A / E represents EVS pixels. It should be understood that Figure 1 the array size in it is only for illustrative purposes and should not be understood as the only limitation on the array size of the overall pixel array in actual applications. Moreover, the layout and quantity of the two types of pixels in the overall pixel array can be determined according to the actual application scenario, and this embodiment does not make a unique limitation. As Figure 2 shown is the basic flowchart of the dual - mode fusion sensor calibration method provided by an embodiment of the present application. The dual - mode fusion sensor calibration method specifically includes the following steps:

[0024] Step 201: Obtain a single original image file output by the dual-modal fusion sensor when photographing the calibration board to be photographed at the test working moment.

[0025] Specifically, the original image file in this embodiment includes the first gray value corresponding to the APS pixels and the readout voltage value corresponding to the EVS pixels. The first gray value and the readout voltage value corresponding to different pixel array regions in the original image file are different. The division of the pixel array regions in this embodiment can be flexibly defined according to the actual application scenario. The number of EVS pixels and APS pixels in each pixel array region is at least one. Please refer to Figure 1 , the above-mentioned overall pixel array in this embodiment is divided into 9 pixel array regions, and each pixel array region includes at least 2*2 pixels. It should be understood that the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, that is, the overall photosensitive area of the dual-modal fusion sensor is divided into photosensitive units of different pixel types, and different photosensitive units correspond to the APS data mode and the EVS data mode respectively. Compared with the multiple sensor modules separately arranged in the conventional implementation, the device volume of the sensor module is effectively compressed, which is more conducive to the miniaturization of the overall hardware architecture, and eliminates the parallax and image fusion matching problems between different sensors. In addition, the dual-modal fusion sensor outputs an original image file (i.e., a RAW file) during the test working stage. The RAW file refers to a binary format file in which the image sensor converts the captured light source signal into a digital signal. In practical applications, the EVS pixels and the APS pixels are exposed synchronously. There is a photodiode configured in the EVS pixels. The photodiode is integrated with a capacitor that accumulates charges, and it generates a photocurrent in response to the incident light intensity, and then generates a corresponding readout voltage value according to the photocurrent.

[0026] In an optional implementation manner of this embodiment, the specific implementation manner of obtaining a single original image file output by the dual-modal fusion sensor for the calibration board to be photographed at the test working moment includes but is not limited to the following two:

[0027] Method 1: Obtain a single original image file output after photographing the calibration board to be photographed with multiple partitions having different reflectivities at the test working moment of the dual-modal fusion sensor; wherein, different partitions on the calibration board to be photographed correspond to imaging of different pixel array regions on the overall pixel array.

[0028] Specifically, in this embodiment, different partitions on the same calibration board are configured with different reflectivities, and the value of the reflectivity can be any value between 0 and 100%. It should be understood that different reflectivities can be set in the form of a geometric sequence, a pseudo-random sequence, etc. When the dual-mode fusion sensor takes a picture of the calibration board to be photographed, different pixel array regions on the overall pixel array respectively sense different calibration board partitions. Since the reflectivity of each calibration board partition is different, the gray values of the APS pixels and the readout voltage values of the EVS pixels in different pixel array regions are also different. As Figure 3 shown is a schematic diagram of the partitions of a calibration board to be photographed provided in this embodiment. There are 9 partitions with different reflectivities Re on the calibration board, which can be imaged corresponding to Figure 1 the 9 pixel array regions in the overall pixel array described above. Thus, through a single shot of the calibration board by the sensor, multiple EVS gray data and voltage data distributed in a stepped manner can be obtained.

[0029] Method 2: Obtain a single raw image file output after the dual-mode fusion sensor simultaneously takes pictures of multiple calibration boards to be photographed with different relative distances and arranged in a staggered manner at the test working moment; wherein, different calibration boards to be photographed correspond to imaging with different pixel array regions on the overall pixel array.

[0030] Specifically, in this embodiment, multiple calibration boards with different relative distances and arranged in a staggered manner can also be set in the shooting area of the dual-mode fusion sensor. The set distances of different calibration boards can be set according to the rules of , or can also be distances in other sequences, such as a geometric sequence, an arithmetic sequence, a pseudo-random sequence, etc. The staggered arrangement means that the calibration boards do not block each other in the imaging area of the sensor, ensuring that different pixel array regions in the overall pixel array of the sensor can respectively collect the reflected light of different calibration boards. Similar to Method 1, through a single shot of multiple calibration boards by the sensor, multiple stepped gray data can be obtained.

[0031] Step 202: For each pixel array region, calculate the second gray value corresponding to the EVS pixel based on the first gray value of the APS pixel respectively.

[0032] Specifically, in practical applications, the APS pixel can directly output the gray value, while the EVS pixel can only output the voltage value and cannot output the gray value. Each pixel array region in this embodiment includes EVS pixels and APS pixels. Then, for the EVS pixels in the pixel array region, the analog gray value of the EVS pixel can be calculated by referring to the gray value of the APS pixel in the neighborhood of the EVS pixel in the pixel array region. Please refer to Figure 1, for a pixel array region including 2×2 pixels, the second gray value of an EVS pixel can be calculated based on the first gray values of 3 adjacent APS pixels in each pixel array region.

[0033] More specifically, in this embodiment, the first gray values of the neighboring APS pixels in each pixel array region can be weighted and averaged to obtain the second gray value of the EVS pixel in this pixel array region. Of course, in this embodiment, different weights can be assigned to different APS pixels based on the relative distance between the APS pixel and the EVS pixel. For example, an APS pixel closer to the coordinate position of the EVS pixel can be assigned a larger weight, while an APS pixel farther from the coordinate position of the EVS pixel can be assigned a smaller weight.

[0034] Step 203: Construct a gray value mapping model based on multiple sample points composed of different second gray values and corresponding readout voltage values.

[0035] Specifically, each sample point in this embodiment is composed of a readout voltage value and a corresponding second gray value, that is, the representation form of each sample point can be (X: readout voltage value, Y: second gray value). Each sample point can be understood as a sample data for constructing the gray value mapping model, and multiple sample points form a sample set. It should be noted that after calculating the simulated gray values of the EVS pixels in different pixel array regions in this embodiment, the simulated gray value can be associated with the readout voltage value actually output by the EVS pixel to obtain the above sample points. In this embodiment, according to a single original image file output by the dual-mode fusion sensor at the same test working moment, multiple sample points corresponding to the EVS pixels in different pixel array regions can be obtained respectively, and then this embodiment combines the multiple sample points to construct a gray value mapping model.

[0036] In an alternative implementation manner of this embodiment, multiple sample points composed of multiple different second gray values and corresponding readout voltage values can be input into a preset theoretical mapping model to calculate the parameter values of the fitting parameters in the theoretical mapping model; the calculated parameter values are substituted into the theoretical mapping model to obtain the gray value mapping model.

[0037] Specifically, the theoretical mapping model can be expressed as: I = f(V out ), where V out represents the readout voltage value in the sample point, and I represents the second gray value in the sample point. For example Figure 4The figure shows a schematic diagram of a fitting curve according to this embodiment, illustrating the implementation principle of curve fitting for 6 sample points to be fitted. All the points in the figure are the coordinate points corresponding to the sample points. It should be understood that the mapping relationship of f(·) in the above theoretical mapping model of this embodiment can be in the form of exponential, linear, logarithmic, polynomial, etc., and this embodiment does not make a unique limitation on this.

[0038] Among them, in one implementation manner, if the above theoretical mapping model is implemented using a linear mapping relationship, the theoretical mapping model can be expressed as: I = a * V out + b, where V out represents the readout voltage value in the sample point, I represents the second gray value in the sample point, and a and b represent fitting parameters.

[0039] In another implementation manner, if the above theoretical mapping model is implemented using a logarithmic mapping relationship, the theoretical mapping model can be expressed as:

[0040] I = t * I 0 * log[(V - V out ) / k],

[0041] where V out represents the readout voltage value in the sample point, t represents the APS exposure time, I represents the second gray value in the sample point, I 0 , V, and k represent fitting parameters.

[0042] In yet another implementation manner, if the above theoretical mapping model is implemented using a polynomial mapping relationship, the theoretical mapping model can be expressed as:

[0043] I = a + b 1 V out + b 2 V out + b 3 V out ,

[0044] where V out represents the readout voltage value in the sample point, I represents the second gray value in the sample point, and a, b 1 , b 2 , b 3 are constants.

[0045] In still another implementation manner, if the above theoretical mapping model is implemented using an exponential mapping relationship, the theoretical mapping model can be expressed as:

[0046]

[0047] where V outV represents the readout voltage value in the sample point, I represents the second grayscale value in the sample point, t represents the APS exposure time, m is a constant, and I 0 , V, and k represent fitting parameters.

[0048] Further preferably, the value of the above-mentioned m is the natural constant e, that is, the above-mentioned theoretical mapping model can further be expressed as:

[0049]

[0050] Of course, in practical applications, the value of the above-mentioned constant m can be flexibly set according to the application scenario, and this embodiment does not make a unique limitation on this. In this embodiment, a theoretical mapping model is pre-customized. The form of this theoretical mapping model is not limited to the above form provided in this embodiment, and it can also be Fourier expansion, etc.

[0051] This embodiment can input the readout voltage values and the second grayscale values (i.e., the analog grayscale values) in multiple groups of sample points corresponding to different EVS pixels as known quantities into the above-mentioned theoretical mapping model based on the exponential mapping relationship, and solve the unknown quantities I 0 , V, and k in the theoretical mapping model. Finally, substitute the obtained solutions into the theoretical mapping model to obtain a fitting curve as the grayscale value mapping model.

[0052] Step 204: Calibrate all EVS pixels in the dual-mode fusion sensor based on the grayscale value mapping model.

[0053] Specifically, this embodiment generates a calibration file according to the grayscale value mapping model and sends the calibration file to the dual-mode fusion sensor for sensor calibration. After the calibration file takes effect on the dual-mode fusion sensor, the calibrated grayscale value mapping model can be used for the dual-mode fusion sensor to calculate the analog grayscale value corresponding to the readout voltage value of each EVS pixel in the original image file generated during actual working hours. That is, during the actual working process of the dual-mode fusion sensor, for the original image file generated at each moment, the readout voltage value of the EVS pixel can be input into the target grayscale value mapping model to obtain the corresponding analog grayscale value. Then, by combining the calculated analog grayscale values of all EVS pixels and the grayscale values of all APS pixels included in the original image file, a grayscale image of the entire pixel array can be generated, overcoming the grayscale loss caused by EVS pixels in the related art and improving the resolution of the grayscale image output by the dual-mode fusion sensor; in addition, it should also be understood that APS pixels have the defect of grayscale blur in high-dynamic motion scenes, while EVS pixels have the characteristic of high dynamic range. The grayscale image of this embodiment incorporates the perceptual data of EVS pixels, thereby enabling high-frame-rate and high-dynamic-range grayscale output.

[0054] In an alternative implementation of this embodiment, the specific implementation method for calibrating all EVS pixels in the dual-mode fusion sensor based on the gray value mapping model includes but is not limited to the following two methods:

[0055] Method 1: Uniformly calibrate all EVS pixels in the dual-mode fusion sensor based on the same gray value mapping model.

[0056] In one implementation, this embodiment obtains the gray value mapping model by combining different sample points of EVS pixels in different pixel array regions, and then makes the gray value mapping model take effect by calibration on all EVS pixels in the overall pixel array at the same time.

[0057] Method 2: Perform corresponding transformations on the gray value mapping model based on the array position attributes of different EVS pixels to obtain multiple different transformed gray value mapping models; based on the different transformed gray value mapping models, calibrate different EVS pixels in the dual-mode fusion sensor respectively.

[0058] In another implementation, this embodiment takes into account that the array positions of different EVS pixels on the overall pixel array are different. Affected by the sensor manufacturing process and the like, the photosensitive characteristics of EVS pixels at different positions will be different. If the same gray value mapping model is used, there will be a certain gray value calculation error. Based on this, this embodiment uses the obtained gray value mapping model as a reference model, and does not directly calibrate it to the dual-mode fusion sensor. Instead, it adaptively transforms the reference model according to the array position where different EVS pixels are located to obtain multiple transformed gray value mapping models corresponding to different EVS pixels, and then uses the transformed model as the final model to calibrate the corresponding EVS pixels respectively to improve the accuracy of gray value simulation of EVS pixels during the actual working process.

[0059] In an alternative implementation of this embodiment, the step of obtaining the gray value mapping model based on multiple sample points composed of different second gray values and corresponding readout voltage values includes: performing multiple samplings on the sample points composed of different second gray values and corresponding readout voltage values according to the sampling rules corresponding to the array position attributes of different EVS pixels to obtain multiple sample sets corresponding to different EVS pixels respectively; constructing corresponding gray value mapping models according to multiple sample points in each sample set respectively. Correspondingly, the step of calibrating all EVS pixels in the dual-mode fusion sensor based on the gray value mapping model includes: calibrating the corresponding EVS pixels in the dual-mode fusion sensor based on different gray value mapping models respectively.

[0060] Specifically, considering that a single grayscale value mapping model cannot accurately apply to EVS pixels at different array positions, in this embodiment, instead of directly combining the sample points of all EVS pixels in the overall pixel array to generate a single grayscale value mapping model, for the EVS pixels in each pixel array region, adaptive sampling is performed based on their array positions in the overall pixel array. For example, sample points of this EVS pixel and multiple other EVS pixels whose distance from this EVS pixel is less than a preset distance threshold are sampled. That is, a sample set composed of multiple sample points that are referenceable to the EVS pixels in each pixel array region is separately sampled. Then, a grayscale value mapping model is adaptively obtained based on the multiple sample points sampled for each EVS pixel, and finally, the EVS pixels at different array positions are differentially calibrated.

[0061] In another alternative implementation manner of this embodiment, the step of constructing a grayscale value mapping model based on multiple sample points composed of different second grayscale values and corresponding readout voltage values includes: performing multiple random samplings on the original sample set composed of different second grayscale values and corresponding readout voltage values to obtain multiple random sub-sample sets; respectively constructing corresponding grayscale value mapping models according to the multiple sample points in each sub-sample set; and determining the final grayscale value mapping model based on all the grayscale value mapping models.

[0062] Specifically, in this embodiment, multiple sample points in each sub-sample set are respectively input into a preset theoretical mapping model, and the values of the fitting parameters in the theoretical mapping model are calculated; the calculated parameter values are substituted into the theoretical mapping model to obtain a grayscale value mapping model. The theoretical mapping model is expressed as:

[0063] I = f(V out );

[0064] where, V out represents the readout voltage value in the sample point, and I represents the second grayscale value in the parameter pair;

[0065] In this embodiment, the original sample set is randomly sampled and combined to perform multiple sub-sample set fitting attempts, and multiple fitting curves can be obtained. The fitting objects (i.e., sample points) of different fitting curves are different, and the obtained parameter pairs are also different. Eventually, there are also certain differences in the fitting effects. The quality of the fitting effect is mainly evaluated by the number of fly points. Fly points can be understood as sample points with a relatively low approximation degree to the fitting curve, that is, sample points with a relatively large distance from the fitting curve. Fly points are mainly unreasonable sampling points caused by errors. Based on this, in this embodiment, for each gray value mapping model, an evaluation index of the approximation degree of the coordinate positions of all the sample points it fits relative to the gray value mapping model is obtained respectively; the coordinate points with the approximation degree evaluation index lower than the preset index threshold are determined as fly points; referring to the gray value mapping model with the least number of fly points among all gray value mapping models, the final gray value mapping model is determined.

[0066] In this embodiment, the coordinate points with the approximation degree lower than the preset threshold are determined as fly points. Thus, the number of fly points of multiple fitting curves corresponding to different sub-sample sets can be determined respectively. Finally, the gray value mapping model with the lowest number of fly points is used as the preliminary gray value mapping model, and then the final gray value mapping model is further determined. It should be understood that in this embodiment, the gray value mapping model with the lowest number of fly points can be directly determined as the final gray value mapping model, or the final gray value mapping model can be indirectly obtained through the gray value mapping model with the lowest number of fly points.

[0067] In an alternative embodiment of this embodiment, the step of referring to the gray value mapping model with the least number of fly points among all gray value mapping models to determine the final gray value mapping model includes: determining the gray value mapping model with the least number of fly points among all gray value mapping models as the preliminary gray value mapping model; screening out the sample points corresponding to the fly points from the target sample set correspondingly fitted by the preliminary gray value mapping model; and determining the corresponding final gray value mapping model according to the remaining multiple groups of sample points in the target sample set.

[0068] Specifically, in this embodiment, among the multiple gray value mapping models obtained by preliminary fitting, the gray value mapping model with the least number of flying points is only a relatively optimal gray value mapping model, not an absolutely optimal gray value mapping model. In order to further improve the accuracy of the final gray value mapping model in this embodiment, the gray value mapping model with the least number of flying points is used as the preliminary gray value mapping model. Then, the sample points corresponding to the flying points in this preliminary gray value mapping model are screened out from the sub-sample set previously fitted by this preliminary gray value mapping model. Then, the sub-sample set after screening out the flying points is substituted into the aforementioned theoretical mapping model again to calculate the unknown fitting parameters, and the calculated values of the fitting parameters are substituted back into the unknown fitting parameters in the theoretical mapping model, that is, the final gray value mapping model is obtained. Thus, the finally calculated gray value mapping model has as few flying points as possible or even no flying points, greatly improving the effectiveness of the model.

[0069] Next, this embodiment also provides a refined calibration method for a dual-modal fusion sensor, as Figure 5 shown in the schematic flowchart of the refinement of a calibration method for a dual-modal fusion sensor provided by an embodiment of the present application, which specifically includes the following steps:

[0070] Step 501: Obtain a single original image file output after the dual-modal fusion sensor takes pictures of a calibration board with multiple different reflectivity partitions at the test working moment;

[0071] Step 502: For each pixel array region in the overall pixel array of the dual-modal fusion sensor, obtain the first gray value of the APS pixel from the original image file and calculate the second gray value corresponding to the EVS pixel;

[0072] Step 503: According to the sampling rules corresponding to the array position attributes of different EVS pixels, collect all the data composed of the second gray values and readout voltage values of different EVS pixels, and respectively obtain sample points corresponding to different EVS pixels;

[0073] Step 504: Input the data of multiple sample points into a preset theoretical mapping model respectively, and calculate the parameter values of the unknown fitting parameters in the theoretical mapping model;

[0074] Step 505: Substitute the calculated parameter values into the theoretical mapping model to obtain a gray value mapping model;

[0075] Step 506: Calibrate the EVS pixels at the corresponding array positions in the dual-modal fusion sensor based on different gray value mapping models respectively.

[0076] It should be understood that the sequence numbers of the steps in this embodiment do not indicate the sequence of execution of the steps. The execution sequence of each step should be determined according to its function and internal logic, and should not uniquely limit the implementation process of the embodiments of this application.

[0077] Figure 6 A dual-modal fusion sensor calibration device provided by an embodiment of this application can be used to implement the dual-modal fusion sensor calibration method in the foregoing embodiment. The dual-modal fusion sensor calibration device mainly includes:

[0078] A file acquisition module 601, configured to acquire a single original image file output by the dual-modal fusion sensor when photographing a calibration board to be photographed at the test working moment; wherein, the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, and the original image file includes the first gray value corresponding to the APS pixels and the readout voltage value corresponding to the EVS pixels. The first gray value and the readout voltage value corresponding to different pixel array regions in the original image file are different;

[0079] A gray value calculation module 602, configured to calculate, for each pixel array region, a second gray value corresponding to the EVS pixels respectively based on the first gray value of the APS pixels;

[0080] A model construction module 603, configured to construct a gray value mapping model based on a plurality of sample points composed of different second gray values and corresponding readout voltage values;

[0081] A sensor calibration module 604, configured to calibrate all EVS pixels in the dual-modal fusion sensor based on the gray value mapping model.

[0082] In some implementation manners of this embodiment, the file acquisition module is specifically configured to: acquire a single original image file output after the dual-modal fusion sensor photographs a calibration board to be photographed at the test working moment, where the calibration board to be photographed has multiple partitions with different reflectivities, and different partitions on the calibration board to be photographed correspond to imaging of different pixel array regions on the overall pixel array; or, acquire a single original image file output after the dual-modal fusion sensor simultaneously photographs a plurality of calibration boards to be photographed with different relative distances and arranged in a staggered manner at the test working moment, where different calibration boards to be photographed correspond to imaging of different pixel array regions on the overall pixel array.

[0083] In some embodiments of this embodiment, the sensor calibration module is specifically configured to: uniformly calibrate all EVS pixels in the dual-modal fusion sensor based on the same gray value mapping model; or, perform corresponding transformations on the gray value mapping model based on the array position attributes of different EVS pixels, and correspondingly obtain multiple different transformed gray value mapping models; based on the different transformed gray value mapping models, calibrate different EVS pixels in the dual-modal fusion sensor respectively.

[0084] In some embodiments of this embodiment, the model acquisition module is specifically configured to: according to the sampling rules corresponding to the array position attributes of different EVS pixels, obtain multiple sample points composed of different second gray values and corresponding readout voltage values, and obtain a sample set composed of sample points corresponding to different EVS pixels; construct a corresponding gray value mapping model according to different sample points in the sample set. Correspondingly, the sensor calibration module is specifically configured to: calibrate the corresponding EVS pixels in the dual-modal fusion sensor based on different gray value mapping models respectively.

[0085] In some embodiments of this embodiment, the model acquisition module is specifically configured to: perform multiple random samplings on multiple sample points composed of different second gray values and corresponding readout voltage values to obtain multiple sub-sample sets; construct corresponding gray value mapping models according to the sample points in each sub-sample set respectively; determine the final gray value mapping model based on all gray value mapping models.

[0086] In some embodiments of this embodiment, when the model acquisition module executes the function of determining the final gray value mapping model based on all gray value mapping models, it is specifically configured to: for each gray value mapping model, respectively obtain the approximation degree evaluation index of the coordinate positions of all sample points fitted by itself relative to the gray value mapping model; determine the outlier points with the approximation degree evaluation index lower than the preset index threshold; refer to the gray value mapping model with the least number of outlier points among all gray value mapping models to determine the final gray value mapping model.

[0087] It should be noted that the dual-modal fusion sensor calibration methods in the foregoing embodiments can all be implemented based on the dual-modal fusion sensor calibration device provided in this embodiment. Those of ordinary skill in the art can clearly understand that for the convenience and brevity of description, the specific working process of the dual-modal fusion sensor calibration device described in this embodiment can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0088] Based on the technical solution of the foregoing embodiment of the present application, a single original image file output by the dual-modal fusion sensor when photographing a calibration board to be photographed at the test working moment is obtained. The original image file includes the first gray value corresponding to the APS pixel and the readout voltage value corresponding to the EVS pixel. The first gray value and the readout voltage value corresponding to different pixel array regions in the original image file are different. For each pixel array region, the second gray value corresponding to the EVS pixel is calculated respectively based on the first gray value of the APS pixel. A gray value mapping model is constructed based on multiple sample points composed of different second gray values and the corresponding readout voltage values. All EVS pixels in the dual-modal fusion sensor are calibrated based on the gray value mapping model. Through the implementation of the solution of the present application, the calibrated dual-modal fusion sensor can calculate the analog gray value of the EVS pixel, and then can generate a gray image by combining the gray values of all pixels, effectively improving the resolution, frame rate and photosensitive dynamic range of the gray image. In addition, the gray value mapping model required for calibration can be generated based on the original image file output by a single photograph, effectively improving the calibration efficiency.

[0089] Figure 7 An electronic device provided in an embodiment of the present application can be used to implement the dual-modal fusion sensor calibration method in the foregoing embodiment, and mainly includes:

[0090] A memory 701, a processor 702, and a computer program 703 stored on the memory 701 and executable on the processor 702. The memory 701 and the processor 702 are communicatively connected. When the processor 702 executes the computer program 703, the dual-modal fusion sensor calibration method in the foregoing embodiment is implemented. The number of processors can be one or more.

[0091] It should be noted that the memory can be an internal storage unit, such as a hard disk or a memory; the memory can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory can also include both an internal storage unit and an external storage device, and the memory can also be used to temporarily store data that has been output or will be output. It should be noted that when the processor is a neural network chip, the electronic device may not include a memory. Whether the electronic device needs to use the memory to store the corresponding computer program depends on the type of the processor.

[0092] In addition, the processor may be a Central Processing Unit (CPU), and it may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), neural network chips, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0093] An embodiment of the present application further provides a computer-readable storage medium. This computer-readable storage medium may be disposed in the aforementioned electronic device, and this computer-readable storage medium may be the Figure 7 memory in the illustrated embodiment above.

[0094] A computer program is stored on this computer-readable storage medium. When the computer program is executed by the processor, the procedures of the aforementioned dual-modal fusion sensor calibration method can be implemented. Further, this computer-readable storage medium may also be various media such as USB flash drives, external hard drives, Read-Only Memories (ROMs), RAMs, magnetic disks, or optical discs that can store program codes.

[0095] It should be noted that the devices and methods disclosed in several embodiments provided in the present application can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other may be through some interfaces. The indirect couplings or communication connections of devices or modules can be in electrical, mechanical, or other forms.

[0096] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0097] In addition, in each embodiment of the present application, each functional module can be integrated into a processing module, or each module can exist physically alone, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module.

[0098] If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a readable storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned readable storage medium includes: various media such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical discs that can store program codes.

[0099] It should be noted that for the foregoing method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0100] In the above embodiments, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] The above is the description of the dual-modal fusion sensor calibration method, device, equipment, and medium provided by the present application. For those skilled in the art, according to the idea of the embodiments of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A calibration method for a dual - mode fusion sensor, characterized in that, the overall pixel array of the dual - mode fusion sensor includes APS pixels and EVS pixels, and the calibration method for the dual - mode fusion sensor includes: Obtaining a single original image file output by the dual - mode fusion sensor when photographing a calibration board to be photographed at the test working moment; wherein, the original image file includes the first gray - scale value corresponding to the APS pixels and the read - out voltage value corresponding to the EVS pixels, and the first gray - scale value and the read - out voltage value corresponding to different pixel array regions in the original image file are different; For each of the pixel array regions, respectively calculating a second gray - scale value corresponding to the EVS pixels based on the first gray - scale value of the APS pixels; Constructing a gray - scale value mapping model based on multiple sample points composed of different second gray - scale values and the corresponding read - out voltage values; Calibrating all the EVS pixels in the dual - mode fusion sensor based on the gray - scale value mapping model.

2. The calibration method for a dual - mode fusion sensor according to claim 1, characterized in that, the step of obtaining a single original image file output by the dual - mode fusion sensor when photographing a calibration board to be photographed at the test working moment includes: Obtaining a single original image file output after the dual - mode fusion sensor photographs a calibration board to be photographed at the test working moment, where the calibration board to be photographed has multiple partitions with different reflectivities; wherein, different partitions on the calibration board to be photographed correspond to different pixel array regions on the overall pixel array; Or, obtaining a single original image file output after the dual - mode fusion sensor simultaneously photographs multiple calibration boards to be photographed with different relative distances and misaligned arrangements at the test working moment; wherein, different calibration boards to be photographed correspond to different pixel array regions on the overall pixel array.

3. The calibration method for a dual - mode fusion sensor according to claim 1, characterized in that, the step of calibrating all the EVS pixels in the dual - mode fusion sensor based on the gray - scale value mapping model includes: Uniformly calibrating all the EVS pixels in the dual - mode fusion sensor based on the same gray - scale value mapping model; Or, performing corresponding transformations on the gray - scale value mapping model based on the array position attributes of different EVS pixels, and correspondingly obtaining multiple different transformed gray - scale value mapping models; Calibrating different EVS pixels in the dual - mode fusion sensor respectively based on different transformed gray - scale value mapping models.

4. The calibration method for a dual - mode fusion sensor according to claim 1, characterized in that, the step of constructing a gray - scale value mapping model based on multiple sample points composed of different second gray - scale values and the corresponding read - out voltage values includes: Sampling multiple sample points composed of different second gray - scale values and the corresponding read - out voltage values according to the sampling rules corresponding to the array position attributes of different EVS pixels, and respectively obtaining multiple sample sets corresponding to different EVS pixels; Construct corresponding gray value mapping models respectively according to multiple sample points in each of the sample sets; The step of calibrating all the EVS pixels in the dual - mode fusion sensor based on the gray value mapping model includes: Calibrate the corresponding EVS pixels in the dual - mode fusion sensor respectively based on different gray value mapping models.

5. The dual - mode fusion sensor calibration method according to claim 1, wherein, The step of constructing a gray value mapping model based on multiple sample points composed of different second gray values and corresponding read - out voltage values includes: Perform multiple random samplings on multiple sample points composed of different second gray values and corresponding read - out voltage values to obtain multiple sample sets; Construct corresponding gray value mapping models respectively according to multiple sample points in each of the sample sets; Determine the final gray value mapping model based on all the gray value mapping models.

6. The dual - mode fusion sensor calibration method according to claim 4 or 5, wherein, The step of constructing corresponding gray value mapping models respectively according to multiple sample points in each of the sample sets includes: Input multiple random sample points in each of the sample sets into a preset theoretical mapping model respectively, and calculate the values of the fitting parameters in the theoretical mapping model; the theoretical mapping model is expressed as: I = f(V out ); Among them, V out represents the readout voltage value in the sample point, and I represents the second gray value in the sample point; Substitute the calculated values of the parameters into the theoretical mapping model to obtain a gray value mapping model.

7. The dual - mode fusion sensor calibration method according to claim 5, wherein, The step of determining the final gray value mapping model based on all the gray value mapping models includes: For each gray value mapping model, respectively obtain the approximation degree evaluation index of the coordinate positions of all the sample points it fits relative to the gray value mapping model; Determine the outlier points as the coordinate points whose approximation degree evaluation index is lower than the preset index threshold; Refer to the gray value mapping model with the least number of outlier points among all the gray value mapping models to determine the final gray value mapping model.

8. A dual - mode fusion sensor calibration device, wherein, it includes: A file acquisition module, configured to acquire a single original image file output by the dual - mode fusion sensor when photographing a calibration board to be photographed at the test working moment; wherein, the overall pixel array of the dual - mode fusion sensor includes APS pixels and EVS pixels, the original image file includes the first gray value corresponding to the APS pixels and the read - out voltage value corresponding to the EVS pixels, and the first gray value and the read - out voltage value corresponding to different pixel array regions in the original image file are different; A gray value calculation module, configured to calculate the second gray value corresponding to the EVS pixels respectively based on the first gray value of the APS pixels for each pixel array region; A model construction module, configured to construct a gray value mapping model based on multiple sample points composed of different second gray values and corresponding read - out voltage values; A sensor calibration module, configured to calibrate all the EVS pixels in the dual-modal fusion sensor based on the gray value mapping model.

9. An electronic device, characterized in that it includes a memory and a processor, wherein: the processor is configured to execute a computer program stored on the memory; when the processor executes the computer program, the steps in the dual-modal sensor calibration method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the steps in the dual-modal fusion sensor calibration method according to any one of claims 1 to 7 are implemented.

Images