Sensor calibration method, device, equipment and storage medium
By calculating and calibrating the EVS pixel simulated grayscale value of the dual-mode fusion sensor, the problem of low resolution of the grayscale image of the dual-mode fusion sensor is solved, and image details and resolution are improved.
Patent Information
- Application Number
- CN202211654942.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-22
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-12-22
AI Technical Summary
The dual-mode fusion sensor can only output grayscale images based on the information collected by APS pixels, resulting in a lower resolution of the grayscale images.
By obtaining multiple original image files output from different test working hours of the dual-mode fusion sensor, the second gray value corresponding to the EVS pixel is calculated based on the gray value of the APS pixel in the target pixel array area where the EVS pixel is located, and The target grayscale value calculation model is obtained based on the voltage value and the second grayscale value, and a calibration file is generated, and sent to the dual-mode fusion sensor for calibration to calculate the simulated grayscale value of the EVS pixel.
The resolution and richness of image details of grayscale images are improved, allowing the dual-modal fusion sensor to generate grayscale images in combination with grayscale values of all pixels.
Smart Images

Figure CN118279401B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of image sensors, and in particular to a sensor calibration method, device, equipment and storage medium. Background Art
[0002] The event-based vision sensor (EVS) configured in the event camera is a new type of sensor that functions similarly to the human retina. It responds to pixel pulses of brightness changes caused by motion. EVS pixels detect changes in light intensity. When the light intensity change reaches a certain threshold, the pixel will output its coordinates in the image plane and be time-stamped by the synchronization system. These changes are called events, and each event contains four elements: pixel row coordinates, pixel column coordinates, the moment the event occurred, and the polarity of the light intensity change, that is, brightening or dimming.
[0003] At present, although event cameras have solved the problems of high power consumption, low frame rate and poor dynamic range of traditional cameras (APS, Active Pixel Sensor), they only retain the motion edge information in imaging and lose the rich detail information of the object. Based on this, a dual-modal fusion sensor is proposed in the related art to fuse the functions of APS pixels and EVS pixels. However, when using a dual-modal fusion sensor to output a grayscale image, only the grayscale image corresponding to the APS pixel can be output, and the information collected by the EVS pixel does not participate in the grayscale image output, resulting in a low resolution of the grayscale image output by the dual-modal fusion sensor. Summary of the invention
[0004] The embodiments of the present application provide a sensor calibration method, apparatus, device and storage medium, which can at least solve the problem of low resolution of grayscale images caused by the dual-modal fusion sensor provided in the related art being able to output grayscale images only based on information collected by APS pixels.
[0005] The first aspect of the embodiment of the present application provides a sensor calibration method, including: obtaining multiple original image files output by a dual-modal fusion sensor at different test working moments; wherein the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, and each of the original image files includes a first grayscale value corresponding to the APS pixel and a voltage value corresponding to the EVS pixel; for each of the original image files, respectively based on the first grayscale value of the APS pixel in the target pixel array area where each of the EVS pixels is located, calculate the second grayscale value corresponding to each of the EVS pixels; respectively obtain a corresponding target grayscale value calculation model according to multiple groups of mapping parameters of each of the EVS pixels; wherein the mapping parameters include the voltage value and the corresponding second grayscale value; generate a corresponding calibration file based on the target grayscale value calculation model, and send the calibration file to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor; wherein the target grayscale value calculation model is used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
[0006] According to a second aspect of an embodiment of the present application, there is provided a sensor calibration device, comprising: a first acquisition module, for acquiring a plurality of original image files outputted by a dual-modal fusion sensor at different test working moments; wherein the overall pixel array of the dual-modal fusion sensor comprises APS pixels and EVS pixels, and each of the original image files comprises a first grayscale value corresponding to the APS pixel and a voltage value corresponding to the EVS pixel; and a calculation module, for calculating, for each of the original image files, a first grayscale value of each of the EVS pixels based on the first grayscale value of the APS pixel in the target pixel array region where each of the EVS pixels is located. a second acquisition module, used to obtain a corresponding target grayscale value calculation model according to multiple groups of mapping parameters of each EVS pixel, wherein the mapping parameters include the voltage value and the corresponding second grayscale value; a calibration module, used to generate a corresponding calibration file based on the target grayscale value calculation model, and send the calibration file to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor; wherein the target grayscale value calculation model is used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
[0007] A third aspect of an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the processor is used to execute a computer program stored in the memory, and when the processor executes the computer program, it implements each step of the sensor calibration method provided in the first aspect of the embodiment of the present application.
[0008] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the sensor calibration method provided in the first aspect of the embodiment of the present application is implemented.
[0009] As can be seen from the above, according to the sensor calibration method, device, equipment and storage medium provided by the scheme of the present application, multiple original image files outputted by the dual-modal fusion sensor at different test working moments are obtained; for each original image file, the corresponding second grayscale value of each EVS pixel is calculated based on the first grayscale value of the APS pixel in the target pixel array area where each EVS pixel is located; the corresponding target grayscale value calculation model is obtained according to the multiple pairs of voltage value-second grayscale value of each EVS pixel; the corresponding calibration file is generated based on the target grayscale value calculation model, and the calibration file is sent to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor. Through the implementation of the scheme of the present application, the calibration file is obtained based on the grayscale value of the APS pixel and the voltage value of the EVS pixel, so that the calibrated dual-modal fusion sensor can calculate the analog grayscale value of the EVS pixel, and then the grayscale value of all pixels can be combined to generate a grayscale image, which effectively improves the resolution of the grayscale image and the richness of the image details. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;
[0011] Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;
[0012] Figure 3 A schematic diagram of the basic flow of a sensor calibration method provided in one embodiment of the present application;
[0013] Figure 4 A schematic diagram of an overall pixel array provided by an embodiment of the present application;
[0014] Figure 5 A schematic diagram of a fitting curve provided in one embodiment of the present application;
[0015] Figure 6 A detailed flow chart of a sensor calibration method provided in one embodiment of the present application;
[0016] Figure 7 A schematic diagram of a program module of a sensor calibration device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0017] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0018] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0019] A sensor calibration method, device, equipment and storage medium according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.
[0020] In order to solve the problem that the dual-modal fusion sensor provided in the related art can only output a grayscale image based on the information collected by the APS pixels, resulting in a low resolution of the grayscale image, an embodiment of the present application provides a sensor calibration method, which is applied to Figure 1 The scenario shown in the application scenario may include a dual-modal fusion camera 101 and an electronic device 102. The dual-modal fusion camera is configured with a dual-modal fusion sensor. The overall pixel array of the dual-modal fusion sensor includes APS (Active-Pixel Sensor) pixels and EVS (Event-based Vision Sensor) pixels. In actual applications, the layout and quantity of the two types of pixels in the overall pixel array may depend on the actual application scenario, and this embodiment does not make a sole limitation.
[0021] It is worth mentioning that an active pixel sensor is a commonly used image sensor in which each pixel sensor unit has a photodetector and at least one active transistor. In a metal oxide semiconductor (MOS) active pixel sensor, a MOS field effect transistor (MOSFET) is used as an amplifier. There are many types of APS, including the early NMOS APS and the more common CMOS APS.
[0022] The event monitoring vision sensor is a new type of sensor that simulates the human retina and responds to pixel pulses caused by brightness changes due to movement. Therefore, it can capture brightness changes (that is, changes in light intensity) of the scene at an extremely high frame rate, and record events at specific time points and specific pixel positions, forming an event stream rather than a frame stream, thereby solving problems such as information redundancy, large data storage volume, and large real-time processing volume of traditional cameras.
[0023] In addition, electronic devices are various terminal devices with data processing functions, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, vehicle-mounted terminals, and airborne terminals.
[0024] exist Figure 1 In the application scenario shown, the dual-modal fusion sensor configured by the dual-modal fusion camera 101 outputs multiple original image files (i.e., RAW files) during the test working stage. The RAW file refers to the original data of the image sensor converting the captured light source signal into a digital signal. The RAW file is then sent to the electronic device 102, and then the electronic device 102 executes the following sensor calibration method process: first, multiple original image files output by the dual-modal fusion sensor at different test working moments are obtained; wherein the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, and each original image file includes a first grayscale value corresponding to the APS pixel and a voltage value corresponding to the EVS pixel; then Then, for each original image file, the corresponding second grayscale value of each EVS pixel is calculated based on the first grayscale value of the APS pixel in the target pixel array area where each EVS pixel is located; next, the corresponding target grayscale value calculation model is obtained according to the multiple groups of mapping parameters of each EVS pixel; wherein the mapping parameters include voltage values and corresponding second grayscale values; finally, a corresponding calibration file is generated based on the target grayscale value calculation model, and the calibration file is sent to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor; wherein the target grayscale value calculation model is used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
[0025] like Figure 2 The figure shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. The electronic device mainly includes: a memory 201 and a processor 202, the number of the processor 202 can be one or more, the memory 201 stores a computer program 203 that can be run on the processor 202, the memory 201 and the processor 202 are in communication connection, and when the processor 202 executes the computer program 203, the process of the aforementioned sensor calibration method is implemented.
[0026] It should be noted that the memory 201 can be a high-speed random access memory (RAM) memory or a non-volatile memory, such as a disk memory. The memory 201 is used to store executable program codes, and the processor 202 is coupled to the memory 201 .
[0027] An embodiment of the present application further provides a computer-readable storage medium, which may be disposed in the aforementioned electronic device. The computer-readable storage medium may be the aforementioned Figure 2 Memory in the illustrated embodiment.
[0028] The computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the process of the sensor calibration method can be implemented. Furthermore, the computer storable medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a RAM, a magnetic disk or an optical disk, and other media that can store program codes.
[0029] like Figure 3 A basic flow chart of a sensor calibration method provided in an embodiment of the present application. The sensor calibration method can be performed by Figure 1 or Figure 2 The electronic device in the embodiment is executed, specifically comprising the following steps:
[0030] Step 301: Acquire multiple original image files output by the dual-modal fusion sensor at different test working moments.
[0031] In this embodiment, the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, that is, the overall photosensitive area of the dual-modal fusion sensor is divided into multiple sub-photosensitive areas, and the pixel arrays of the multiple sub-photosensitive areas correspond to the APS data mode and the EVS data mode, respectively. Compared with the multiple sensor modules that are separately set in the conventional implementation, the device volume of the sensor module is effectively compressed, which is more conducive to the miniaturization of the overall hardware architecture. In addition, it should be noted that each original image file of this embodiment includes the first grayscale value corresponding to the APS pixel and the voltage value corresponding to the EVS pixel. In actual applications, the EVS pixel is synchronously exposed with the APS pixel. A photodiode is configured in the EVS pixel. The photodiode is integrated with a capacitor that collects charge, and generates a photocurrent in response to the incident light intensity, and then generates a corresponding voltage value according to the photocurrent.
[0032] Step 302 : for each original image file, calculate the corresponding second grayscale value of each EVS pixel based on the first grayscale value of the APS pixel in the target pixel array region where each EVS pixel is located.
[0033] Specifically, in the present embodiment, APS pixels can directly output grayscale values, while EVS pixels cannot output grayscale values. The present embodiment can divide a target pixel array area including the EVS pixel in the overall pixel array for each EVS pixel, and then calculate the analog grayscale value of the EVS pixel with reference to the grayscale values of the APS pixels in the EVS neighborhood in the array area.
[0034] In an optional method of the present embodiment, before the step of calculating the corresponding second grayscale value of each EVS pixel based on the first grayscale value of the APS pixel in the target pixel array area where each EVS pixel is located, it also includes: obtaining the neighborhood pixel distribution information of each EVS pixel in the overall pixel array; and dividing the target pixel array area corresponding to each EVS pixel in the overall pixel array according to the neighborhood pixel distribution information.
[0035] Specifically, in this embodiment, in order to ensure the accuracy of the grayscale value simulation of the EVS pixel, this embodiment refers to the neighborhood pixel distribution information of each EVS pixel in the overall pixel array to determine its corresponding target pixel array area. In practical applications, the EVS pixel and multiple directly adjacent APS pixels can be demarcated as the same target pixel array area. Of course, the more APS pixels there are in the target pixel array area, the higher the reference can be provided for the grayscale value simulation of the EVS pixel. This embodiment can also further obtain the APS pixels indirectly adjacent to the EVS pixel when the total number of APS pixels directly adjacent to the EVS pixel is limited, and then the EVS pixel and the directly adjacent and indirectly adjacent APS pixels are demarcated as the same target pixel array area. It should be understood that the form of the target pixel array area divided corresponding to different EVS pixels in this embodiment may be different.
[0036] In an optional method of the present embodiment, the above-mentioned step of calculating the corresponding second grayscale value of each EVS pixel based on the first grayscale value of the APS pixels in the target pixel array area where each EVS pixel is located includes: performing weighted average calculation based on the first grayscale values of all APS pixels in the target pixel array area where each EVS pixel is located, to obtain the corresponding second grayscale value of each EVS pixel.
[0037] Specifically, in the present embodiment, the grayscale values of all APS pixels in the pixel array region where the target EVS pixel is located may be weighted averaged to simulate the grayscale value of the target EVS pixel. When the weighted average is calculated, the weights of all APS pixels may be equal. However, considering that the relative positions of different APS pixels and EVS pixels in the target pixel array region are different, the grayscale values of different APS pixels may have different references to the EVS pixels. Therefore, in the present embodiment, a larger weight may be assigned to an APS pixel that is closer to the EVS pixel coordinate position, and a smaller weight may be assigned to an APS pixel that is farther away from the EVS pixel coordinate position, so that the calculated virtual grayscale value may be more accurate.
[0038] like Figure 4 The figure shows a schematic diagram of an overall pixel array provided by the present embodiment. The present embodiment exemplarily divides a target pixel array area 401 for an EVS pixel 4011. The target pixel array area 401 includes a total of 8 pixels, including 1 EVS pixel (i.e., indicated by the letter E in the figure) and 7 APS pixels in the neighborhood (i.e., indicated by the letter A in the figure). The present embodiment assigns the same weight to the grayscale values of the 7 APS pixels in the original image file for weighted averaging, which is essentially an arithmetic mean operation. The obtained grayscale value is the simulated grayscale value of the EVS pixel 4011. It should be noted that, when the present embodiment divides the target pixel array area for different EVS pixels, some APS pixels can be shared by different target pixel array areas. Of course, different target pixel array areas can also be completely separate array areas, that is, no shared APS pixels will appear in different target pixel array areas.
[0039] Step 303: Obtain corresponding target gray value calculation models according to multiple groups of mapping parameters of each EVS pixel.
[0040] Specifically, each group of mapping parameters in this embodiment consists of a voltage value and a corresponding second grayscale value. It should be noted that after calculating the analog grayscale value of each EVS pixel, this embodiment can map the analog grayscale value with the voltage value actually output by the EVS pixel to obtain the above-mentioned mapping parameters. In this embodiment, the dual-modal fusion sensor outputs multiple original image files at multiple test working moments, and multiple groups of mapping parameters for the same EVS pixel can be obtained based on the multiple original image files. Then, this embodiment combines the multiple groups of mapping parameters to obtain the target grayscale value calculation model of a single EVS pixel.
[0041] In an optional implementation of the present embodiment, the above-mentioned steps of obtaining the corresponding target grayscale value calculation model according to the multiple sets of mapping parameters of each EVS pixel include: obtaining multiple sets of mapping parameters corresponding to multiple original image files for each EVS pixel; performing multiple random sampling on all mapping parameters to obtain multiple mapping parameter combinations; determining the corresponding grayscale value calculation model according to the multiple sets of mapping parameters in each mapping parameter combination; and obtaining the target grayscale value calculation model based on all grayscale value calculation models.
[0042] Specifically, the analog grayscale values of EVS pixels are calculated respectively for multiple original image files output by the dual-modal fusion sensor, and then the analog grayscale values of all EVS pixels are mapped to voltage values. In this way, multiple sets of mapping parameters for the same EVS pixel can be obtained. For example, if the number of original image files is 10, then 10 sets of mapping parameters can be obtained for the same EVS pixel.
[0043] Next, this embodiment determines the grayscale value calculation model of the EVS pixel based on multiple sets of mapping parameters of each EVS pixel. It should be understood that, considering that the accuracy of the grayscale value calculation model obtained by a single calculation based on multiple sets of mapping parameters is low, this embodiment performs random sampling on all the acquired mapping parameters. Each sampling randomly selects multiple sets of mapping parameters from all mapping parameters of the same EVS pixel to form a mapping parameter combination. For example, any 6 sets of mapping parameters are selected from the 10 sets of mapping parameters in the aforementioned example to obtain a mapping parameter combination. After multiple random samplings, multiple mapping parameter combinations can be obtained.
[0044] Furthermore, this embodiment can use multiple groups of mapping parameters in each mapping parameter combination to calculate the corresponding gray value calculation model, and finally, combine multiple gray value calculation models to obtain the final target gray value calculation model. It should be noted that the random sampling algorithm of this embodiment can be a RANSEC algorithm, etc.
[0045] In an optional implementation of the present embodiment, the above-mentioned steps of determining the corresponding grayscale value calculation model according to the multiple groups of mapping parameters in each mapping parameter combination include: inputting the multiple groups of mapping parameters in each mapping parameter combination into a preset fitting function, calculating the parameter values of the fitting parameters in the fitting function; substituting the calculated parameter values into the fitting function to obtain the grayscale value calculation model.
[0046] Specifically, the fitting function can be expressed as: I = f(V out ), where V out represents the voltage value in the mapping parameter, and I represents the second grayscale value in the mapping parameter. It should be understood that the mapping relationship of f(·) in the above fitting function of this embodiment can be in the form of exponential, linear, logarithmic, polynomial, etc., and this embodiment does not make a sole limitation on this.
[0047] Furthermore, in one implementation, if the above fitting function is implemented using a linear mapping relationship, the fitting function can be expressed as: I = a*V out +, where V out represents the voltage value in the mapping parameters, I represents the second grayscale value in the mapping parameters, and a and b represent fitting parameters.
[0048] In another implementation, if the above fitting function is implemented using a logarithmic mapping relationship, the fitting function can be expressed as:
[0049]
[0050] Among them, V out represents the voltage value in the mapping parameters, t represents the working duration, I represents the second grayscale value in the mapping parameters, i and j are constants, and I0, V and k represent fitting parameters.
[0051] In another implementation, if the above fitting function is implemented using a polynomial mapping relationship, the fitting function can be expressed as:
[0052]
[0053] Among them, V out represents the voltage value in the mapping parameter, I represents the second gray value in the mapping parameter, a and k are constants, and V represents the fitting parameter.
[0054] In another implementation, if the above fitting function is implemented using an exponential mapping relationship, the fitting function can be expressed as:
[0055]
[0056] Among them, V out represents the voltage value in the mapping parameters, I represents the second grayscale value in the mapping parameters, t represents the working duration, i, j, m are constants, I0, V and k represent fitting parameters.
[0057] Further preferably, the values of i and j are both 1, and the value of m is the natural constant e, that is, the fitting function can be further expressed as:
[0058]
[0059] Of course, in practical applications, the values of the above constants i, j, and m can be flexibly set according to the application scenario, and this embodiment does not make a sole limitation on this.
[0060] Specifically, in this embodiment, a fitting function is predefined, and the form of the fitting function is not limited to the above form provided in this embodiment, and can also be Fourier expansion, etc. In this embodiment, the voltage value, the second gray value (i.e., the analog gray value) in multiple groups of mapping parameters corresponding to the same EVS pixel, and the test working time corresponding to the EVS pixel when the voltage value is output can be input as known quantities into the above fitting function based on the exponential mapping relationship, and the unknown quantities I0, V and k in the fitting function are solved, and finally the obtained solution is substituted into the fitting function, that is, a fitting curve is obtained as a gray value calculation model.
[0061] In an optional implementation of the present embodiment, the above-mentioned step of obtaining the target grayscale value calculation model based on all grayscale value calculation models includes: respectively obtaining the approximation degree evaluation index of the coordinate positions of all mapping parameters fitted by all grayscale value calculation models relative to the grayscale value calculation model; determining the coordinate points whose approximation degree evaluation index is lower than a preset index threshold as flying points; and obtaining the target grayscale value calculation model with reference to the grayscale value calculation model with the least number of flying points among all grayscale value calculation models.
[0062] Specifically, in this embodiment, multiple mapping parameter combinations for the same EVS pixel can obtain multiple fitting curves, and the fitting objects (i.e., mapping parameters) of different fitting curves are different, and ultimately there are certain differences in the fitting effects. The quality of the fitting effect is mainly evaluated by the number of flying points. Figure 5 The figure shows a schematic diagram of a fitting curve provided by the present embodiment, which shows the implementation principle of curve fitting for 6 mapping parameters. All points in the figure are coordinate points corresponding to the mapping parameters, and the coordinate points of the black filling style with a lower approximation degree relative to the fitting curve represent flying points. In the present embodiment, the approximation degree of each fitted coordinate point relative to the fitting curve is calculated. The lower the approximation degree, the farther the deviation from the fitting curve is. Therefore, the present embodiment determines the coordinate points with an approximation degree lower than a preset threshold as flying points. Thus, the number of flying points of multiple fitting curves of the same EVS pixel can be determined respectively. Finally, the final target gray value calculation model is determined according to the gray value calculation model with the lowest number of flying points. It should be understood that the present embodiment can directly determine the gray value calculation model with the lowest number of flying points as the target gray value calculation model, and can also indirectly obtain the target gray value calculation model through the gray value calculation model with the lowest number of flying points.
[0063] In an optional implementation manner of the present embodiment, the above-mentioned step of obtaining the target grayscale value calculation model by referring to the grayscale value calculation model with the least number of flying points among all grayscale value calculation models includes: determining the grayscale value calculation model with the least number of flying points among all grayscale value calculation models as the preliminary grayscale value calculation model; screening out the mapping parameters corresponding to the flying points from the target mapping parameter combination preliminarily fitted by the preliminary grayscale value calculation model; and determining the corresponding target grayscale value calculation model according to the remaining multiple groups of mapping parameters in the target mapping parameter combination.
[0064] Specifically, in the present embodiment, among the multiple grayscale value calculation models of the same EVS pixel, the grayscale value calculation model with the least number of flying points is only a relatively optimal grayscale value calculation model, not an absolutely optimal grayscale value calculation model. In order to further improve the accuracy of the final target grayscale value calculation model, the present embodiment uses the grayscale value calculation model with the least number of flying points as the preliminary grayscale value calculation model, and then screens out the mapping parameters corresponding to the flying points in the preliminary grayscale value calculation model from the mapping parameter combination previously fitted by the preliminary grayscale value calculation model, and then substitutes the mapping parameter combination after the flying points are screened out into the aforementioned fitting function calculation function to calculate the unknown fitting parameters, and substitutes the calculated fitting parameter values into the unknown fitting parameters, so as to obtain the final target grayscale value calculation model. Therefore, the grayscale value calculation model finally calculated has as few or even no flying points as possible, so that the effectiveness of the model is greatly improved.
[0065] Step 304: Generate a corresponding calibration file based on the target gray value calculation model, and send the calibration file to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor.
[0066] Specifically, after obtaining the target grayscale value calculation model for each EVS pixel, the present embodiment generates a calibration file in combination with the target grayscale value calculation models of all EVS pixels, and sends the calibration file to the dual-modal fusion sensor for sensor calibration. After the calibration file takes effect on the dual-modal fusion sensor, the calibrated target grayscale value calculation model can be used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment. That is, during the actual working process of the dual-modal fusion sensor, for the original image file generated at each moment, the voltage value of the EVS pixel can be input into the target The grayscale value calculation model obtains the corresponding simulated grayscale value, and then combines the calculated simulated grayscale values of all EVS pixels and the grayscale values of all APS pixels included in the original image file to generate a grayscale image of the full pixel array, which overcomes the grayscale loss caused by EVS pixels in the related technology and improves the resolution of the grayscale image output by the dual-modal fusion sensor. In addition, it should be understood that APS pixels have the defect of grayscale blur in high-dynamic motion scenes, while EVS pixels have high dynamic characteristics. The grayscale image of this embodiment integrates the perception data of EVS pixels, thereby achieving grayscale output with high frame rate and high dynamic range.
[0067] Figure 6 The method in is a refined sensor calibration method provided by an embodiment of the present application, which is applied to the calibration of a dual-modal fusion sensor. The overall pixel array of the dual-modal fusion sensor is configured with multiple pixels, and the pixel types include APS pixels and EVS pixels. The implementation process of the sensor calibration method includes the following steps:
[0068] Step 601: Acquire multiple original image files output by the dual-modal fusion sensor at different test working moments.
[0069] In this embodiment, each original image file includes a first grayscale value corresponding to an APS pixel and a voltage value corresponding to an EVS pixel.
[0070] Step 602: divide the target pixel array area corresponding to each EVS pixel in the overall pixel array according to the neighborhood pixel distribution information of each EVS pixel in the overall pixel array.
[0071] Step 603 : performing weighted average calculation based on the first grayscale values of all APS pixels in the target pixel array region where each EVS pixel is located, to obtain a second grayscale value corresponding to each EVS pixel.
[0072] Step 604 : performing random sampling based on multiple groups of mapping parameters corresponding to multiple original image files of each EVS pixel to obtain multiple mapping parameter combinations.
[0073] In this embodiment, each mapping parameter includes the second grayscale value of each EVS pixel and a corresponding voltage value in the original image file.
[0074] Step 605: input the multiple groups of mapping parameters in each mapping parameter combination into a preset fitting function, calculate the parameter values of the fitting parameters in the fitting function, and substitute the calculated parameter values into the fitting function to obtain a gray value calculation model.
[0075] In this embodiment, in the process of solving the fitting function, that is, the process of mapping parameters as curve fitting of coordinate points in the coordinate system, the fitting function can be expressed as:
[0076]
[0077] Step 606: Determine the grayscale value calculation model with the least number of flying points among the multiple grayscale value calculation models of the same EVS pixel as the preliminary grayscale value calculation model.
[0078] In this embodiment, the flying point is a coordinate point with a larger deviation and a smaller approximation relative to the fitting curve during fitting.
[0079] Step 607: Screen out the mapping parameters corresponding to the flying points from the target mapping parameter combination preliminarily fitted by the preliminary gray value calculation model.
[0080] Step 608: re-input the remaining multiple groups of mapping parameters in the target mapping parameter combination into the fitting function, calculate the parameter values of the fitting parameters in the fitting function, and substitute the calculated parameter values into the fitting function to obtain the target gray value calculation model.
[0081] Step 609: Generate a corresponding calibration file in combination with the target grayscale value calculation model of all EVS pixels, and send the calibration file to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor.
[0082] In this embodiment, the target gray value calculation model is used by the dual-modal fusion sensor to calculate the analog gray value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
[0083] It should be understood that the size of the serial number of each step in this embodiment does not mean the order of execution of the steps. The execution order of each step should be determined by its function and internal logic, and should not constitute a sole limitation on the implementation process of the embodiment of this application.
[0084] Figure 7 A sensor calibration device is provided in one embodiment of the present application. The sensor calibration device can be used to implement the sensor calibration method in the above embodiment. The sensor calibration device mainly includes:
[0085] The first acquisition module 701 is used to acquire multiple original image files output by the dual-modal fusion sensor at different test working moments; wherein the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, and each original image file includes a first grayscale value corresponding to the APS pixel and a voltage value corresponding to the EVS pixel;
[0086] A calculation module 702 is used to calculate, for each original image file, a second grayscale value corresponding to each EVS pixel based on the first grayscale value of the APS pixel in the target pixel array area where each EVS pixel is located;
[0087] A second acquisition module 703 is used to acquire a corresponding target gray value calculation model according to multiple groups of mapping parameters of each EVS pixel; wherein the mapping parameters include a voltage value and a corresponding second gray value;
[0088] The calibration module 704 is used to generate a corresponding calibration file based on the target grayscale value calculation model, and send the calibration file to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor; wherein the target grayscale value calculation model is used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
[0089] In some implementations of this embodiment, the sensor calibration device also includes: a division module, used to obtain neighborhood pixel distribution information of each EVS pixel in the overall pixel array; and divide the target pixel array area corresponding to each EVS pixel in the overall pixel array according to the neighborhood pixel distribution information.
[0090] In some implementations of this embodiment, the calculation module is specifically used to: perform weighted average calculation based on the first grayscale values of all APS pixels in the target pixel array area where each EVS pixel is located, to obtain the corresponding second grayscale value of each EVS pixel.
[0091] In some implementations of the present embodiment, the second acquisition module is specifically used to: obtain multiple sets of mapping parameters corresponding to multiple original image files for each EVS pixel; perform multiple random sampling on all mapping parameters to obtain multiple mapping parameter combinations; determine the corresponding grayscale value calculation model according to the multiple sets of mapping parameters in each mapping parameter combination; and obtain the target grayscale value calculation model based on all grayscale value calculation models.
[0092] In some implementations of the present embodiment, when the acquisition module executes the above-mentioned function of determining the corresponding grayscale value calculation model according to the multiple groups of mapping parameters in each mapping parameter combination, it is specifically used to: input the multiple groups of mapping parameters in each mapping parameter combination into a preset fitting function, calculate the parameter values of the fitting parameters in the fitting function; substitute the calculated parameter values into the fitting function to obtain the grayscale value calculation model.
[0093] The fitting function is expressed as:
[0094]
[0095] Among them, V out represents the voltage value in the mapping parameters, I represents the second gray value in the mapping parameters, t represents the working duration, i, j, m are constants, I0, V and k represent fitting parameters.
[0096] In some implementations of the present embodiment, when the acquisition module executes the above-mentioned function of acquiring the target grayscale value calculation model based on all grayscale value calculation models, it is specifically used to: respectively acquire the approximation degree evaluation index of the coordinate positions of all mapping parameters fitted by all grayscale value calculation models relative to the grayscale value calculation model; determine the coordinate points whose approximation degree evaluation index is lower than the preset index threshold as flying points; and acquire the target grayscale value calculation model with reference to the grayscale value calculation model with the least number of flying points among all grayscale value calculation models.
[0097] In some implementations of the present embodiment, when the acquisition module executes the above-mentioned function of obtaining the target grayscale value calculation model by referring to the grayscale value calculation model with the least number of flying points among all grayscale value calculation models, it is specifically used to: determine the grayscale value calculation model with the least number of flying points among all grayscale value calculation models as the preliminary grayscale value calculation model; screen out the mapping parameters corresponding to the flying points from the target mapping parameter combination preliminarily fitted by the preliminary grayscale value calculation model; and determine the corresponding target grayscale value calculation model according to the remaining multiple groups of mapping parameters in the target mapping parameter combination.
[0098] It should be noted that the sensor calibration methods in the aforementioned embodiments can all be implemented based on the sensor calibration device provided in this embodiment. Ordinary technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working process of the sensor calibration device described in this embodiment can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0099] Based on the technical solution of the above-mentioned embodiment of the present application, a plurality of original image files outputted by the dual-modal fusion sensor at different test working moments are obtained; for each original image file, the corresponding second grayscale value of each EVS pixel is calculated based on the first grayscale value of the APS pixel in the target pixel array area where each EVS pixel is located; the corresponding target grayscale value calculation model is obtained according to the multiple pairs of voltage value-second grayscale value of each EVS pixel; the corresponding calibration file is generated based on the target grayscale value calculation model, and the calibration file is sent to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor. Through the implementation of the solution of the present application, the calibration file is obtained based on the grayscale value of the APS pixel and the voltage value of the EVS pixel, so that the calibrated dual-modal fusion sensor can calculate the analog grayscale value of the EVS pixel, and then the grayscale value of all pixels can be combined to generate a grayscale image, which effectively improves the resolution of the grayscale image and the richness of the image details.
[0100] It should also be noted that the devices and methods disclosed in the several embodiments provided in this application can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0101] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0102] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.
[0103] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a readable storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned readable storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.
[0104] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.
[0105] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0106] The above is a description of the sensor calibration method, device, equipment and storage medium provided in this application. For technicians in this field, according to the ideas of the embodiments of this application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on this application.
Claims
1. A sensor calibration method, characterized in that: include: Acquire multiple original image files output by the dual-modal fusion sensor at different test working moments; wherein the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, and each of the original image files includes a first grayscale value corresponding to the APS pixel and a voltage value corresponding to the EVS pixel; For each of the original image files, respectively calculating a second grayscale value corresponding to each of the EVS pixels based on the first grayscale value of the APS pixel in the target pixel array region where each of the EVS pixels is located; Obtaining corresponding target grayscale value calculation models according to multiple groups of mapping parameters of each of the EVS pixels; wherein the mapping parameters include the voltage value and the corresponding second grayscale value; A corresponding calibration file is generated based on the target grayscale value calculation model, and the calibration file is sent to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor; wherein the target grayscale value calculation model is used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
2. The sensor calibration method according to claim 1, characterized in that: Before the step of calculating the second grayscale value corresponding to each of the EVS pixels based on the first grayscale value of the APS pixels in the target pixel array region where each of the EVS pixels is located, the method further includes: Obtaining neighborhood pixel distribution information of each of the EVS pixels in the overall pixel array; According to the neighborhood pixel distribution information, the target pixel array area corresponding to each of the EVS pixels is divided in the overall pixel array.
3. The sensor calibration method according to claim 1, characterized in that: The step of calculating the corresponding second grayscale value of each of the EVS pixels based on the first grayscale value of the APS pixels in the target pixel array area where each of the EVS pixels is located comprises: A weighted average calculation is performed based on the first grayscale values of all the APS pixels in the target pixel array area where each of the EVS pixels is located, so as to obtain a second grayscale value corresponding to each of the EVS pixels.
4. The sensor calibration method according to claim 1, characterized in that: The step of obtaining the corresponding target gray value calculation model according to the multiple groups of mapping parameters of each EVS pixel includes: Acquire multiple groups of mapping parameters corresponding to the multiple original image files for each of the EVS pixels; Performing multiple random sampling on all the mapping parameters to obtain multiple mapping parameter combinations; Determining corresponding gray value calculation models according to a plurality of the mapping parameters in each mapping parameter combination; A target grayscale value calculation model is acquired based on all the grayscale value calculation models.
5. The sensor calibration method according to claim 4, characterized in that: The step of determining the corresponding gray value calculation model according to the plurality of mapping parameters in each mapping parameter combination comprises: The multiple groups of mapping parameters in each mapping parameter combination are respectively input into a preset fitting function, and the parameter values of the fitting parameters in the fitting function are calculated; the fitting function is expressed as: I=f(V out ), Among them, V out represents the voltage value in the mapping parameter, and I represents the second grayscale value in the mapping parameter; Substitute the calculated parameter values into the fitting function to obtain a gray value calculation model.
6. The sensor calibration method according to claim 4, characterized in that: The gray value calculation model is a fitting curve; the step of obtaining a target gray value calculation model based on all the gray value calculation models comprises: Respectively obtaining an evaluation index of the approximation degree of the coordinate positions of all the mapping parameters fitted by all the gray value calculation models relative to the gray value calculation model; Determine the coordinate point whose approximation evaluation index is lower than the preset index threshold as a flying point; The target grayscale value calculation model is obtained by referring to the grayscale value calculation model with the least number of flying points among all the grayscale value calculation models.
7. The sensor calibration method according to claim 6, characterized in that: The step of obtaining a target grayscale value calculation model by referring to the grayscale value calculation model with the least number of flying points among all the grayscale value calculation models comprises: Determine the grayscale value calculation model with the least number of flying points among all the grayscale value calculation models as a preliminary grayscale value calculation model; Screening out the mapping parameters corresponding to the flying points from the target mapping parameter combination preliminarily fitted by the preliminary gray value calculation model; A corresponding target grayscale value calculation model is determined according to the remaining multiple mapping parameters in the target mapping parameter combination.
8. A sensor calibration device, characterized in that: include: A first acquisition module is used to acquire a plurality of original image files outputted by the dual-modal fusion sensor at different test working moments; wherein the overall pixel array of the dual-modal fusion sensor includes APS pixels and EVS pixels, and each of the original image files includes a first grayscale value corresponding to the APS pixel and a voltage value corresponding to the EVS pixel; A calculation module, for calculating, for each of the original image files, a second grayscale value corresponding to each of the EVS pixels based on the first grayscale value of the APS pixel in the target pixel array region where each of the EVS pixels is located; A second acquisition module, used to acquire a corresponding target grayscale value calculation model according to a plurality of mapping parameters of each of the EVS pixels; wherein the mapping parameters include the voltage value and the corresponding second grayscale value; A calibration module is used to generate a corresponding calibration file based on the target grayscale value calculation model, and send the calibration file to the dual-modal fusion sensor to calibrate the dual-modal fusion sensor; wherein the target grayscale value calculation model is used by the dual-modal fusion sensor to calculate the analog grayscale value corresponding to the voltage value of each EVS pixel in the original image file generated at the actual working moment.
9. An electronic device, characterized in that: The device comprises a memory and a processor, wherein: The processor is used to execute the computer program stored in the memory; When the processor executes the computer program, the steps in the sensor calibration method described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps in the sensor calibration method described in any one of claims 1 to 7 are implemented.
