Real-time response correction method, system and equipment based on in-orbit image sensor and storage medium

Abstract

The invention discloses an on-orbit-based image sensor real-time response correction method, system and equipment and a storage medium, belongs to the technical field of image correction, and solves the problems that an existing method cannot accurately realize on-orbit real-time temperature drift correction of a sensor, the effective dynamic range of an image is reduced, the data transmission broadband is increased, and the data transmission efficiency is improved. And meanwhile, the ground data calculation amount is increased. Comprising the following steps: S1, carrying out ground radiation calibration according to an on-orbit imaging working condition, and obtaining an on-orbit temperature drift calibration coefficient; s2, performing on-orbit real-time temperature drift correction on the image sensor; step S3, performing digital domain TDI and digital domain Bining processing on the real-time response of the image sensor after temperature drift correction to obtain data with a bit width of 16 bits; and S4, converting the data with the 16-bit width into data with the 12-bit width by adopting a segmentation mapping method, and outputting the data with the 12-bit width, so as to complete the correction of the real-time response of the image sensor.

Description

Technical Field

[0001] The present invention relates to the technical field of image correction, and particularly to a real-time response correction method, system, device and storage medium for an image sensor based on orbit. Background Art

[0002] In the prior art, Chinese Patent CN111182240A discloses an "image sensor temperature drift self-compensation method", which solves the problem of pixel distortion of the image sensor output caused by high ambient temperature through linearization processing of the acquired image and implementation of temperature pixel drift compensation. However, this method ignores the bright-field response differences of each pixel of the image sensor and only compensates for the dark-field temperature drift. Chinese Patent CN110519537A discloses an "image sensor array and its temperature drift compensation method", which performs temperature drift compensation on an uncooled infrared focal plane array: obtaining the output temperature drift of the blind pixel array in the image sensor array; obtaining the output temperature drift of the photosensitive pixels according to the output temperature drift of the blind pixel array; removing the output temperature drift of the photosensitive pixels from the originally acquired output change amount of the image sensor array to obtain the corrected output change amount. Through blind pixel reading and deduction, the temperature drift correction of the dark field is completed, and the problem of bright-field temperature drift correction cannot be solved. Chinese Patent CN117714905A discloses a "precision correction method for radiation response characteristics of a CMOS image sensor", and the method specifically includes: S1, proposing a precision temperature drift correction model for the radiation response of a CMOS image sensor and establishing a multivariate non-linear function relationship between the theoretical response value, actual response value and temperature of the image sensor; S2, formulating a radiation calibration temperature drift test process to realize multi-condition traversal of temperature and light intensity; S3, collecting radiation calibration data by using the process described in step S2 and determining the parameters of the temperature drift model; S4, substituting the parameters of the temperature drift model and the actual response value of the image sensor into the precision temperature drift correction model to calculate and obtain the radiation theoretical response value of the image sensor, and realizing the precision correction of the radiation response characteristics of the image sensor. This method performs temperature drift correction processing on the ground-end L0-level product image after camera imaging, sacrifices the dynamic range of the image under high gain and high integration levels, makes the image sensitivity not ideal enough, and needs to traverse each imaging condition of each spectral band when calibrating the temperature drift coefficient. In order to completely retain the data, the data is downloaded with a 16-bit width, which not only increases the data bandwidth but also increases the computational amount of ground radiation calculation.

[0003] In summary, the existing methods cannot accurately achieve temperature drift correction, reduce the effective dynamic range of the image, increase both the data transmission bandwidth and the ground data computational amount. Summary of the Invention

[0004] The present invention solves the problems that the existing methods cannot accurately achieve temperature drift correction, and while increasing the data transmission bandwidth, the amount of ground data calculation is increased.

[0005] A real-time response correction method for an in-orbit image sensor according to the present invention includes the following steps:

[0006] Step S1, perform ground radiation calibration according to the in-orbit imaging conditions to obtain the in-orbit temperature drift calibration coefficient;

[0007] Step S2, perform in-orbit real-time temperature drift correction on the image sensor;

[0008] Step S3, perform digital domain TDI and digital domain Bining processing on the real-time response of the image sensor after temperature drift correction to obtain data with a 16-bit width;

[0009] Step S4, use the piecewise mapping method to convert the 16-bit width data into 12-bit width data and output, then complete the correction of the real-time response of the image sensor.

[0010] Further, in an embodiment of the present invention, in the step S1, the ground radiation calibration to obtain the in-orbit temperature drift calibration coefficient includes the following steps:

[0011] Step S101, sequentially set the energy levels of the integrating sphere;

[0012] Step S102, set the imaging parameters, traverse the in-orbit use gain, and set the integration level to one level;

[0013] Step S103, read the real-time temperature of the image sensor and start imaging, respectively save the image gray value and the real-time temperature of the image sensor for each imaging, and continue for several minutes;

[0014] Step S104, determine whether the tests for all the energy levels of the integrating sphere are completed. If completed, end the test. If not completed, return to step S101, increase the energy level of the integrating sphere, and continue the test.

[0015] Further, in an embodiment of the present invention, in the step S102, setting the integration level to one level significantly reduces the number of radiation calibration conditions compared with the ground temperature drift correction.

[0016] Further, in an embodiment of the present invention, in the step S103, the "continue for several minutes" is greater than or equal to 15 minutes.

[0017] Further, in an embodiment of the present invention, in the step S2, the temperature drift correction of the real-time temperature of the image sensor includes the following steps:

[0018] Step S101, during the imaging process of the image sensor, read and record the real-time temperature of the image sensor;

[0019] Step S102, load the temperature drift coefficient corresponding to the working condition according to the imaging parameters, and substitute both the real-time temperature and the temperature drift coefficient into the temperature drift correction model, so as to complete the temperature drift correction of the real-time response of the image sensor.

[0020] Furthermore, in an embodiment of the present invention, in the step S102, the temperature drift correction model is specifically:

[0021] μ y =(a*μ y,real +b)*T + c*μ y,real +e;

[0022] Wherein, μ y is the actual gray value of the sensor, μ y,real is the theoretical gray value of the sensor, T is the sensor temperature, a is the bright field temperature drift gain constant, b is the dark field temperature drift gain constant, c is the bright field temperature drift offset constant, and e is the dark field temperature drift offset constant.

[0023] A real-time response correction system for an in-orbit image sensor according to the present invention includes the following modules:

[0024] Module S1, perform ground radiation calibration according to the in-orbit imaging working condition to obtain the in-orbit temperature drift calibration coefficient;

[0025] Module S2, perform in-orbit real-time temperature drift correction on the image sensor;

[0026] Module S3, perform digital domain TDI and digital domain Bining processing on the real-time response of the image sensor after temperature drift correction to obtain data with a 16-bit width;

[0027] Module S4, use the segmented mapping method to convert the 16-bit width data into 12-bit width data and output it, then complete the correction of the real-time response of the image sensor.

[0028] An electronic device according to the present invention includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0029] The memory is used to store a computer program;

[0030] The processor is used to implement the method steps described in any one of the above methods when executing the program stored in the memory.

[0031] A computer-readable storage medium according to the present invention stores a computer program therein, and when the computer program is executed by a processor, the method steps described in any of the above methods are implemented.

[0032] The present invention solves the problems that the existing methods cannot accurately achieve temperature drift correction, reduce the effective dynamic range of the image, increase the data transmission bandwidth, and increase the amount of ground data calculation at the same time. The specific beneficial effects include:

[0033] A real-time response correction method for an on-orbit image sensor according to the present invention performs temperature drift correction before the digital domain TDI of the CMOS image sensor at the imaging end. This correction method can simultaneously and accurately remove the image background noise in real time, reduce the number of working conditions of the temperature drift coefficient calibration experiment, and can also improve the effective response dynamic range of the image. In addition, the present invention also designs an adaptive 16-bit segmented mapping for 12-bit quantization bit data download, which can improve the effective dynamic range of the image, reduce the data transmission bandwidth, and reduce the amount of ground data calculation without losing the image accuracy;

[0034] A real-time response correction method for an on-orbit image sensor according to the present invention can improve the effective dynamic range, sensitivity, stability, signal-to-noise ratio, and relative radiation accuracy of the image in real time, obtain calibration coefficients with fewer experimental working conditions, and use segmented mapping to achieve quantization bit conversion output, which is suitable for high-precision imaging of pushbroom cameras in the field of space optical remote sensing;

[0035] A real-time response correction method for an on-orbit image sensor according to the present invention is also applicable to CCD image sensors. Since the CCD image sensor integrates in the charge domain, the radiation correction method mentioned in the present invention will not reduce the number of working conditions for calibrating the temperature drift coefficient in the laboratory, but can also achieve the effect of improving the dynamic range. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the following description of the embodiments in conjunction with the drawings, wherein:

[0037] Figure 1 is a comparison diagram of the original image and the image after relative temperature drift correction described in Embodiment 1;

[0038] Figure 2 is a flowchart of the imaging process based on the sensor TDICMOS described in Embodiment 1;

[0039] Figure 3 is a 16-bit-12-bit response value segmented mapping rule diagram described in Embodiment 1;

[0040] Figure 4It is a block diagram of the imaging principle of a satellite camera based on the sensor TDICMOS described in Embodiment 1. Specific Embodiment

[0041] The following will clearly and completely describe various embodiments of the present invention with reference to the accompanying drawings. The embodiments described by referring to the accompanying drawings are exemplary and are intended to explain the present invention, and should not be construed as a limitation to the present invention.

[0042] Embodiment 1. A real-time response correction method for an image sensor based on orbit, comprising the following steps:

[0043] Step S1, perform ground radiation calibration according to the on-orbit imaging conditions to obtain the on-orbit temperature drift calibration coefficient;

[0044] Step S2, perform on-orbit real-time temperature drift correction on the image sensor;

[0045] Step S3, perform digital domain TDI and digital domain Bining processing on the real-time response of the image sensor after temperature drift correction to obtain data with a 16-bit width;

[0046] Step S4, use the piecewise mapping method to convert the 16-bit width data into 12-bit width data and output, then complete the correction of the real-time response of the image sensor.

[0047] In this embodiment, in the step S1, the step of loading the temperature drift coefficient corresponding to the working condition according to the imaging parameters includes the following steps:

[0048] Step S101, sequentially set the energy levels of the integrating sphere;

[0049] Step S102, set the imaging parameters, traverse the on-orbit use gain, and set the integration level to one level;

[0050] Step S103, read the real-time temperature of the image sensor and start imaging, respectively save the image gray value and the real-time temperature of the image sensor for each imaging, and continue for several minutes;

[0051] Step S104, determine whether the tests for all the energy levels of the integrating sphere are completed correspondingly. If completed, end the test. If not completed, return to step S101, increase the energy level of the integrating sphere, and continue the test.

[0052] In this embodiment, in the step S102, the integration level is set to one level, which significantly reduces the number of radiation calibration working conditions compared with the ground temperature drift correction.

[0053] In this embodiment, in the step S103, the "continue for several minutes" is greater than or equal to 15 minutes.

[0054] In this embodiment, in step S2, the real-time temperature of the image sensor is subjected to temperature drift correction, which includes the following steps:

[0055] Step S101, during the imaging process of the image sensor, read and record the real-time temperature of the image sensor;

[0056] Step S102, load the temperature drift coefficient corresponding to the working condition according to the imaging parameters, and substitute both the real-time temperature and the temperature drift coefficient into the temperature drift correction model, so as to complete the temperature drift correction of the real-time response of the image sensor.

[0057] In this embodiment, in step S102, the temperature drift correction model is specifically:

[0058] μ y =(a*μ y,real +b)*T + c*μ y,real +e;

[0059] Where, μ y is the actual gray value of the sensor, μ y,real is the theoretical gray value of the sensor, T is the sensor temperature, a is the bright field temperature drift gain constant, b is the dark field temperature drift gain constant, c is the bright field temperature drift offset constant, and e is the dark field temperature drift offset constant.

[0060] In the prior art, it is impossible to accurately achieve on-orbit real-time temperature drift correction, which reduces the effective dynamic range of the image, increases the data transmission bandwidth, and increases the ground data calculation amount.

[0061] To solve the above technical problems, in this embodiment, during the satellite imaging process, the temperature drift characteristics of the sensor are precisely radiometrically corrected at the imaging end, and thus a method for real-time response correction of an image sensor based on on-orbit is proposed, including the following steps:

[0062] Step S1, during the camera imaging process, read and record the real-time temperature of the sensor, load the calibrated temperature drift coefficient according to the imaging parameters, substitute the temperature and the temperature drift coefficient into the temperature drift correction model, so as to obtain the true response of the image sensor after temperature drift correction. This on-orbit correction method realizes noise floor removal at the imaging end, improves the effective dynamic range of the image, and effectively reduces the calculation of redundant data and resource waste at the backend. The effects before and after temperature drift correction are as Figure 1 shown.

[0063] On-orbit real-time radiometric correction of the image sensor is to perform temperature drift correction inside the camera FPGA, calculate the radiometric theoretical response value of the image sensor according to the temperature drift correction model, and realize precise correction of the radiation characteristics of the image sensor. The used camera imaging system is as Figure 4 shown.

[0064] The process of real-time temperature drift correction for the on-orbit image sensor includes the following steps:

[0065] Step S101: Align the camera with a uniform target scene for imaging;

[0066] Step S102: At the start of the test, power on the camera and the sensor step by step;

[0067] Step S103: Load the temperature drift coefficient into the FPGA according to the imaging working condition;

[0068] Step S104: Read the temperature of the temperature sensor and the actual response value of the image sensor line by line. Substitute the read temperature, image response DN value, and temperature drift coefficient into the temperature drift correction model to calculate the theoretical radiation response value of the image sensor, and achieve precise correction of the radiation characteristics of the image sensor.

[0069] The precise temperature drift correction model for the radiation response of the CMOS image sensor proposed in this embodiment establishes a multivariate nonlinear function relationship among the theoretical response value, actual response value, and temperature of the image sensor;

[0070] The specific precise temperature drift correction model is as follows:

[0071] μ y =(a*μ y,real +b)*T + c*μ y,real +e;

[0072] Where, μ y is the actual gray value of the sensor, μ y,real is the theoretical gray value of the sensor, T is the sensor temperature, a is the bright-field temperature drift gain constant, b is the dark-field temperature drift gain constant, c is the bright-field temperature drift offset constant, and e is the dark-field temperature drift offset constant. The dark field refers to the working condition without light, and the dark-field temperature drift parameter is a parameter independent of the light intensity. The bright field refers to the working condition with light, and the bright-field temperature drift parameter is a temperature drift parameter related to the light intensity.

[0073] The determination of the parameters of the precise temperature drift model is specifically as follows:

[0074] When the energy level of the integrating sphere is L0, that is, the theoretical gray value of the sensor μ y,real =0, μ y =b*T + e; By recording μ y at two or more different sensor temperatures T, the parameter values of b and e corresponding to each pixel can be calculated by the least squares method; when the energy level of the integrating sphere is other levels, using the already obtained parameter values of b and e, and obtaining μ y at three or more different sensor temperatures T, the parameter values of a and c corresponding to each pixel can be calculated by the least squares method.

[0075] The calculation of the temperature drift coefficient in this embodiment includes the following steps:

[0076] Step S105: At the start of the test, power on the integrating sphere and the test development board.

[0077] Step S106: Set the energy levels of the integrating sphere to L0 - L80 in sequence. The energy levels increase sequentially based on every 10 units.

[0078] Step S107: Set the imaging parameters, traverse the gains used in orbit, set the integration level to one level, and power on the camera.

[0079] Step S108: Read the sensor temperature and start imaging. Save the image gray values and the sensor temperature for each imaging, lasting for greater than or equal to 15 minutes to ensure sufficient sensor heating and cover the temperature range during the operation of the on - orbit sensor.

[0080] Step S109: Power off the camera and wait for the camera to cool down.

[0081] Step S110: Determine whether the tests for all the energy levels of the integrating sphere are completed. If completed, power off the test development board and end the test; if not completed, return to Step S105, increase the energy level of the integrating sphere, and continue the test.

[0082] Step S2: Select to perform before the digital - domain TDI and digital - domain Binning of the CMOS image sensor. When performing on - orbit real - time radiation correction on the image sensor, it is necessary to load the corresponding temperature drift coefficients according to the imaging parameters. The imaging parameters involved include gain and integration level. Since the camera imaging integration level is one level before digital - domain TDI, only the temperature drift coefficients corresponding to the gain and one - level integration level need to be loaded. That is, when calibrating the temperature drift coefficients in the laboratory, only the gains need to be traversed, and the integration level is set to one level. This can reduce the number of calibration conditions in the laboratory, save time, and reduce the memory for coefficient storage in orbit. The imaging processing flow is as Figure 2 shown.

[0083] The CMOS sensor involved in Step S3 adopts the digital - domain TDI and digital - domain Binning processing method. This choice enables temperature drift correction at the front end of integration, reducing the number of temperature drift coefficient calibration conditions.

[0084] The CMOS in this embodiment is a 3265 - type sensor with an effective pixel of 9344 * 7000 and a dark pixel of 96 * 7000. The line - array push - broom imaging mode is selected, and the imaging spectral bands include panchromatic, RGB, and near - infrared spectral bands. The imaging conditions for laboratory tests include four gains of 1.25x, 3x, 4x, and 6x, and the integration level is 1 level.

[0085] Step S4: After performing digital domain TDI on the true image response after temperature drift correction, data with a 16-bit width is obtained. The 16-bit data is converted to 12-bit data for output using a piecewise mapping method. The low response segment from 0 to x1 is mapped to 0 to y1, the medium response segment from x1 to x2 is mapped to y1 to y2, and the high response segment from x2 to x3 is mapped to y2 to y3. This piecewise mapping method can not only effectively improve the image representation accuracy, but also reduce the data transmission bandwidth, and at the same time can also reduce the ground calculation amount for 16-bit data. The mapping rules are as Figure 3 shown.

[0086]

[0087] The 16-bit and 12-bit piecewise mapping of the data proposed in this embodiment. The image data after temperature drift correction and digital domain TDI is 16-bit wide. In order to minimize the data transmission bandwidth without loss of image accuracy, and at the same time to reduce the ground data processing calculation amount, the 16-bit data is converted to 12-bit for data download through the piecewise mapping rule

[0088] The 16-bit data piecewise mapping rule is as follows:

[0089] According to the solar altitude angle, reflectivity, and camera imaging conditions (gain, integration level) of the shot, evaluate the response range of the captured image, and determine the values of the segmentation points in the mapping rule. In this embodiment, (x1, y1) = (3072, 500), (x2, y2) = (52428, 3072), (x3, y3) = (65535, 4095) are adopted;

[0090] According to the set mapping rule, map and output the 16-bit data to 12-bit data to obtain 12-bit data.

[0091] In summary, an on-orbit image sensor real-time response correction method described in this embodiment can simultaneously and accurately remove image background noise in real time, reduce the number of working conditions in the temperature drift coefficient calibration experiment, and can also improve the effective response dynamic range of the image. In addition, this embodiment also designs an adaptive 16-bit piecewise mapping and 12-bit quantization bit data download, which can reduce both the data transmission bandwidth and the ground data operation amount without loss of image accuracy.

[0092] Embodiment 2: An on-orbit image sensor real-time response correction system described in this embodiment includes the following modules:

[0093] Module S1 performs ground radiation calibration according to the on-orbit imaging conditions to obtain the on-orbit temperature drift calibration coefficient;

[0094] Module S2 performs on-orbit real-time temperature drift correction on the image sensor;

[0095] Module S3 processes the real-time response of the temperature drift-corrected image sensor in the digital domain for TDI and digital domain Bining to obtain data with a 16-bit width;

[0096] Module S4 converts the 16-bit width data to 12-bit width data and outputs it using the piecewise mapping method, thus completing the correction of the real-time response of the image sensor.

[0097] Embodiment 3. An electronic device according to this embodiment includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0098] The memory is used to store computer programs;

[0099] The processor is used to implement the method steps described in Embodiment 1 when executing the programs stored on the memory.

[0100] Embodiment 4. A computer-readable storage medium according to this embodiment stores a computer program in the computer-readable storage medium. When the computer program is executed by a processor, it implements the method steps described in Embodiment 1.

[0101] The above has introduced in detail a method, system, device, and storage medium for real-time response correction of an on-orbit image sensor proposed by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A real-time response correction method for an image sensor based on orbit, characterized in that It includes the following steps: Step S1: Perform ground radiation calibration according to the on-orbit imaging conditions to obtain the on-orbit temperature drift calibration coefficient; Step S2: Perform on-orbit real-time temperature drift correction on the image sensor; Step S3: Perform digital domain TDI and digital domain Bining processing on the real-time response of the temperature drift-corrected image sensor to obtain data with a 16-bit width; Step S4: Use the piecewise mapping method to convert the 16-bit width data into 12-bit width data and output it, thus completing the correction of the real-time response of the image sensor.

2. The real-time response correction method of an on-orbit image sensor according to claim 1, wherein In the above-mentioned step S1, the ground radiation calibration to obtain the on-orbit temperature drift calibration coefficient includes the following steps: Step S101: Sequentially set the energy levels of the integrating sphere; Step S102: Set the imaging parameters, traverse the on-orbit use gain, and set the integration level to one level; Step S103: Read the real-time temperature of the image sensor and start imaging, and separately save the image gray value and the real-time temperature of the image sensor for each imaging, lasting for several minutes; Step S104: Determine whether the tests for all the energy levels of the integrating sphere are completed. If completed, end the test. If not completed, return to step S101, increase the energy level of the integrating sphere, and continue the test.

3. A real-time response correction method for an in-orbit image sensor according to claim 2, characterized in that In the above-mentioned step S102, setting the integration level to one level significantly reduces the number of radiation calibration conditions compared with the ground temperature drift correction.

4. A real-time response correction method for an in-orbit image sensor according to claim 2, characterized in that, In the above-mentioned step S103, the "lasting for several minutes" is greater than or equal to 15 minutes.

5. A real-time response correction method for an in-orbit image sensor according to claim 1, characterized in that, In the above-mentioned step S2, the temperature drift correction of the real-time temperature of the image sensor includes the following steps: Step S101: During the imaging process of the image sensor, read and record the real-time temperature of the image sensor; Step S102: Load the temperature drift coefficient corresponding to the working condition according to the imaging parameters, and substitute both the real-time temperature and the temperature drift coefficient into the temperature drift correction model, thereby completing the temperature drift correction of the real-time response of the image sensor.

6. The real-time response correction method for an on-orbit image sensor according to claim 5, characterized in that In the above-mentioned step S102, the temperature drift correction model is specifically: μ y = (a * μ y,real + b) * T + c * μ y,real + e; where μ y is the actual gray value of the sensor, μ y,real is the theoretical gray value of the sensor, T is the sensor temperature, a is the bright-field temperature drift gain constant, b is the dark-field temperature drift gain constant, c is the bright-field temperature drift offset constant, and e is the dark-field temperature drift offset constant.

7. A real-time response correction system for an in-orbit image sensor, characterized in that, It includes the following modules: Module S1: Perform ground radiation calibration according to the on-orbit imaging conditions to obtain the on-orbit temperature drift calibration coefficient; Module S2: Perform on-orbit real-time temperature drift correction on the image sensor; Module S3: Perform digital domain TDI and digital domain Bining processing on the real-time response of the temperature drift-corrected image sensor to obtain data with a 16-bit width; Module S4: Use the piecewise mapping method to convert the 16-bit width data into 12-bit width data and output it, thus completing the correction of the real-time response of the image sensor.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus; The memory is used to store computer programs; The processor is used to implement the method steps described in any one of claims 1-6 when executing the programs stored on the memory.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the method steps described in any one of claims 1-6.

Images