Image restoration processing method based on joint calibration of area array CMOS and linear array CCD

Abstract

The present disclosure provides an image restoration processing method based on joint calibration of area array CMOS and linear array CCD, wherein the area array CMOS is a rolling shutter area array CMOS, the area array CMOS is installed on both sides of the linear array CCD, the rolling shutter direction of the area array CMOS is perpendicular to the push-broom direction, and the method comprises the following steps: constructing an area array CMOS-linear array CCD joint calibration model according to the area array CMOS image and the linear array CCD image at the same time; obtaining a CMOS jitter parameter curve according to the sequence autocorrelation CMOS image; obtaining linear array CCD jitter curve information according to the CMOS jitter parameter curve and the area array CMOS-linear array CCD joint calibration model; and performing restoration processing on the linear array CCD image by using a steady-state re-imaging method according to the linear array CCD jitter curve information. The present disclosure can realize accurate perception of the linear array CCD jitter characteristics under the condition of fast maneuvering through the area array CMOS-linear array CCD joint calibration model, improve the accuracy of the linear array CCD jitter information, provide accurate input for subsequent restoration processing, and lay a foundation for platform multi-mode data acquisition and ground high-quality data processing.

Description

Technical Field

[0001] This disclosure relates to the fields of payload design and image data processing technology, and in particular to an image restoration processing method based on joint calibration of area array CMOS and linear array CCD. Background Technology

[0002] Flutter is a common problem in platform imaging. Considering that the impact of platform flutter will become even more sensitive when the imaging angular resolution reaches 0.1 arcseconds or even higher in the future, the platform's maneuverability is constantly improving, and imaging in motion is gradually becoming an important imaging mode, making the platform flutter characteristics more complex. Traditional flutter detection and compensation methods have the problem of being unable to accurately sense and measure the flutter error of ultra-high resolution images. Summary of the Invention

[0003] This disclosure aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first aspect of this disclosure proposes an image restoration processing method based on joint calibration of an area array CMOS and a linear array CCD, characterized in that the area array CMOS is a rolling shutter area array CMOS; the area array CMOS is mounted on both sides of the linear array CCD, and the rolling shutter direction and the sweeping direction of the area array CMOS are perpendicular to each other; the method includes:

[0005] Based on the area array CMOS image and the linear array CCD image at the same time, a joint calibration model of area array CMOS and linear array CCD is constructed.

[0006] CMOS flutter parameter curves are obtained from sequence autocorrelation CMOS images;

[0007] Based on the CMOS dizziness parameter curve and the joint calibration model of the area array CMOS-linear array CCD, the dizziness curve information of the linear array CCD is obtained;

[0008] Based on the flutter curve information of the linear CCD, the linear CCD image is restored using a steady-state re-imaging method.

[0009] The second aspect of this disclosure proposes an image restoration processing apparatus based on joint calibration of an area array CMOS and a linear array CCD, characterized in that the area array CMOS is a rolling shutter area array CMOS; the area array CMOS is mounted on both sides of the linear array CCD, and the rolling shutter direction and the sweeping direction of the area array CMOS are perpendicular to each other; the apparatus includes:

[0010] The module is used to construct a joint calibration model of area array CMOS and line array CCD based on area array CMOS images and line array CCD images at the same time.

[0011] The first acquisition module is used to obtain CMOS flutter parameter curves based on the sequence autocorrelation CMOS image;

[0012] The second acquisition module is used to obtain the linear CCD dithering curve information based on the CMOS dithering parameter curve and the area array CMOS-linear array CCD joint calibration model.

[0013] The restoration module is used to restore the linear CCD image using a steady-state re-imaging method based on the flutter curve information of the linear CCD.

[0014] A third aspect of this disclosure provides an electronic device comprising: a processor; and a memory for storing processor-executable instructions; wherein the instructions are executed by the processor to enable the processor to perform the method described in the first aspect above.

[0015] This fourth aspect of the disclosure provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the method described in the first aspect above.

[0016] According to the image restoration processing method based on joint calibration of area array CMOS and linear array CCD according to the embodiments of this disclosure, the flutter characteristics of linear array CCD under rapid maneuvering conditions can be directly and accurately sensed through the joint calibration model of area array CMOS and linear array CCD, thereby improving the accuracy of linear array CCD flutter information, providing accurate input for subsequent restoration processing, and laying the foundation for multi-mode data acquisition and high-quality ground data processing of the platform.

[0017] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this disclosure will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A schematic flowchart of an image restoration processing method based on joint calibration of area array CMOS and linear array CCD provided in an embodiment of this disclosure;

[0020] Figure 2 This is a schematic diagram of the flutter transmission model of a CMOS-linear CCD array provided in an embodiment of this disclosure;

[0021] Figure 3 A flowchart illustrating a method for obtaining CMOS chatter parameter curves provided in this embodiment of the disclosure;

[0022] Figure 4 A structural block diagram of an image restoration processing device based on joint calibration of area array CMOS and linear array CCD provided in an embodiment of this disclosure;

[0023] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0024] Embodiments of this disclosure are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0025] In recent years, the imaging quality of area-array CMOS (Complementary Metal Oxide Semiconductor) image sensors has steadily improved. They offer advantages such as low cost, high speed, on-chip data processing units, strong radiation resistance, and low power consumption, leading to their increasingly widespread application. Area-array CMOS sensors typically employ a rolling shutter (RS) method for imaging, where each row is exposed and the exposure ends sequentially at certain time intervals. Due to the different imaging times for each row, rapid relative movement between the CMOS sensor and the illuminated object causes corresponding distortion in the image, known as the RS effect. Based on this unique data acquisition method and pattern, rolling shutter area-array CMOS sensors have become a novel type of platform flutter sensing device.

[0026] This disclosure provides an image restoration processing method based on joint calibration of an area-array CMOS and a linear-array CCD. The method mounts a rolling shutter area-array CMOS and a linear-array CCD on the same focal plane. Utilizing the RS effect, the dithering error at the focal plane can be inverted and applied to the dithering error compensation of the linear-array CCD, providing crucial support for achieving multi-mode data acquisition and high-quality processing. Specifically, the image restoration processing method based on joint calibration of an area-array CMOS and a linear-array CCD according to embodiments of this disclosure is described below with reference to the accompanying drawings.

[0027] Figure 1 This is a schematic flowchart illustrating an image restoration processing method based on joint calibration of a planar CMOS array and a linear CCD array, provided in an embodiment of this disclosure. The planar CMOS array is a rolling shutter planar CMOS array, with two planar CMOS arrays mounted on either side of a linear CCD (Charge-Coupled Device). The rolling shutter direction and the sweeping direction of the planar CMOS arrays are perpendicular to each other.

[0028] It should be noted that in a rolling shutter area array CMOS, each row of pixels begins and ends exposure sequentially, with the same exposure time for each row but different imaging times. Based on this characteristic, a RS effect occurs when there is relative motion between the acquisition position and the load. The RS effect includes signals generated by factors such as internal load distortion, relative motion, and jitter. Jitter signal detection can be achieved through error separation. Therefore, in this embodiment, the rolling shutter area array CMOS is mounted on both sides of a linear CCD (Charge-Coupled Device), with the rolling shutter direction and the push-broom direction of the area array CMOS perpendicular to each other, to achieve accurate sensing and transmission of jitter signals.

[0029] In some embodiments of this disclosure, the performance parameters of the array CMOS can be set in advance based on prior knowledge of its operating characteristics and the working mechanism of the array CMOS.

[0030] Optionally, in some embodiments of this disclosure, since the flutter signal detection principle in this disclosure is mainly based on the detection of flutter characteristics using sequential autocorrelation CMOS images, it is necessary to ensure that there is overlap between the correlated images and that the overlap area is sufficiently large. As an example, assume that the CMOS window area is m rows × n columns, where m rows represent the window size in the rolling shutter direction, n columns represent the window size in the motion direction, the size of a single pixel is a, and the actual image movement velocity on the focal plane is v. img The time interval between row exposures is T. row To ensure image overlap and improve the reliability of flutter detection, the CMOS window size design can satisfy the following relationship:

[0031] mT row v img <na

[0032]

[0033] Optionally, in some embodiments of this disclosure, based on past long-term on-orbit characteristics, the flutter frequencies have been detected and analyzed to be mainly concentrated at 100Hz, 200Hz, and 300Hz. Therefore, this disclosure sets the single-line exposure time length and the window size m in the rolling shutter direction based on the above prior information and rapid maneuvering characteristics to ensure that the CMOS frame rate does not overlap with the prior knowledge frequency, thereby achieving full coverage of the flutter frequency. As an example, assume the magnitude of the prior knowledge flutter frequency is f. sat The following relationship can be satisfied:

[0034]

[0035] Optionally, in some embodiments of this disclosure, the fundamental frequency of the flutter is used as a reference. Flutter above the fundamental frequency will cause image blurring, while flutter below the fundamental frequency will cause complex image deformation. Since blur restoration is more difficult than deformation restoration, this disclosure sets the single-line exposure time length and the window size m in the roller shutter direction for cases where the flutter frequency is lower than the fundamental frequency. As an example, assuming the line exposure time of the TDICCD is T, the fundamental frequency threshold is 1 / T. The line exposure time T of the TDICCD is equal to the product of the integration time T0 and the integration series. Setting the single-line exposure time length and the window size m in the roller shutter direction can satisfy the following relationship:

[0036]

[0037] like Figure 1 As shown, the image restoration processing method based on joint calibration of area array CMOS and linear array CCD includes the following steps:

[0038] Step 101: Construct a joint calibration model of area array CMOS and line array CCD based on the area array CMOS image and the line array CCD image at the same time.

[0039] Optionally, in some embodiments of this disclosure, such as Figure 2 As shown, a detector pointing angle model is used to uniformly characterize the positional relationship between the area array CMOS and the linear array CCD in the load coordinate system, and the relative position parameters of the area array CMOS and the linear array CCD are accurately calibrated based on calibration field reference data. Dense point matching is performed between the area array CMOS image and the linear array CCD image at the same time and the calibration field reference data to obtain dense sample points covering the entire imaging field of view, and a joint calibration model of area array CMOS and linear array CCD is constructed. Under the same installation matrix conditions, the detector pointing angle coefficients of the area array CMOS and the linear array CCD are calculated using least squares, and the accuracy of the calibration results is evaluated. Using the detector pointing angle model as a connector, for imaging time t, the flutter errors in the x and y directions at the area array CMOS can be transferred to the load coordinate system, and then transferred to the linear array CCD, obtaining the accurate flutter information of the linear array CCD at that time, avoiding the coupling of flutter detection errors.

[0040] Step 102: Obtain the CMOS jitter parameter curve based on the sequence autocorrelation CMOS image.

[0041] In other words, a sequence of autocorrelation CMOS images is acquired using a rolling CMOS sensor, and dizziness is detected to obtain CMOS dizziness information. One possible implementation is to obtain the relative and absolute dizziness values ​​of the sequence of autocorrelation CMOS images. Based on the relative and absolute dizziness values, a CMOS dizziness parameter curve is obtained.

[0042] Step 103: Obtain the linear CCD chatter curve information based on the CMOS chatter parameter curve and the joint calibration model of area array CMOS-linear array CCD.

[0043] That is, based on the area array CMOS-linear array CCD joint calibration model constructed in step 101, the high-precision conversion of dizziness information is completed, and the dizziness curve information of the linear array CCD is determined according to the CMOS dizziness parameter curve.

[0044] Step 104: Based on the flutter curve information of the linear CCD, the steady-state re-imaging method is used to restore the linear CCD image.

[0045] One possible implementation involves designing a complete virtual TDI CCD linear array on the focal plane of the linear CCD. The virtual TDI CCD linear array shares a set of orbital attitude, camera focal length, and principal point parameters with the real TDI CCD. A steady-state geometric correction model is constructed based on the virtual TDI CCD linear array, the overall fitted attitude, and rigorous geometric imaging equations. A flutter geometric correction model is also constructed based on the real TDI CCD linear array, the flutter curve information of the linear CCD, and rigorous geometric imaging equations. The attitude data is fitted as a whole, and a coordinate mapping relationship between the virtual scan scene image points and the object space is established based on the steady-state geometric correction model. While ensuring consistent object-space geometric positioning accuracy, a sliding window fitting is performed on the attitude data, and a coordinate mapping relationship between the object space and the image points of the linear CCD image is established based on the flutter geometric correction model. This establishes a correspondence between the image point coordinates of the virtual scan scene and the linear CCD image. The linear CCD image is then restored through grayscale resampling to generate a flutter-compensated linear CCD image, thus achieving data restoration from motion-based imaging degradation.

[0046] Among them, steady-state re-imaging correction is performed on the platform flutter image, and modeling is carried out based on a rigorous geometric imaging equation of the probe pointing angle, which can be referred to as the following formula:

[0047]

[0048] (X g ,Y g Z g (ψ) represents the object coordinates of a ground feature point; x (s),ψ y (s) represents the pointing angle of probe s; (X gps ,Y gps Z gps ) and (B X B Y B Z ) represent the object coordinates of the ground load photography center and the GPS eccentricity error, respectively; λ represents the scaling factor; These represent the rotation matrices from the WGS84 coordinate system to the J2000 coordinate system, from the J2000 coordinate system to the platform body coordinate system, and from the body coordinate system to the load measurement coordinate system, respectively.

[0049] According to the image restoration processing method based on joint calibration of area array CMOS and linear array CCD according to the embodiments of this disclosure, the flutter characteristics of linear array CCD under rapid maneuvering conditions can be directly and accurately sensed through the joint calibration model of area array CMOS and linear array CCD, thereby improving the accuracy of linear array CCD flutter information, providing accurate input for subsequent restoration processing, and laying the foundation for multi-mode data acquisition and high-quality ground data processing of the platform.

[0050] Figure 3 A flowchart illustrating a method for obtaining CMOS chatter parameter curves provided in this embodiment of the disclosure. Figure 3 As shown, the method may include, but is not limited to, the following steps.

[0051] Step 301: Perform line-by-line image shift compensation and resampling on the sequence autocorrelation CMOS image to obtain a corrected sequence autocorrelation CMOS image.

[0052] It should be noted that in the fast maneuver mode, the area array CMOS roll-up mode can acquire continuous frame data with a certain degree of overlap. In the absence of an absolute reference, in order to obtain the absolute magnitude of the jitter signal, the distorted sequence image (sequence autocorrelation CMOS image) is first subjected to line-by-line image shift compensation and resampling.

[0053] This disclosure employs a strong correlation between frames in a sequential rolling CMOS data sequence to invert dizziness characteristics. Since the start time of each row is sequential and the exposure time is the same for each row, the image shift distance is the same for each row, and consequently, the spread function of each row's image points is consistent. Therefore, according to the derivation, the linear image shift caused by the imaging time difference has no effect on the inversion of CMOS autocorrelation dizziness parameters; thus, only the RS effect caused by linear motion needs to be compensated. As an example, assume that the imaging time of the first row of a CMOS image frame is t1, and the imaging time of the m-th row is t... n The actual image velocity on the focal plane is v img , (x n y n () indicates the image row and column number, then the grayscale value before compensation. Compared with the compensated grayscale value The following resampling relationship is satisfied:

[0054]

[0055] In addition, due to the inherent characteristics of CMOS devices, there are areas with low gray values ​​at the edges of CMOS images. To improve the matching effect between subsequent frames, it is necessary to resample the effective area of ​​each frame's data.

[0056] Step 302: Based on the block-dense matching and rigorous geometric imaging model, obtain the relative and absolute jitter of the corrected sequence autocorrelation CMOS image.

[0057] It should be noted that, based on operational engineering experience, due to the isolation provided by the flexible device, the dizziness frequency transmitted from the platform to the load is generally below 200 Hz. However, the exposure time interval for a conventional area array CMOS sensor reaches 25 microseconds, resulting in a maximum sampling frequency of 40,000 Hz. Therefore, according to the sampling theorem, this sampling frequency is well within the range required for dizziness signal detection, and a block-matching detection primitive N can be set. sat The effects of flutter on the data blocks are basically the same. Block matching detection primitive N sat The setup must consider the linear CCD's fundamental frequency and its rapid on-orbit maneuvering characteristics to ensure full coverage detection of flutter signals. As an example, the block matching detection primitive N... sat Specifically, the following relationship is satisfied:

[0058]

[0059] Optionally, based on operational engineering experience, N sat The value range can be [15, 35].

[0060] This allows for the acquisition of a sufficient number of uniformly distributed, high-precision corresponding points. This disclosure proposes a matching method combining spatial domain matching (least square matching) and frequency domain matching (phase matching) to obtain the coordinates of corresponding points between images. Theoretically, corresponding points intersect at the same point, but due to various errors, the coordinates projected onto the object point by the corresponding points will differ, i.e., relative coordinate error. Therefore, using the established rigorous imaging geometric model, the coordinates of the object point projected onto the reference elevation surface by the corresponding points can be calculated separately, obtaining the relative coordinate error. The relative coordinate error calculated by the rigorous imaging geometric model reflects the overall error in the geometric model, including load internal parameter errors, attitude observation errors, orbit observation errors, terrain data errors, and corresponding point matching errors. To obtain accurate flutter information, other errors need to be removed.

[0061] Therefore, when the flutter characteristics are uncertain, the pointing angle coefficient of the area array CMOS detector obtained in the previous embodiment can be substituted into the rigorous imaging geometry model to eliminate the interference of internal parameter errors such as CMOS lens distortion. Since the time interval between frames is short and the intersection angle is small, the influence of terrain on the relative error can be ignored. External errors such as attitude and trajectory are considered linear errors in the short term. The relative coordinate error is fitted using an affine transformation model, and the fitting residual is the nonlinear flutter relative error (flutter signal).

[0062] Step 303: Construct a spatiotemporal sensing model of flutter information based on the relative flutter amount and the absolute flutter amount.

[0063] Because the attitude angle changes continuously during rapid maneuvers, the impact of flutter on the imaging focal plane also changes accordingly, resulting in complex spatiotemporal patterns of flutter characteristics. Besides variations in the time dimension, flutter also changes spatially with the satellite's state. Traditional single models that only consider temporal characteristics are insufficient to accurately describe the spatiotemporal variations of flutter characteristics. Therefore, this disclosure proposes to use a mathematical model with both time and attitude angle as variables as the fitting model for the time-series curves of flutter characteristic parameters, i.e., a spatiotemporal sensing model for flutter information, as detailed below:

[0064] f(t)=F(ψ,ω,κ)W(t) (7)

[0065] Where f(t) represents the flutter characteristic parameters, ψ, ω, κ represent the pitch angle, roll angle and yaw angle of the satellite attitude, respectively, F(ψ, ω, κ) represents the flutter spatial characteristic function with attitude angle as independent variable, and W(t) represents the flutter temporal characteristic function with sine function as kernel function.

[0066] Step 304: Obtain the CMOS flutter parameter curve based on the spatiotemporal sensing model of flutter information.

[0067] By implementing the embodiments of this disclosure, and by processing the sequence autocorrelation CMOS image accordingly, the jitter information of the area array CMOS can be accurately perceived, and a more accurate CMOS jitter parameter curve can be obtained.

[0068] Figure 4 This is a structural block diagram of an image restoration processing apparatus based on joint calibration of a planar CMOS array and a linear CCD array, provided in an embodiment of this disclosure. The planar CMOS array is a rolling shutter planar CMOS array, mounted on both sides of the linear CCD array, with the rolling shutter direction and the sweeping direction of the planar CMOS array perpendicular to each other. Figure 4 As shown, the image restoration processing device based on joint calibration of area array CMOS and linear array CCD includes: a construction module 401, a first acquisition module 402, a second acquisition module 403, and a restoration module 404. Among them,

[0069] Module 401 is used to construct a joint calibration model of area array CMOS and line array CCD based on area array CMOS images and line array CCD images at the same time.

[0070] The first acquisition module 402 is used to obtain CMOS flutter parameter curves based on the sequence autocorrelation CMOS image.

[0071] The second acquisition module 403 is used to obtain the linear CCD dithering curve information based on the CMOS dithering parameter curve and the area array CMOS-linear array CCD joint calibration model.

[0072] The restoration module 404 is used to restore the linear CCD image using a steady-state re-imaging method based on the flutter curve information of the linear CCD.

[0073] In some embodiments of this disclosure, the construction module 401 is specifically used to: uniformly characterize the positional relationship between the array CMOS and the linear CCD in the load coordinate system using the probe pointing angle model, and accurately calibrate the relative position parameters of the array CMOS and the linear CCD based on the calibration field reference data; perform dense point matching between the array CMOS image and the linear CCD image at the same time and the calibration field reference data to obtain dense sample points covering the entire imaging field of view, and construct a joint calibration model of array CMOS and linear CCD.

[0074] In some embodiments of this disclosure, the first acquisition module 402 is specifically used to: obtain the relative jitter and absolute jitter of the sequence autocorrelation CMOS image based on the sequence autocorrelation CMOS image; construct a spatiotemporal sensing model of jitter information based on the relative jitter and absolute jitter; and obtain the CMOS jitter parameter curve based on the spatiotemporal sensing model of jitter information.

[0075] In some embodiments of this disclosure, the first acquisition module 402 is specifically used to: perform line-by-line image shift compensation and resampling on the sequence autocorrelation CMOS image to obtain a corrected sequence autocorrelation CMOS image; and obtain the relative jitter and absolute jitter of the corrected sequence autocorrelation CMOS image based on block dense matching and rigorous geometric imaging model.

[0076] In some embodiments of this disclosure, the restoration module 404 is specifically used for: designing a complete virtual TDI CCD linear array on the focal plane of the linear CCD; wherein the virtual TDI CCD linear array shares a set of orbital attitude, camera focal length, and principal point parameters with the real TDI CCD; constructing a steady-state geometric correction model based on the virtual TDI CCD linear array, the overall fitted attitude, and the rigorous geometric imaging equation, and constructing a flutter geometric correction model based on the real TDI CCD linear array, the flutter curve information of the linear CCD, and the rigorous geometric imaging equation; performing overall fitting of the attitude data, and establishing a coordinate mapping relationship between the virtual scan scene image points and the object space based on the steady-state geometric correction model; performing sliding window fitting of the attitude data while ensuring consistent object space geometric positioning accuracy, and establishing a coordinate mapping relationship between the object space and the image points of the linear CCD image based on the flutter geometric correction model, establishing a correspondence between the image point coordinates of the virtual scan scene and the linear CCD image, and performing restoration processing on the linear CCD image through grayscale resampling to generate a flutter-compensated linear CCD image.

[0077] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0078] According to the image restoration processing device based on the joint calibration of area array CMOS and linear array CCD according to the embodiments of this disclosure, the flutter characteristics of linear array CCD under rapid maneuvering conditions can be directly and accurately sensed through the joint calibration model of area array CMOS and linear array CCD, thereby improving the accuracy of linear array CCD flutter information, providing accurate input for subsequent restoration processing, and laying the foundation for multi-mode data acquisition and high-quality ground data processing of the platform.

[0079] To implement the above embodiments, this disclosure also provides an electronic device. Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this disclosure. Figure 5 As shown, the electronic device 500 may include a memory 501, a processor 502, and a computer program 503 stored in the memory 501 and executable on the processor 502. When the processor 502 executes the computer program 503, it executes the image restoration processing method based on the joint calibration of area array CMOS and linear array CCD as described in any of the above embodiments of the present disclosure.

[0080] To implement the above embodiments, this disclosure also proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the image restoration processing method based on joint calibration of area array CMOS and linear array CCD as described in any of the above embodiments of this disclosure.

[0081] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0082] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0083] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0084] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0085] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0086] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present disclosure have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An image restoration processing method based on joint calibration of area array CMOS and linear array CCD, characterized in that, The area array CMOS is a rolling shutter area array CMOS; the area array CMOS is mounted on both sides of the linear CCD, and the rolling shutter direction and the sweeping direction of the area array CMOS are perpendicular to each other. The method includes: Based on the area array CMOS image and the linear array CCD image at the same time, a joint calibration model of area array CMOS and linear array CCD is constructed. CMOS flutter parameter curves are obtained from sequence autocorrelation CMOS images; Based on the CMOS dizziness parameter curve and the joint calibration model of the area array CMOS-linear array CCD, the dizziness curve information of the linear array CCD is obtained; Based on the flutter curve information of the linear CCD, the steady-state re-imaging method is used to restore the linear CCD image. The step of constructing a joint calibration model of area array CMOS and linear array CCD based on area array CMOS images and linear array CCD images at the same time includes: The positional relationship between the area array CMOS and the linear array CCD in the load coordinate system is uniformly characterized by the probe pointing angle model, and the relative position parameters of the area array CMOS and the linear array CCD are accurately calibrated based on the calibration field reference data. The area array CMOS image and the linear array CCD image at the same time are matched with the calibration field reference data to obtain dense sample points covering the entire imaging field of view, and the area array CMOS-linear array CCD joint calibration model is constructed. The step of restoring the linear CCD image using a steady-state re-imaging method based on the linear CCD flutter curve information includes: A complete virtual TDI CCD linear array is designed on the focal plane of the linear CCD; wherein the virtual TDI CCD linear array shares a set of orbital attitude, camera focal length and principal point parameters with the real TDI CCD; A steady-state geometric correction model is constructed based on the virtual TDI CCD linear array, the overall fitted attitude, and the rigorous geometric imaging equation. A flutter geometric correction model is constructed based on the real TDI CCD linear array, the flutter curve information of the linear array CCD, and the rigorous geometric imaging equation. The attitude data is fitted as a whole, and the coordinate mapping relationship between the virtual scan scene points and the object space is established based on the steady-state geometric correction model. Under the condition of consistent object-space geometric positioning accuracy, the attitude data is fitted with a sliding window, and a coordinate mapping relationship between the object space and the image points of the linear CCD image is established based on the flutter geometric correction model. A correspondence relationship between the virtual scanning scene and the image point coordinates of the linear CCD image is established. The linear CCD image is restored by grayscale resampling to generate a flutter-compensated linear CCD image.

2. The method according to claim 1, characterized in that, The step of obtaining the CMOS dizziness parameter curve based on the sequence autocorrelation CMOS image includes: Based on the sequence autocorrelation CMOS image, the relative jitter and absolute jitter of the sequence autocorrelation CMOS image are obtained; Based on the relative flutter amount and the absolute flutter amount, a spatiotemporal perception model of flutter information is constructed; The CMOS flutter parameter curve is obtained based on the spatiotemporal sensing model of the flutter information.

3. The method according to claim 2, characterized in that, The step of obtaining the relative and absolute jitter of the sequence autocorrelation CMOS image based on the sequence autocorrelation CMOS image includes: The sequence autocorrelation CMOS image is subjected to line-by-line image shift compensation and resampling to obtain a corrected sequence autocorrelation CMOS image; Based on block-based dense matching and rigorous geometric imaging models, the relative and absolute jitter of the corrected sequence autocorrelation CMOS images are obtained.

4. An image restoration processing device based on joint calibration of area array CMOS and linear array CCD, characterized in that, The area array CMOS is a rolling shutter area array CMOS; the area array CMOS is mounted on both sides of the linear CCD, and the rolling shutter direction and the sweeping direction of the area array CMOS are perpendicular to each other. The device includes: The module is used to construct a joint calibration model of area array CMOS and line array CCD based on area array CMOS images and line array CCD images at the same time. The first acquisition module is used to obtain CMOS flutter parameter curves based on the sequence autocorrelation CMOS image; The second acquisition module is used to obtain the linear CCD dithering curve information based on the CMOS dithering parameter curve and the area array CMOS-linear array CCD joint calibration model. The restoration module is used to restore the linear CCD image using a steady-state re-imaging method based on the flutter curve information of the linear CCD. The construction module is specifically used to: uniformly characterize the positional relationship between the area array CMOS and the linear array CCD in the load coordinate system using the probe pointing angle model, and accurately calibrate the relative position parameters of the area array CMOS and the linear array CCD based on the calibration field reference data; perform dense point matching between the area array CMOS image and the linear array CCD image at the same time and the calibration field reference data to obtain dense sample points covering the entire imaging field of view, and construct the area array CMOS-linear array CCD joint calibration model; The restoration module is specifically used for: designing a complete virtual TDI CCD linear array on the focal plane of the linear CCD; wherein the virtual TDI CCD linear array shares a set of orbital attitude, camera focal length, and principal point parameters with the real TDI CCD; constructing a steady-state geometric correction model based on the virtual TDI CCD linear array, the overall fitted attitude, and the rigorous geometric imaging equation, and constructing a flutter geometric correction model based on the real TDI CCD linear array, the flutter curve information of the linear CCD, and the rigorous geometric imaging equation; performing overall fitting of the attitude data, and establishing a coordinate mapping relationship between the virtual scan scene image points and the object space based on the steady-state geometric correction model; performing sliding window fitting of the attitude data while ensuring consistent object space geometric positioning accuracy, and establishing a coordinate mapping relationship between the object space and the image points of the linear CCD image based on the flutter geometric correction model, establishing a correspondence between the image point coordinates of the virtual scan scene and the linear CCD image, and performing restoration processing on the linear CCD image through grayscale resampling to generate a flutter-compensated linear CCD image.

5. The apparatus according to claim 4, characterized in that, The first acquisition module is specifically used for: Based on the sequence autocorrelation CMOS image, the relative jitter and absolute jitter of the sequence autocorrelation CMOS image are obtained; Based on the relative flutter amount and the absolute flutter amount, a spatiotemporal perception model of flutter information is constructed; The CMOS flutter parameter curve is obtained based on the spatiotemporal sensing model of the flutter information.

6. The apparatus according to claim 5, characterized in that, The first acquisition module is specifically used for: The sequence autocorrelation CMOS image is subjected to line-by-line image shift compensation and resampling to obtain a corrected sequence autocorrelation CMOS image; Based on block-based dense matching and rigorous geometric imaging models, the relative and absolute jitter of the corrected sequence autocorrelation CMOS images are obtained.

7. An electronic device, characterized in that, include: processor; Memory for storing processor-executable instructions; wherein the instructions are executed by the processor to enable the processor to perform the method of any one of claims 1-3.

8. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method of any one of claims 1-3.

Images