TDI camera flutter degraded image restoration method

By constructing one-dimensional and two-dimensional equivalent flutter curves for geometric distortion pre-correction and combining them with a lightweight neural network training dataset, the image blurring problem caused by flutter in TDI camera imaging was solved, achieving efficient and accurate image restoration and improving the quality of remote sensing images.

CN118918044AActive Publication Date: 2024-11-08ZHEJIANG UNIV +1

Patent Information

Application Number
CN202410950060.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08
Estimated Expiration
2044-07-16

AI Technical Summary

Technical Problem

Existing technologies suffer from image blurring and image quality degradation due to flutter in TDI camera imaging, especially in remote sensing image processing where there is a lack of effective datasets and deep learning methods, making it difficult to accurately restore high-quality images.

Method used

By constructing one-dimensional and two-dimensional equivalent flutter curves, geometric distortion pre-correction is performed, and combined with a lightweight neural network training dataset, fine image restoration is achieved.

Benefits of technology

It improves the resolution and imaging quality of remote sensing images, provides targeted training data, and enhances the image restoration effect of deep learning methods.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918044A_ABST
    Figure CN118918044A_ABST
Patent Text Reader

Abstract

The invention discloses a TDI camera flutter degraded image restoration method. The method comprises the following steps: firstly, constructing a one-dimensional equivalent flutter curve based on a flutter curve of a TDI camera; thirdly, converting the one-dimensional equivalent flutter curve into a two-dimensional equivalent flutter curved surface; then, geometric distortion pre-correction is carried out on the actually shot TDI degraded image by using the two-dimensional equivalent flutter curved surface, and a TDI degraded image after pre-correction is obtained; constructing a simulated TDI degraded image data set, and training a neural network by using the pre-corrected simulated degraded image to obtain a flutter degraded image restoration network; and finally, restoring the actually shot and pre-corrected TDI degraded image by using the flutter degraded image restoration network to obtain a corresponding restored image. According to the method, modeling simulation and image restoration can be efficiently and accurately carried out on the flutter degradation of TDI camera push-broom imaging, so that the resolution and the imaging quality of remote sensing images are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to a remote sensing image quality improvement method in the field of computational imaging, and specifically relates to a chatter-degraded image restoration method for a time delayed and integration (TDI) camera. Background Art

[0002] In the field of high-resolution Earth imaging observations by optical remote sensing satellites, particularly in sun-synchronous and very low orbits, TDI cameras are primarily used as imaging sensors. Due to their temporal integration characteristics, TDI cameras offer a higher signal-to-noise ratio, making them suitable for high-resolution imaging under steady-state relative motion and low-light conditions. However, the key to high-quality TDI imaging lies in matching the relative motion between the imaging target and the device with the speed of charge transfer within the TDI detector. However, the platform inevitably experiences non-steady-state vibrations, which can lead to motion mismatch during imaging, resulting in blurred and distorted images and failure to meet the designed resolution and imaging quality requirements. Post-processing is necessary to further eliminate degradation effects and improve image quality.

[0003] Traditional image restoration methods use the point spread function (PSF) as prior information to reconstruct a clear image. However, these methods fail to account for inter-row overlap, misalignment, and content distortion. They suffer from the drawbacks of calculating and restoring the PSF row by row, resulting in high computational complexity. Furthermore, the restoration results are susceptible to PSF reconstruction errors. In recent years, deep learning has made significant progress in the field of image restoration, demonstrating its effectiveness in addressing image degradation issues such as blur and noise. However, deep learning methods require large amounts of targeted training data, which is relatively difficult to obtain.

[0004] A series of research efforts have been conducted both domestically and internationally on flutter degradation simulation and image restoration techniques for optical remote sensing low-orbit push-broom imaging. Traditional remote sensing image degradation simulation and restoration methods use blur kernels or analytical modulation transfer functions to process the entire image. For example, Wahballah et al. proposed a method that uses the pointing accuracy and angular velocity of a TDI camera to construct the MTF of the entire image. However, the blur of degraded TDI images is typically spatially variable, so these methods require improvement. Wang Mi et al. from Wuhan University proposed a CDSM model to model push-broom imaging. They determined the number of subdivisions in the CDSM (Continuous Dynamic Shooting Model) using an integral transfer function, achieving more accurate PSF estimation. They then combined this with Wiener filtering for context-based adaptive image restoration. Liu Shijie et al. from Tongji University built on this work by performing PSF compensation, improving restored image quality. Researchers at Jilin University proposed a vibration modulation transfer function based on statistical properties, applying a different vibration MTF to each row to simulate spatially variable image degradation. Researchers from Harbin Institute of Technology and Zhejiang University proposed a row-by-row PSF convolution process. For each time series, the PSF at that moment must first be estimated using the relative motion velocity at that moment. The blurred rows of different time series are then superimposed to simulate the charge accumulation process. This method can more accurately simulate the image degradation process caused by vibration, but it is difficult to analytically calculate the degraded PSF when the relative motion is more complex. The image degradation simulation methods mentioned above are all based on PSF convolution. They lack consideration for inter-row overlap, misalignment, and content deformation. They have the disadvantages of row-by-row PSF calculation and restoration, which is computationally intensive. Furthermore, the restoration results are susceptible to PSF reconstruction errors.

[0005] With the rapid development of artificial intelligence, many research teams are committed to using deep learning to solve image restoration problems. For example, Syed Waqas Zamir and others from the Origin Artificial Intelligence Research Institute proposed Restormer, which uses a multi-head attention mechanism and feedforward neural networks to achieve long-range pixel interaction and the ability to process large images. Sun Jian and others from Kuangshi University proposed NAFNet, an activation-free network based on Unet, introducing LayerNorm, simplified gating units, and a simplified channel attention mechanism. NAFNet eliminates redundant attention and activation function designs and boasts fast training convergence, a small number of network parameters, and a low computational load. While learning-based image restoration methods have achieved promising results, these methods often perform restoration on area array cameras, primarily considering blur caused by camera shake rather than the degradation factors of TDI cameras. Summary of the Invention

[0006] To address the problems and needs in the background art, the present invention provides a TDI camera flutter degradation simulation and image restoration method. Through push-broom imaging flutter degradation modeling and simulation, image geometric distortion pre-correction based on flutter measurement data, and high-efficiency image fine restoration based on lightweight neural networks, the present invention addresses issues such as rapid simulation of push-broom imaging, dataset construction, and high-quality image restoration, further improving the quality of low-orbit push-broom imaging. The present invention's in-depth research and modeling simulation of push-broom imaging flutter degradation mechanisms will facilitate more accurate restoration of degraded images and provide a data and methodological foundation for the application of deep learning in remote sensing.

[0007] The technical solutions of the present invention are as follows:

[0008] 1. A TDI Camera Flutter-Degraded Image Restoration Method

[0009] (1) Construct a one-dimensional equivalent chatter curve based on the chatter curve of the TDI camera;

[0010] (2) Converting the one-dimensional equivalent flutter curve into a two-dimensional equivalent flutter surface;

[0011] (3) Using a two-dimensional equivalent flutter surface to pre-correct the geometric distortion of the actually captured TDI degraded image, the actual TDI degraded image after pre-correction is obtained;

[0012] (4) Using the pre-corrected TDI degradation simulation image dataset to train a neural network to obtain a chatter degradation image restoration network;

[0013] (5) The pre-corrected actual TDI degraded image is restored using the flutter-degraded image restoration network to obtain the corresponding restored image.

[0014] In (1), for the dither curve in the x-direction or y-direction, the dither curves of different levels at each pixel position are accumulated and sampled within each level exposure time to obtain a one-dimensional equivalent dither curve describing the offset of the current pixel position during the entire push-scan imaging process.

[0015] Said (2) is specifically:

[0016] After combining the one-dimensional equivalent chatter curves in the x-direction and the y-direction, the pixel chatter state is replicated and extended perpendicular to the push-scan direction based on the combined pixel chatter state to obtain the equivalent chatter surface describing the position offset of each pixel in the two-dimensional push-scan image.

[0017] In (3), the negative value of the equivalent flutter surface is used as the flutter correction optical flow, and the flutter correction optical flow is used to perform geometric distortion pre-correction on the actually captured TDI degraded image.

[0018] In (4), the steps of constructing the pre-corrected TDI degraded image dataset include:

[0019] For the aerial image dataset, first, each image in the aerial image dataset is converted into a grayscale image and then converted to the brightness domain to obtain a brightness domain image; then, a flutter degradation simulation is performed on each brightness domain image, and then a geometric distortion pre-correction is performed to obtain a pre-corrected TDI degradation simulation image and the corresponding equivalent flutter surface. All images are traversed and processed to obtain a pre-corrected TDI degradation simulation image dataset.

[0020] After performing the flutter degradation simulation and pre-correction on each brightness domain image, a pre-corrected TDI degradation simulation image and a corresponding equivalent flutter surface are obtained, specifically:

[0021] Firstly, flutter curves in the x-direction and y-direction are randomly generated and recorded as simulated flutter curves. Then, corresponding one-dimensional equivalent flutter curves are constructed based on the simulated flutter curves. Then, the one-dimensional equivalent flutter curves are converted into two-dimensional equivalent flutter surfaces. Next, each brightness domain image is positionally resampled to obtain a flutter degradation simulation image. The two-dimensional equivalent flutter surface is used to perform geometric distortion pre-correction on the flutter degradation simulation image to obtain a pre-corrected TDI degradation simulation image.

[0022] In (4) and (5), the input of the flutter-degraded image restoration network is the pre-corrected TDI-degraded simulation image and the pre-corrected actual TDI-degraded image, or the pre-corrected TDI-degraded simulation image and the equivalent flutter surface and the pre-corrected actual TDI-degraded image and the equivalent flutter surface.

[0023] 2. A computer device

[0024] The device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0025] 3. A computer-readable storage medium

[0026] The medium stores a computer program, which implements the steps of the method when executed by a processor.

[0027] 4. A computer program product

[0028] The product comprises a computer program / instructions which, when executed by a processor, implement the steps of the method.

[0029] The beneficial effects of the present invention are:

[0030] The present invention can efficiently and accurately model and simulate the flutter degradation of TDI camera push-scan imaging, providing a large amount of targeted training data for deep learning methods.

[0031] The present invention can effectively improve the resolution and imaging quality of remote sensing images by combining distortion pre-correction and neural network fine restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Schematic diagram of the TDI camera vibration degradation simulation and image restoration technology process.

[0033] Figure 2 Schematic diagram of push-scan imaging of the TDI camera.

[0034] Figure 3 The calculation process of the equivalent flutter curve, where (a) is the flutter curve and (b) is the equivalent flutter model.

[0035] Figure 4 The equivalent chatter curve is highly robust to measurement noise, where (a) is the ideal chatter curve, (b) is the chatter curve with noise, (c) is the equivalent chatter curve of (a), and (d) is the equivalent chatter curve of (b).

[0036] Figure 5 Equivalent flutter surface and flutter degradation simulation process.

[0037] Figure 6 Schematic diagram of position resampling.

[0038] Figure 7 are the simulated degraded images with different TDI integration levels, where (a) is the original image, (b) is the simulated degraded image with integration level k=1, (c) is the simulated degraded image with integration level k=16, and (d) is the simulated degraded image with integration level k=128.

[0039] Figure 8 Simulated degradation images with different chatter amplitudes, where (a) is a simulated degradation image with a chatter amplitude of 1 pixel, (b) is a simulated degradation image with a chatter amplitude of 7 pixels, (c) is a simulated degradation image with a chatter amplitude of 12 pixels, and (d) is a simulated degradation image with a chatter amplitude of 16 pixels.

[0040] Figure 9 These are the simulated degradation images of different chatter frequencies, where (a) is the simulated degradation image with a chatter frequency of 30 pixels, (b) is the simulated degradation image with a chatter frequency of 100 pixels, (c) is the simulated degradation image with a chatter frequency of 300 pixels, and (d) is the simulated degradation image with a chatter frequency of 1000 pixels.

[0041] Figure 10Distortion pre-correction and fine image restoration, where (a) is a flow chart of distortion pre-correction and fine image restoration, (b) is a process diagram of distortion pre-correction, and (c) is a neural network structure diagram in fine image restoration (the input is a degraded image).

[0042] Figure 11 This is another structural diagram of a fine image restoration network (the input is an equivalent chatter surface and a degraded image, and the geometric distortion pre-correction stage is integrated into the image restoration network).

[0043] Figure 12 These are the restoration results of low-frequency chatter and distorted degraded images, where (a) is the reference image, (b) is the low-frequency chatter and distorted degraded image, (c) is the restoration result of the MIRNetV2 method, and (d) is the restoration result of the present invention.

[0044] Figure 13 These are the restoration results of high-frequency chatter blurred and degraded images, where (a) is the reference image, (b) is the high-frequency chatter distorted and degraded image, (c) is the restoration result of the MIRNetV2 method, and (d) is the restoration result of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the present invention and thus to more clearly define the scope of protection claimed by the present invention, the present invention is described in detail below with respect to certain specific embodiments and drawings of the present invention. It should be noted that the following are only certain specific implementation methods of the present invention, which are only part of the embodiments of the present invention, wherein the specific and direct description of the relevant structures is only for the convenience of understanding the present invention, and the specific features do not naturally and directly limit the scope of implementation of the present invention. The conventional selections and replacements made by those skilled in the art under the guidance of the present invention, as well as the reasonable arrangement and combination of several technical features under the guidance of the present invention, should all be deemed to be within the scope of protection claimed by the present invention.

[0046] In order to solve the problem of image quality degradation of TDI camera caused by vibration and improve the push-scan imaging quality of TDI camera, the present invention discloses a method for restoring image degraded by vibration of TDI camera, such as Figure 1 As shown in the figure, it includes three parts: "TDI flutter degradation modeling", "TDI flutter degradation image simulation", and "TDI flutter degradation image restoration". The specific implementation includes the following steps:

[0047] (1) Construct a one-dimensional equivalent chatter curve based on the chatter curve of the TDI camera;

[0048] For a TDI camera that includes a gyroscope (or other posture measurement device), the gyroscope measures the angular velocity ω around the x-axis at time t. x(t), the angular position of the optical axis relative to the x direction at the starting moment θ x (0) = 0, the exposure time is T, and the optical axis swing angle position at time t (t1≤t≤t1+T) is:

[0049]

[0050] Among them, ω x (t′) is the angular velocity at time t′.

[0051] set up Among them, θ x (t) is the angular position of the optical axis relative to the x direction at time t, A i is the amplitude, f i is the frequency, For phase, such as Figure 2 As shown in the figure, the object-image relationship during the imaging process, the measured chatter curve and the displacement of the pixel plane on the sensor are:

[0052]

[0053] Among them, J x (t) is the chatter curve in the x direction, l is the image distance, and P is the pixel size.

[0054] For a TDI camera with K integration levels, different levels start recording at different timings, but share a common dither curve. Based on this characteristic, an equivalent dither curve can be calculated. For the dither curve in the x-direction or y-direction, the dither curves of different levels at each pixel position are accumulated and sampled within the exposure time of each level to obtain a one-dimensional equivalent dither curve describing the current pixel position offset during the entire push-scan imaging process, as shown in the figure: Figure 3 The calculation formula of the one-dimensional equivalent flutter curve is:

[0055]

[0056] Where X(t) and Y(t) are the equivalent flutter curves in the x-direction and y-direction, respectively. x is the chatter curve in the x direction, J y is the chatter curve in the y direction, t represents the time, K is the integration level, M is the number of sampling subdivisions of the chatter curve within each level exposure time, τ is the single exposure time, k is the level number, and m is the sampling number.

[0057] The equivalent chatter curve can resist the influence of chatter measurement noise. Figure 4 As shown in FIG, in the case of measurement noise, the equivalent flutter curve with noise is consistent with the ideal equivalent flutter curve, and the noise does not have a significant impact on the equivalent flutter curve.

[0058] (2) Converting the one-dimensional equivalent flutter curve into a two-dimensional equivalent flutter surface;

[0059] (2) Specifically:

[0060] Since the linear arrays of the same level have the same vibration state in the push-sweep direction during the push-sweep process, and the dither state of each pixel in the linear imaging array is the same, the state of each pixel position can be obtained by copying the dither curve. After combining the one-dimensional equivalent dither curves in the x-direction and y-direction, the pixel dither state is copied and extended perpendicular to the push-sweep direction based on the combined pixel dither state to obtain an equivalent dither surface that describes the offset of each pixel position in the two-dimensional push-sweep image. In this embodiment, the equivalent dither curves in the x-direction and y-direction accumulated at k levels are combined and copied h times, where h is the number of pixels perpendicular to the push-sweep direction. The equivalent dither surface F(x,y) that describes the offset of each pixel position (x,y) in the two-dimensional push-sweep image is obtained:

[0061] F(x,y)=concat{repeat[X(t),h],repeat[Y(t),h]}

[0062] Here, "concat" represents concatenation of two-dimensional tensors in the third dimension, and "repeat[X(t),h]" replicates the one-dimensional curve X(t) h times perpendicular to the push-sweep direction. X(t) and Y(t) are the equivalent flutter curves in the x and y directions, respectively, and t is a positive integer. The number of pixels parallel to the push-sweep direction is determined by the push-sweep duration. To facilitate image processing, the image can be cropped into a rectangular shape along the push-sweep direction, and the equivalent flutter surface is also cropped to the corresponding size. Figure 5 A schematic diagram of the equivalent flutter surface is included.

[0063] (3) Taking the negative value of the equivalent flutter surface as the flutter correction optical flow, the flutter correction optical flow is used to pre-correct the geometric distortion of the actually captured TDI degraded image to obtain the actual TDI degraded image after pre-correction;

[0064] (4) Using the pre-corrected TDI degradation simulation image dataset to train a neural network to obtain a chatter degradation image restoration network;

[0065] In (4), the steps for constructing the pre-corrected TDI degradation simulation image dataset include:

[0066] For high-quality aerial image datasets, first, each high-resolution aerial RGB image in the aerial image dataset is converted into a grayscale image, and then inverse gamma correction is applied to the image before converting it to the luminance domain to obtain a luminance domain image; then, after flutter degradation simulation and geometric distortion pre-correction are performed on each luminance domain image, the pre-corrected TDI degradation simulation image and the corresponding equivalent flutter surface are obtained, such as Figure 5 As shown in FIG, all aerial RGB images are traversed and processed, and the aerial RGB images are used as true value images to obtain the pre-corrected TDI degradation simulation image dataset.

[0067] After performing flutter degradation simulation and pre-correction on each brightness domain image, the pre-corrected TDI degradation simulation image and the corresponding equivalent flutter surface are obtained, specifically:

[0068] Firstly, random jitter curves in the x- and y-directions are generated to simulate the jitter state of the TDI camera, which are recorded as simulated jitter curves. Then, the corresponding one-dimensional equivalent jitter curves are constructed based on the simulated jitter curves. The one-dimensional equivalent jitter curves are then converted into two-dimensional equivalent jitter surfaces. Then, each brightness domain image is positionally resampled to obtain a jitter-degraded simulated image. The geometric distortion of the jitter-degraded simulated image is pre-corrected using the two-dimensional equivalent jitter surface to obtain the pre-corrected TDI degraded image.

[0069] The specific mathematical expression is as follows: the flutter of each level of the linear array is considered independently, and the k-th level equivalent flutter curve x in the x and y directions of fine sampling is considered. k (t), y k (t) is:

[0070]

[0071] Where τ is the single exposure time. The x and y direction curves are combined into the k-th level two-dimensional chatter degradation information f describing the pixel position offset k (x,y):

[0072] f k (x,y)=concat{repeat[x k (t),h],repeat[y k (t),h]}

[0073] h is the number of pixels of the TDI linear array (perpendicular to the push-scan direction);

[0074] The k-th level degraded image is:

[0075] B k =grid_sample(I,f k )+n k

[0076] Among them, B k is the fuzzy graph of the single-level integral, n k is the noise, I is the brightness domain image. grid_sample is the position resampling, which means the two-dimensional dither offset information f k The pixel positions of the clear image I are resampled. Figure 6 This is a schematic diagram of the position resampling process. The black dots represent pixel locations and the boxes represent pixels.

[0077] By resampling the pixel position, the intensity value of the original pixel is transferred to the sampled pixel position. The TDI push-scan image is the accumulation of each level, and the final degraded image is:

[0078]

[0079] According to the above formula, the image degradation caused by factors such as imaging platform vibration amplitude, frequency, TDI level, exposure time and noise can be simulated. Figure 7 (a) Figure 7 (b) Figure 7 (c) and Figure 7 (d) Figure 8 (a) Figure 8 (b) Figure 8 (c) and Figure 8 (d) and Figure 9 (a) Figure 9 (b) Figure 9 (c) and Figure 9 (d) are the simulation results of the chatter degradation images under different parameter settings.

[0080] Figure 6 、 Figure 10 (a) and Figure 10 (b) shows this process. With the help of the vibration correction optical flow, the distortion of the image is initially repaired, but there are still burrs or blurred degradation in the image details.

[0081] (5) According to the actual TDI degraded image after pre-correction, the image is restored using the vibration-degraded image restoration network to obtain the corresponding restored image.

[0082] In (4) and (5), the input of the chatter-degraded image restoration network is the pre-corrected actual TDI-degraded image, or the pre-corrected actual TDI-degraded image and the equivalent chatter surface.

[0083] When the input of the chatter-degraded image restoration network is only the pre-corrected TDI-degraded image, the available network structures are as follows: Figure 10 As shown in (c).

[0084] When the input of the chatter-degraded image restoration network is the pre-corrected TDI-degraded image and the equivalent chatter surface, the geometric distortion pre-correction stage is integrated into the image restoration network. The available network structures are as follows: Figure 11 shown.

[0085] For any image restoration neural network, the simulated pre-corrected TDI degraded image B' and the equivalent chatter surface F are taken as input, the clear image without degradation I is taken as the target, the restoration parameter is θ, and the output of the restoration network is Network() is the image restoration neural network. The optimization problem can be expressed as:

[0086]

[0087] in The loss function is used to optimize the parameters of the restoration network by minimizing the above loss function so that it can generate a restoration result as close as possible to the target clear image I on the input degraded image.

[0088] The running time of the method of the present invention is tested on a single TDI degraded image of size 1024x1024. Table 1 shows the results on a single RTX 3090 GPU, and the results are averaged over 10 images. Figure 10 The method presented here is carried out in stages 1 and 2, respectively. Stage 1 represents the pre-correction stage, and Stage 2 represents the fine restoration stage. It can be seen that the pre-correction stage takes about 0.13 seconds, while the total time is about 1 second, demonstrating the efficiency of the proposed method.

[0089] Table 1 is the two-stage running time statistics

[0090] Stage 1 Stage 2 Total time Running time (unit: seconds) 0.13 1.05 1.18

[0091] The present invention uses PSNR and SSIM parameters on simulated flutter-degraded images to measure the restoration effectiveness of the present method. Tables 2 and 3 show the restoration results for TDI levels of 1 and 128, respectively. It can be seen that in both cases, the present method achieves high objective parameters. Figure 12 (a)- Figure 12 (d) and Figure 13 (a)- Figure 13 (d) are all visual restoration results. It can be seen that the restoration results of the method proposed by the present invention are more detailed. The above results also prove the effectiveness of the method of the present invention.

[0092] Table 2 shows the restoration parameters of the chatter-degraded image with TDI level 1.

[0093] PSNR(dB) SSIM Degraded image 13.57 0.0814 Restored image 39.02 0.9493

[0094] Table 3 shows the restoration parameters of the chatter-degraded image with a TDI level of 128.

[0095] PSNR(dB) SSIM Degraded image 13.66 0.0904 Restored image 36.52 0.9072

[0096] Finally, it should be noted that the above embodiments and explanations are intended only to illustrate the technical solutions of the present invention and are not intended to limit the present invention. It should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of the present invention may be made without departing from the spirit and scope of the technical solutions disclosed herein, and all such modifications or equivalent substitutions shall be encompassed within the scope of protection of the claims of the present invention.

Claims

1. A method for restoring a TDI camera flutter-degraded image, characterized in that: The following steps are involved: (1) Construct a one-dimensional equivalent chatter curve based on the chatter curve of the TDI camera; (2) Converting the one-dimensional equivalent flutter curve into a two-dimensional equivalent flutter surface; (3) Pre-correcting the geometric distortion of the actually captured TDI degraded image using a two-dimensional equivalent flutter surface to obtain the actual TDI degraded image after pre-correction; (4) Using the pre-corrected TDI degradation simulation image dataset to train the neural network, a flutter degradation image restoration network is obtained; (5) The pre-corrected actual TDI degraded image is restored using the flutter-degraded image restoration network to obtain the corresponding restored image.

2. The method for restoring a TDI camera flutter-degraded image according to claim 1, characterized in that: In (1), for the dither curve in the x-direction or y-direction, the dither curves of different levels at each pixel position are accumulated and sampled within each level of exposure time to obtain a one-dimensional equivalent dither curve describing the offset of the current pixel position during the entire push-scan imaging process.

3. The method for restoring a TDI camera flutter-degraded image according to claim 1, characterized in that: The above (2) is specifically: After combining the one-dimensional equivalent dither curves in the x-direction and the y-direction, the pixel dither state is then copied and extended along the direction perpendicular to the push-scan direction based on the combined pixel dither state to obtain an equivalent dither surface describing the position offset of each pixel in the two-dimensional push-scan image.

4. The method for restoring a TDI camera flutter-degraded image according to claim 1, characterized in that: In the above (3), the negative value of the equivalent flutter surface is used as the flutter correction optical flow, and the flutter correction optical flow is used to perform geometric distortion pre-correction on the actually captured TDI degraded image.

5. The method for restoring a TDI camera flutter-degraded image according to claim 1, characterized in that: In (4), the step of constructing the pre-corrected TDI degraded image dataset includes: For the aerial image dataset, first, each image in the aerial image dataset is converted into a grayscale image and then converted to the brightness domain to obtain a brightness domain image; then, a flutter degradation simulation is performed on each brightness domain image, and then a geometric distortion pre-correction is performed to obtain a pre-corrected TDI degradation simulation image and the corresponding equivalent flutter surface, and all images are traversed to obtain a pre-corrected TDI degradation simulation image dataset.

6. The method for restoring a TDI camera flutter-degraded image according to claim 5, characterized in that: After performing the flutter degradation simulation and pre-correction on each brightness domain image, a pre-corrected TDI degradation simulation image and a corresponding equivalent flutter surface are obtained, specifically: Firstly, the flutter curves in the x-direction and y-direction are randomly generated and recorded as the simulated flutter curves. Then, the corresponding one-dimensional equivalent flutter curves are constructed based on the simulated flutter curves. Then, the one-dimensional equivalent flutter curves are converted into two-dimensional equivalent flutter surfaces. Then, each brightness domain image is positionally resampled to obtain the flutter degradation simulation image. The two-dimensional equivalent flutter surface is used to perform geometric distortion pre-correction on the flutter degradation simulation image to obtain the pre-corrected TDI degradation simulation image.

7. The method for restoring a TDI camera flutter-degraded image according to claim 1, characterized in that: In (4) and (5), the input of the flutter-degraded image restoration network is the pre-corrected TDI-degraded simulation image and the pre-corrected actual TDI-degraded image, or the pre-corrected TDI-degraded simulation image and the equivalent flutter surface and the pre-corrected actual TDI-degraded image and the equivalent flutter surface.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • TDI flutter image restoration method based on image block adaptive adjustment

    CN107292839A

  • Flutter change simulation method and system for remote sensing image

    CN113870321A

  • Remote sensing image restoration method based on flutter precision detection estimation

    CN114372511A

  • Image restoration processing method based on combined calibration of area array CMOS and linear array CCD

    CN116242392A

  • Method of jitter detection and image restoration for high-resolution TDI CCD satellite images

    US20210104022A1

Cited By

  • Multi-aperture imaging optical system and co-phase correction method thereof

    CN119376098A