A cascaded adaptive optics image blind restoration method

Abstract

The present invention discloses a cascaded adaptive optical image blind restoration method, which is applied to the blind restoration of multi-frame images of ground-based telescopes. The method addresses the problem that the existing technology cannot robustly reconstruct high-resolution adaptive optical images when the data quality fluctuates greatly. The present invention first sends every five consecutive clear-degraded image pairs into a cascade network comprising four image restoration units, and every three adjacent image pairs into three image restoration units in the first stage. Pixel-level optical flow alignment is first performed in each image restoration unit. A spatiotemporal attention mechanism module is then used to redistribute feature weights. After extracting clear pixel blocks using a temporal sharpness prior, the two output parts are sent to an image reconstruction model. A hard case mining strategy is used in the loss function to focus on difficult-to-restore areas. Finally, the restored three frames of clear latent images are sent to the image restoration unit in the second stage to obtain the final restored clear adaptive optical image.

Description

Technical Field

[0001] The present invention belongs to the technical field of adaptive optical image processing, and in particular relates to an adaptive optical image blind restoration technology based on a cascade architecture. Background Art

[0002] Atmospheric turbulence can cause distortion in the wavefront of light waves, severely impacting ground-based telescopes' ability to observe targets. This can cause the imaging resolution of ground-based telescopes to fall far below the expected theoretical diffraction limit, severely impacting image quality. To alleviate these issues, ground-based telescopes often use adaptive optics to compensate for the image degradation caused by atmospheric turbulence. Adaptive optics systems use a wavefront sensor to measure atmospheric disturbances in real time and a deformable mirror to compensate for the wavefront distortion caused by atmospheric turbulence in real time. Adaptive optics can significantly reduce low-order aberrations, effectively improving the imaging quality of optical systems.

[0003] Due to the limitations of the adaptive optics system's own hardware, the correction of wavefront distortion by the adaptive optics system is partial and incomplete, and a large amount of high-frequency information of the target imaging is lost. In the field of image processing, this image degradation process is usually described as:

[0004] g(x,y)=f(x,y)*h(x,y)+n(x,y)

[0005] Where g(x,y) is the output image, f(x,y) is the input image, h(x,y) is the point spread function, n(x,y) is the additive noise, and “*” is the two-dimensional circular convolution. The process of estimating the potential clear image based on the degraded image g(x,y) is called image deconvolution or restoration. If the point spread function is unknown, it is called blind image deconvolution or blind image restoration.

[0006] To further improve the quality of adaptive optics images, post-processing is necessary. Currently, there are four main methods for adaptive optics image post-processing: speckle imaging, phase difference methods, blind deconvolution algorithms, and deep learning-based methods. Speckle imaging utilizes statistical information from atmospheric turbulence to reconstruct its phase and amplitude, typically requiring hundreds of short-exposure images for a single reconstruction. Phase difference methods use the image intensities of a set of focused and defocused images to establish an error metric, minimizing the error metric to reconstruct the target and point spread function. However, this method requires additional imaging equipment, and the algorithm is sensitive to system parameters, presenting several technical difficulties in practical applications. Blind deconvolution algorithms simultaneously reconstruct the target and point spread function from a single or several blurred image frames. However, blind deconvolution is often an ill-posed inverse problem, requiring a large amount of prior information to constrain the algorithm's solution; otherwise, ideal processing results are difficult to achieve. Deep learning-based methods utilize supervised learning, using a large set of blurred-clear image datasets to train a deep neural network. This deep neural network learns the nonlinear mapping relationship between blurred and clear images, ultimately restoring the clear image using the trained network. Existing deep learning-based blind restoration methods for adaptive optical images, such as the EDDNN algorithm based on a codec structure (Asensio Ramos A, de la Cruz Rodríguez J, Yabar AP. Real-time, multiframe, blind deconvolution of solar images [J]. Astronomy & Astrophysics, 2018, 620: A73.) and the CSSTN algorithm based on adaptive filters (Wang S, Chen Q, He C, et al. Blind restoration of solar images via the channel sharing spatiotemporal network [J]. Astronomy & Astrophysics, 2021, 652: A50.), are generally unable to robustly reconstruct high-resolution adaptive optical images when data quality fluctuates greatly. Therefore, it is necessary to construct a method that can stably restore high-resolution adaptive optical images even when image quality is severely degraded. Summary of the Invention

[0007] The technical problem to be solved by the present invention is: how to stably and robustly restore each frame of short-exposure adaptive optical image, thereby achieving high-reliability high-resolution adaptive optical image reconstruction in actual astronomical observation scenarios.

[0008] The technical solution adopted by the present invention is: a cascaded adaptive optical image blind restoration method, based on a cascade network including four image restoration units, which are divided into two stages, wherein the first stage includes three image restoration units and the second stage includes one image restoration unit; each image restoration unit has the same structure, including: a pixel-level explicit optical flow alignment module, a spatiotemporal attention mechanism module, a temporal sharpness prior module, and an image reconstruction module;

[0009] The implementation process of the method is:

[0010] S1. Collect multiple frames of adaptive optics degraded images and corresponding clear reference images, and divide the dataset consisting of the multiple frames of adaptive optics degraded images and the corresponding clear reference images into a training set, a validation set, and a test set; the multiple frames of adaptive optics degraded images are adjacent frames collected in time sequence; the corresponding clear reference images are images reconstructed using an image restoration method using any number of degraded frames as input;

[0011] S2. During the training, validation, and testing processes, the data in the training set, validation set, or test set each uses five consecutive frames of adaptive optical degraded images as input to the cascade network. Three adjacent frames of these five consecutive adaptive optical degraded images are fed into the three image restoration units of the first stage to recover one clear latent image. These five consecutive frames of adaptive optical degraded images include three groups of three adjacent frames of adaptive optical degraded images, thereby recovering three clear latent images. The processing process of each image restoration unit is as follows:

[0012] A1. The middle frame of three adjacent adaptive optics degraded images is recorded as the original reference frame, and the remaining two frames are adjacent frames of this original reference frame;

[0013] A2, a pixel-level explicit optical flow alignment module is a multi-scale feature pyramid structure, which performs optical flow estimation on adjacent frames of the original reference frame described in step A1 to obtain adjacent frames after optical flow estimation alignment;

[0014] A3. The spatiotemporal attention mechanism module includes a temporal attention mechanism submodule and a spatial attention mechanism submodule. The temporal attention mechanism submodule dynamically assigns feature weights to adjacent frames after optical flow estimation based on their similarity with the original reference frame. It then performs feature fusion on the adjacent frames after optical flow estimation and the original reference frame based on the weights.

[0015] The fused features are input into the spatial attention module for spatial affine transformation;

[0016] A4, the temporal sharpness prior module extracts local clear pixels based on the adjacent frames aligned with the original reference frame after optical flow estimation;

[0017] A5. The outputs of the temporal sharpness prior module and the spatiotemporal attention mechanism module are fed into the image reconstruction module to recover a clear latent image. The loss function of the image reconstruction module is calculated based on the current recovered clear latent image and the corresponding clear reference image. The module also uses a hard case mining strategy to focus on some feature areas that are difficult to recover. The formula for the loss function of each image restoration unit is:

[0018]

[0019] Among them, M i represents the mask of hard case mining, λ represents the weight, I i,r Represents the currently restored image, I i,c represents a clear reference image; ⊙ represents pixel-level multiplication, || ||1 represents the L1 norm;

[0020] S3. Send the three frames of restored clear latent images to the second-stage image restoration unit to obtain the final restored clear adaptive optical image.

[0021] The training process includes multiple iterations. In the back-propagation phase of each iteration, the parameters of a single image restoration unit are updated through back-propagation. At the beginning of the next iteration, the updated parameters of the same set of image restoration units are applied to the four image restoration units of the cascade network in the next iteration.

[0022] Beneficial effects of the present invention: Compared to existing adaptive optical image reconstruction technology, the present invention uses pixel-level optical flow estimation based on deep learning to align adjacent frames and reference frames, thereby reducing reconstructed image artifacts, improving image resolution, and enriching fine structures. The present invention uses a two-stage cascade structure to stably restore reconstructed images with higher quantitative evaluation: the two-stage cascade structure adopted by the present invention uses the same set of training parameters to simultaneously restore the images of the first and second stages, thereby accelerating model convergence and reducing trainable parameters as much as possible. When the input image contrast is low, the reconstructed image of the present invention is not affected by the input image and can output more stable high-quality results. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is an implementation flow chart of an embodiment of the present invention.

[0024] Figure 2 It is a schematic diagram of the overall network structure of an embodiment of the present invention.

[0025] Figure 3 yes Figure 2 Schematic diagram of the structure of the spatiotemporal attention module.

[0026] Figure 4 yes Figure 1Schematic diagram of the cascade structure.

[0027] Figure 5 is an embodiment of the present invention;

[0028] Among them, (a) is the adaptive optical image of 256×256 pixels, and (b) is the restoration result of the present invention.

[0029] Figure 6 This is the comparison result of the reconstruction using the original frame with the lowest contrast among the 100 frames of adaptive optics images as input;

[0030] Among them, (a) is the original adaptive optical image of 1408×1408 pixels, (b)-(d) are the corresponding results of EDDNN, CSSTN and the reconstruction of the present invention, respectively, (e) is the standard reference image reconstructed by the speckle method, and (f) is the power spectral density curve corresponding to (a)-(e). DETAILED DESCRIPTION

[0031] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0032] Figure 1 This is a flow chart of a cascaded adaptive optical image blind restoration method described in the present invention, which mainly includes model training, convergence verification and testing processes using real adaptive optical images. Figure 2 The cascade network of the present invention includes four image restoration units (TSAN, Temporal and Spatial Attention Network), which are divided into two stages, of which the first stage includes three image restoration units and the second stage includes one image restoration unit; each image restoration unit has the same structure, including: a pixel-level explicit optical flow alignment module, a spatiotemporal attention mechanism module (TSA, Temporal and Spatial Attention), a temporal sharpness prior module (TSP, Temporal Sharpness Prior), and an image reconstruction module; this embodiment uses optical flow alignment, spatiotemporal attention mechanism, and temporal sharpness prior to extract image features, which are finally fed into the image reconstruction network of the codec structure. The specific process of the present invention is as follows:

[0033] Step (1): Collect multiple frames of adaptive optics degraded images and corresponding clear reference images for model training, verification and testing and perform preprocessing;

[0034] In the embodiment, the training set is a self-collected real solar adaptive optics image dataset, which contains 6700 short-exposure images, of which 5300 are used for training, 500 are used for verification, and 1000 are used for testing. The parameters of the training set are shown in the parameters of the first and second datasets in Table 1.

[0035] Table 1 Parameters of the five data sets used in the present invention for network training, verification and testing

[0036]

[0037] Step (2): Crop all images in the training set to a size of 256 × 256 pixels;

[0038] Step (3): feeding the cropped adaptive optics degraded image and the corresponding clear reference image into a pixel-level explicit optical flow alignment model;

[0039] The pixel-level explicit optical flow alignment model first constructs a multi-scale feature pyramid, uses the features of the l+1 layer to perform a ×2 upsampling operation, and then adds the second image feature Warp to the first image and get the warping result of the lth layer The process can be described as:

[0040]

[0041] where up2(w l+1 )(x) is the estimated result of optical flow after upsampling, w l+1 is the optical flow estimation result of the previous layer. Then calculate the cost volume cv of the lth layer l (x1, x2):

[0042]

[0043] Where x represents the pixel index, which can be x1 or x2, representing the pixel index of the original image feature and the pixel index of the warping result, respectively. T represents the transpose operation, and N is a column vector. After the calculation is completed, the above steps can be repeated for each layer of the feature pyramid to complete the final optical flow estimation from the adjacent degraded frame to the reference degraded frame. i-1 , I i and I i+1 are three frames of continuous degraded adaptive images, and the estimated optical flows are u i+1→i ,u i-1→i , get I after optical flow estimation alignment i Two adjacent frames I i+1 (x+u i+1→i ), I i-1 (x+u i-1→i ).

[0044] Step (4): Send the aligned image to the spatiotemporal attention mechanism module to obtain the features after feature weight redistribution;

[0045] like Figure 3 As shown in Figure 1, the spatiotemporal attention module proposed in this invention is composed of a temporal attention mechanism and a spatial attention mechanism. The temporal attention module designs a method to dynamically assign feature weights to three adjacent frames based on the similarity of adjacent frames. First, the similarity between each adjacent frame and the reference frame is calculated:

[0046]

[0047] where θ(·) and φ(·) represent the embedding space calculated from the input adjacent frames and the reference frame, respectively. Represents the matrix dot product, and Sigmoid adjusts each pixel value of the similarity to the interval [0,1].

[0048] Then, using the calculated h i+1 , h i-1 To adjust the weights of different adjacent frames:

[0049]

[0050]

[0051] where ⊙ is the pixel-wise multiplication.

[0052] Finally, fused convolution is used to fuse the features of the three frames:

[0053]

[0054] The fused features are input into the spatial attention module for spatial affine transformation. The structure of the spatial attention module is as follows: Figure 3 As shown on the right and in Table 2.

[0055] Table 2 Structure of spatial attention mechanism

[0056]

[0057] Step (5): extracting local sharp pixels from the aligned adaptive optics image using the temporal sharpness prior;

[0058] The temporal sharpness prior in step (5) is calculated by the following formula:

[0059]

[0060] Among them I i+j (y+ui+j→i ) and I i (y) represents the adjacent frame output from the previous stage and the original reference frame respectively; w(x) represents the image block centered at the position of pixel x, and the image block size is usually set to 20 or other arbitrary sizes; j represents the adjacent frame, and its value range is {-1,1}.

[0061] Table 3 Schematic diagram of the structure of the feature reconstruction network

[0062]

[0063] Step (6): The output of the temporal sharpness prior and the spatiotemporal attention mechanism module is fed into the feature reconstruction network. The results of the feature reconstruction network are shown in Table 3.

[0064] At the same time, a hard case mining strategy is used to focus on some feature areas that are difficult to recover; the hard case mining strategy is calculated by the following loss function formula:

[0065]

[0066] That is, before the image restoration unit TSAN restores a clear latent image, it calculates the mask M of the hard case mining in the forward propagation stage. i The weight is determined by the parameter λ which can be 0.2 or any other value. i The calculation method is determined by the following formula:

[0067]

[0068] Among them, M i，h It represents the mask that arranges the loss function values ​​of each pixel in image i in descending order and includes the first a% area, M i，r represents a random mask containing b% pixels used to enhance the robustness of hard case mining. The values ​​of a and b can be 50, 10, or any other value between 0 and 100 respectively.

[0069] Step (7): The recovered latent image is fed into a two-stage cascade architecture to recover a clearer result that is close to the clear reference image;

[0070] The schematic diagram of the two-stage cascade structure is as follows Figure 4 As shown in the figure, in the forward propagation phase of each iterative process, a clear latent image is reconstructed for every three consecutive frames: the first phase takes five consecutive adaptive image frames as input and reconstructs three clear latent images. The second phase takes the three consecutive adaptive image frames reconstructed in the first phase as input and reconstructs one clear latent image. The loss function for reconstructing a clear latent image can be expressed as:

[0071]

[0072] That is, the loss functions of the three image restoration units TSAN in the first stage and the loss function of the one image restoration unit TSAN in the second stage are added together.

[0073] During the backpropagation phase of each iteration, a single TSAN unit completes parameter updates through backpropagation. At the start of the next iteration, the same set of updated TSAN unit parameters are reused four times, thus constructing the forward propagation cascade network structure for the next iteration.

[0074] Step (8): Feed the validation set into the model parameters and judge convergence; if not, repeat steps (2)-(8) until the model converges, that is, stop training when the number of training epochs reaches 300;

[0075] Step (9): Save the model parameters, stop training, and use the model parameters to restore the image of the test set to obtain the final blind restored adaptive optical image.

[0076] The converged model was tested on five real adaptive optics image datasets. PSNR (in dB) was used as the evaluation metric. Comparison methods included EDDNN and CSSTN. Table 1 lists the specific parameters of the five test sets. Table 4 shows the average PSNR values ​​of the restoration results of the method of the present invention and the above-mentioned methods on the five real adaptive optics image datasets.

[0077] Table 4 Average PSNR of the reconstruction results of the method of the present invention (CTSAN) and the above methods on five adaptive optics image test sets

[0078]

[0079] It can be seen from Table 4 that the cascaded adaptive optical image blind restoration method proposed in the present invention has achieved good quantitative evaluation results. Figure 5 An example of the present invention is shown, namely the recovery results of the present invention on the first test set. Figure 6 The results and power spectral density curves of the present invention and other comparison methods EDDNN and CSSTN on the lowest and most contrasting source adaptive optics images are compared. Figure 6 Where Normalized Frequency represents normalized frequency, PowerSpectrum represents power spectrum, and arcsec represents arc seconds.

[0080] In specific implementation, the above process can be automatically run based on the Pytorch framework and Ubuntu system, and the system device of the running method should also be within the scope of protection of the present invention.

[0081] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A cascaded adaptive optical image blind restoration method, characterized in that: The cascade network based on [1] includes four image restoration units, which are divided into two stages. The first stage includes three image restoration units, and the second stage includes one image restoration unit. Each image restoration unit has the same structure, including: a pixel-level explicit optical flow alignment module, a spatiotemporal attention mechanism module, a temporal sharpness prior module, and an image reconstruction module. The implementation process of the method is: S1. Collect multiple frames of adaptive optics degraded images and corresponding clear reference images, and divide the dataset consisting of the multiple frames of adaptive optics degraded images and the corresponding clear reference images into a training set, a validation set, and a test set; the multiple frames of adaptive optics degraded images are adjacent frames collected in time sequence; the corresponding clear reference images are images reconstructed using an image restoration method using any number of degraded frames as input; S2. During the training, validation, and testing processes, the data in the training set, validation set, or test set each uses five consecutive frames of adaptive optical degraded images as input to the cascade network. Three adjacent frames of these five consecutive adaptive optical degraded images are fed into the three image restoration units of the first stage to recover one clear latent image. These five consecutive frames of adaptive optical degraded images include three groups of three adjacent frames of adaptive optical degraded images, thereby recovering three clear latent images. The processing process of each image restoration unit is as follows: A1. The middle frame of three adjacent adaptive optics degraded images is recorded as the original reference frame, and the remaining two frames are adjacent frames of this original reference frame; A2, a pixel-level explicit optical flow alignment module is a multi-scale feature pyramid structure, which performs optical flow estimation on adjacent frames of the original reference frame described in step A1 to obtain adjacent frames after optical flow estimation alignment; A3. The spatiotemporal attention mechanism module includes a temporal attention mechanism submodule and a spatial attention mechanism submodule. The temporal attention mechanism submodule dynamically assigns feature weights to adjacent frames after optical flow estimation based on their similarity with the original reference frame. It then performs feature fusion on the adjacent frames after optical flow estimation and the original reference frame based on the weights. The fused features are input into the spatial attention module for spatial affine transformation; A4, the temporal sharpness prior module extracts local clear pixels based on the adjacent frames aligned with the original reference frame after optical flow estimation; A5. The outputs of the temporal sharpness prior module and the spatiotemporal attention mechanism module are fed into the image reconstruction module to recover a clear latent image. The loss function of the image reconstruction module is calculated based on the current recovered clear latent image and the corresponding clear reference image. The module also uses a hard case mining strategy to focus on some feature areas that are difficult to recover. The formula for the loss function of each image restoration unit is: Among them, M i represents the mask of hard case mining, represents the weight, I i,r Represents the currently restored image, I i,c Represents a clear reference image; represents pixel-level multiplication, and ||||1 represents the L1 norm; S3. Send the three frames of restored clear latent images to the second-stage image restoration unit to obtain the final restored clear adaptive optical image.

2. The cascaded adaptive optical image blind restoration method according to claim 1, characterized in that: The training process includes multiple iterations. In the back-propagation phase of each iteration, the single image restoration unit completes the parameter update through back-propagation. At the beginning of the next iteration process, the updated parameters of the same set of image restoration units are applied to the four image restoration units of the cascade network in the next iteration.

Images