Large-field-of-view astronomical image restoration and super-resolution reconstruction method based on deep learning

Abstract

The present invention belongs to the field of image processing, and specifically is a method for large-field astronomical image restoration and super-resolution reconstruction based on deep learning. The method comprises S1. producing a large-field astronomical image dataset; S2. constructing a Resℜ model by simultaneously constraining the restoration task and the super-resolution task, the model comprising a feature extraction module, a feature correction module, and a progressive generation module; and S3. utilizing the Resℜ model for end-to-end image restoration and super-resolution reconstruction: the feature extraction module extracts features of different resolutions, and then the feature correction module corrects the erroneous portions of the image features. The corrected features are then fused with the corresponding resolution features to generate a restored image and a super-resolution image. This method can simultaneously generate high-quality restored images and super-resolution images, and both PSNR and SSIM are improved.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to image restoration and reconstruction, and specifically provides a large-field-of-view astronomical image restoration and super-resolution reconstruction method based on deep learning. Background Art

[0002] In recent decades, through the intervention and research of a large number of experts at home and abroad, image restoration and super-resolution reconstruction technology has made great progress. In particular, with the rise of deep convolutional neural networks, various frameworks based on convolutional neural networks have been proposed. After searching, it was found that the patent document with patent number: CN116523759A discloses an image super-resolution reconstruction method based on frequency decomposition and restart mechanism. The method includes extracting shallow features of low-resolution images, dynamically decomposing shallow features into different frequency components, updating and redistributing all frequency components using self-attention within the same scale and interaction between different scales to obtain new different frequency components, fusing the new different frequency components, and then cyclically performing decomposition, interactive redistribution and fusion operations. Finally, splicing and image reconstruction are performed to obtain the final image residual, upsampling the low-resolution image to obtain a high-resolution image, and superimposing the final image residual with the high-resolution image to obtain a super-resolution reconstructed image. However, this method has the following problems: (1) It does not fully utilize shallow features of different resolutions, which is not conducive to the recovery of weak sources in the image, and (2) This method is mainly targeted at natural images and is not suitable for large-field astronomical images. Existing astronomical image restoration and super-resolution reconstruction methods generally suffer from low efficiency and poor performance when processing large-field-of-view images. In 2023, Wang et al. proposed a conditional denoising diffusion probability model that leverages visibility data to separate signal from noise while preserving spatial information in degraded images. This model is very effective in removing artifacts and recovering faint sources in single-target images, but it has limitations when working with complex multi-target astronomical images. In 2022, Nammour et al. proposed a sparse deconvolution framework that better preserves galaxy shapes and reduces pixel errors. However, this method mistakenly removes some image information as noise during feature extraction, making it ineffective for recovering faint sources. In 2022, Gao et al. proposed a modeling-based approach that leverages prior knowledge, is applicable to low-sample or unsupervised scenarios, and offers interpretability. However, it also suffers from shortcomings such as degradation model assumptions, reliance on prior knowledge, and limitations on image content and structure. In summary, existing methods generally only process astronomical images of fixed-size, scalable objects, and require a step-by-step restoration and super-resolution reconstruction, resulting in low reconstruction efficiency. Summary of the Invention

[0003] The present invention proposes a method for large-field-of-view astronomical image restoration and super-resolution reconstruction based on deep learning to overcome the problems of low efficiency and poor effect in existing technologies when processing large-field-of-view astronomical images, and generally only targeting fixed-size images, resulting in low reconstruction efficiency.

[0004] The specific technical solutions of the present invention are as follows:

[0005] A method for large-field-of-view astronomical image restoration and super-resolution reconstruction based on deep learning, comprising the following steps:

[0006] S1. Create a dataset:

[0007] 1) Produce a large-field astronomical image dataset for restoration tasks;

[0008] 2) Producing a large-field-of-view astronomical image dataset for super-resolution tasks:

[0009] S2. Construct a Res&RecNet model by constraining both the restoration task and the super-resolution task. The Res&RecNet model includes a feature extraction module, a feature correction module, and a progressive generation module.

[0010] S3. Use the Res&RecNet model for end-to-end image restoration and super-resolution reconstruction: After inputting a degraded image, the feature extraction module first extracts features of different resolutions. For the restoration task, the extracted features are directly input into the feature correction module. For the super-resolution reconstruction task, the features are upsampled and then input into the feature correction module. The feature correction module then corrects the erroneous parts of the image features. After the restoration task and the super-resolution task obtain their respective corrected features, the corrected features are fused with the corresponding resolution features of the progressive generation module to generate the restored image and the super-resolution image.

[0011] Furthermore, the feature extraction module described in step S2 is an encoder network composed of a full convolutional layer. The network has a total of 6 CNNBlocks, each of which contains two layers of CNN and BN. The convolutional layers in each CNNBlock are connected through an Identity Connection structure.

[0012] Furthermore, the feature correction module used in the restoration task and the super-resolution reconstruction task in step S2 is independent, and includes the following three parts by referencing the module in NAFNet:

[0013] (1) Mobile convolution, which is based on point-wise and depth-wise convolution with channel attention;

[0014] (2) A feed-forward network module with two fully connected layers implemented by point-wise convolution;

[0015] (3) Upsampling module, which consists of sub-pixel convolution and Conv2d. This module is unique to the super-resolution reconstruction task and is used to adjust the feature resolution and channels in super-resolution reconstruction.

[0016] Furthermore, the progressive generation module described in step S2 consists of 8 CNNBlocks, each of which contains two layers of CNN and Upsampling. The specific method of regularization in each layer of CNN is: randomly generate convolution weights according to the input and output channels of the current network, and initialize the convolution weights to a mean of 0 and a variance of Gaussian distribution of , where a is the negative slope of the activation function, n represents the number of neurons in the previous layer, and the convolution weights are further regularized.

[0017] Furthermore, in step S2, the restored image and the super-resolution image are constrained simultaneously by MSE loss and SSIM loss. The loss function is shown as follows:

[0018]

[0019]

[0020] n is the number of samples, y i is the real image, y' i To generate an image, μ x and μ y represent the average values ​​of x and y respectively, and Denote the variance of x and y, σ xy represents the covariance of x and y, and C1 and C2 are constants.

[0021] Furthermore, the specific steps of 1) producing a large-field astronomical image dataset for the restoration task in step S1 are as follows: using the Webb Space Telescope to take astronomical images, expanding the number of samples by rotation and random cropping to obtain clear images, and then randomly generating a Gaussian blur kernel equal to the number of clear images. Where I0 represents the peak amplitude, r represents the blur kernel radius, and σ represents the standard deviation. Each generated blur kernel is convolved with a clear image, and Poisson noise and Gaussian white noise are added to simulate the photon noise and current noise added when shooting and transmitting images to obtain a blurred image. These blurred images and their corresponding clear images constitute the sample set for training the restoration model.

[0022] Furthermore, the specific steps of 2) in step S1 for producing a large-field-of-view astronomical image dataset for the super-resolution task are as follows: cropping the image expanded in 1) to a size of 256×256 as a high-resolution image, adding the same blur kernel as that used to produce the restoration task sample to the high-resolution image to obtain a corresponding blurred image, downsampling using bicubic interpolation to obtain a low-resolution image of size 64×64, and finally adding Gaussian noise and Poisson noise to the low-resolution image to produce a low-resolution-super-resolution image pair.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] 1) During training, the Res&RecNet network model constructed by the present invention constrains both the restoration task and the super-resolution reconstruction task, so that the resulting network model Res&RecNet can simultaneously generate high-quality restored images and super-resolution images.

[0025] 2) The network provided by the present invention is constructed using fully convolutional layers and can process images of any size. Progressive generation does not use the LayerNorm operation to regularize the input features, but instead regularizes the convolution weights. During the convolution operation, on the one hand, the convolution process uses neighborhood information to locally adjust the features; on the other hand, the convolution process converts the convolution weight regularization step into a global feature adjustment. The regularized convolution weights implicitly adjust the features at both the local and global levels. Because the network can be trained with small-sized samples and can perform image restoration and super-resolution reconstruction end-to-end, the network efficiency and reconstruction efficiency are high.

[0026] 3) After the feature extraction module extracts features of varying resolutions, the features are passed through a feature correction module before being fused with the features of the progression generation module. This differs from existing methods that directly input shallow features of a fixed resolution into the progression generation module. Therefore, the present invention is applicable not only to natural images but also to large-field-of-view astronomical images.

[0027] 4) Compared with the existing methods, the PSNR peak signal-to-noise ratio and SSIM structural similarity of the image restoration results of the invented method are improved by 4.69 / 7.3% respectively, and the PSNR and SSIM of the super-resolution reconstruction results are improved by 1.975 / 7.7% respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is the network structure diagram of the Res&RecNet model in the method of the present invention;

[0029] Figure 2 Create a process structure diagram for the dataset;

[0030] Figure 3 Comparison chart of PSNR and SSIM indicators of Res&RecNet and existing methods. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand and reproduce the present invention, the technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] Example:

[0033] See also Figure 1 The present invention proposes a method for large-field astronomical image restoration and super-resolution reconstruction based on deep learning, comprising the following steps:

[0034] S1. Create a dataset:

[0035] 1) Producing a large-field astronomical image dataset for restoration tasks:

[0036] The specific process of making a dataset is as follows Figure 2 As shown in the figure, astronomical images were taken using the Webb Space Telescope, and the number of samples was expanded through data augmentation operations such as rotation and random cropping. When creating the dataset for the restoration task, 500,000 64×64 clear images were created using the original images. Then, a Gaussian blur kernel equal to the number of clear images was randomly generated.

[0037] Where I0 represents the peak amplitude, r represents the blur kernel radius, and σ represents the standard deviation.

[0038] Each generated blur kernel is convolved with a clear image, and Poisson noise and Gaussian white noise are added to simulate the photon noise and current noise added when the image is captured and transmitted, resulting in a blurred image. These blurred images and their corresponding clear images constitute the sample set for training the restoration model.

[0039] 2) Produce large-field-of-view astronomical image datasets for super-resolution tasks;

[0040] When preparing the dataset for the super-resolution task, the expanded image is cropped to 256×256 as a high-resolution image. The same blur kernel as that used to make the restoration task sample is then added to the high-resolution image to obtain the corresponding blurred image. Then, bicubic interpolation is used for downsampling to obtain a low-resolution image of 64×64 size. Finally, Gaussian noise and Poisson noise are added to the low-resolution image to obtain 250,000 pairs of low-resolution-super-resolution image pairs.

[0041] S2. Construct a Res&RecNet model by constraining both the restoration task and the super-resolution task. The Res&RecNet model includes a feature extraction module, a feature correction module, and a progressive generation module.

[0042] The restored image and super-resolution image are constrained simultaneously by MSE loss and SSIM loss. The loss function is shown as follows:

[0043]

[0044]

[0045] n is the number of samples, y i is the real image, y' i To generate an image, μ x and μ y represent the average values ​​of x and y respectively, and Denote the variance of x and y, σ xy represents the covariance of x and y, and C1 and C2 are constants.

[0046] The feature extraction module is an encoder network composed of fully convolutional layers. The network has a total of 6 CNNBlocks. Each CNNBlock contains two layers of CNN and BN. The convolutional layers in each CNNBlock are connected through the Identity Connection structure.

[0047] The feature correction module used in the restoration task and the super-resolution reconstruction task is independent and includes the following three parts by referencing the module in NAFNet:

[0048] (1) Mobile convolution, which is based on point-wise and depth-wise convolution with channel attention;

[0049] (2) A feed-forward network module with two fully connected layers implemented by point-wise convolution;

[0050] (3) Upsampling module, which consists of sub-pixel convolution and Conv2d. This module is unique to the super-resolution reconstruction task and is used to adjust the feature resolution and channels in super-resolution reconstruction.

[0051] The progressive generation module consists of 8 CNNBlocks, each of which contains two layers of CNN and Upsampling. During the convolution operation, on the one hand, the convolution process uses neighborhood information to adjust the features locally; on the other hand, the convolution process converts the convolution weight regularization step into a global feature adjustment. Specifically, the convolution weights are randomly generated according to the input and output channels of the current network, and the convolution weights are initialized to a mean of 0 and a variance of Gaussian distribution of , where a is the negative slope of the activation function, n represents the number of neurons in the previous layer, and the convolution weights are further regularized.

[0052] S3. Use the Res&RecNet model for end-to-end image restoration and super-resolution reconstruction: After inputting a degraded image, the feature extraction module first extracts features of different resolutions. For the restoration task, the extracted features are directly input into the feature correction module. For the super-resolution reconstruction task, the features are upsampled and then input into the feature correction module. The feature correction module then corrects the erroneous parts of the image features. After the restoration task and the super-resolution task obtain their respective corrected features, the corrected features are fused with the corresponding resolution features of the progressive generation module to generate a restored image or a super-resolution image.

[0053] The restored images and super-resolution images generated by the network model provided by the present invention are compared with the images generated by some common methods in the prior art by comparing the PSNR peak signal-to-noise ratio and SSIM structural similarity indicators. Figure 3 It can be seen that the Res&RecNet of the present invention has the highest PSNR and SSIM indicators when performing image restoration and super-resolution reconstruction. The PSNR and SSIM of the restoration task are 34.39 / 0.910 respectively, and the PSNR and SSIM of the super-resolution task are 30.04 / 0.832 respectively. Compared with other comparison methods in the figure, the present invention has achieved improvements.

[0054] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for large-field astronomical image restoration and super-resolution reconstruction based on deep learning, characterized in that: The steps include: S1. Create a dataset: 1) Produce a large-field astronomical image dataset for restoration tasks; 2) Producing a large-field-of-view astronomical image dataset for super-resolution tasks: S2. Construct a Res&RecNet model by constraining both the restoration task and the super-resolution task. The Res&RecNet model includes a feature extraction module, a feature correction module, and a progressive generation module. S3. Use the Res&RecNet model for end-to-end image restoration and super-resolution reconstruction: After inputting a degraded image, the feature extraction module first extracts features of different resolutions. For the restoration task, the extracted features are directly input into the feature correction module. For the super-resolution reconstruction task, the features are upsampled and then input into the feature correction module. The feature correction module then corrects the erroneous parts of the image features. After the restoration and super-resolution tasks obtain their respective corrected features, the corrected features are fused with the corresponding resolution features from the progressive generation module to generate the restored image and super-resolution image. The feature extraction module described in step S2 is an encoder network composed of fully convolutional layers. The network has a total of 6 CNNBlocks, each of which contains two layers of CNN and BN. The convolutional layers in each CNNBlock are connected through the IdentityConnection structure. The feature correction module used in the restoration task and the super-resolution reconstruction task in step S2 is independent and includes the following three parts by referencing the module in NAFNet: (1) Mobile convolution, which is based on point-wise and depth-wise convolution with channel attention; (2) A feed-forward network module with two fully connected layers implemented by point-wise convolution; (3) Upsampling module, which consists of sub-pixel convolution and Conv2d. This module is unique to the super-resolution reconstruction task and is used to adjust the feature resolution and channels in super-resolution reconstruction; The progressive generation module described in step S2 consists of 8 CNNBlocks. Each CNNBlock contains two layers of CNN and Upsampling. The specific method of regularization in each layer of CNN is: randomly generate convolution weights according to the input and output channels of the current network, and initialize the convolution weights to a mean of 0 and a variance of Gaussian distribution of , where a is the negative slope of the activation function, n represents the number of neurons in the previous layer, and the convolution weights are further regularized.

2. The method for large-field astronomical image restoration and super-resolution reconstruction based on deep learning according to claim 1, characterized in that: In step S2, the restored image and the super-resolution image are constrained simultaneously by MSE loss and SSIM loss. The loss function is shown as follows: n is the number of samples, y i is the real image, y' i To generate an image, μ x and μ y represent the average values ​​of x and y respectively, and Denote the variance of x and y, σ xy represents the covariance of x and y, and C1 and C2 are constants.

3. The method for large-field astronomical image restoration and super-resolution reconstruction based on deep learning according to claim 2, characterized in that: The specific steps of 1) preparing a large-field astronomical image dataset for the restoration task in step S1 are as follows: using the Webb Space Telescope to capture astronomical images, expanding the number of samples by rotation and random cropping to obtain clear images, and then randomly generating a Gaussian blur kernel equal to the number of clear images. Where I0 represents the peak amplitude, r represents the blur kernel radius, and σ represents the standard deviation. Each generated blur kernel is convolved with a clear image, and Poisson noise and Gaussian white noise are added to simulate the photon noise and current noise added when shooting and transmitting images to obtain a blurred image. These blurred images and their corresponding clear images constitute the sample set for training the restoration model.

4. The method for large-field astronomical image restoration and super-resolution reconstruction based on deep learning according to claim 3, characterized in that: The specific steps of 2) in step S1 for producing a large-field-of-view astronomical image dataset for the super-resolution task are as follows: cropping the image expanded in 1) to a size of 256×256 as a high-resolution image, adding the same blur kernel as that used to produce the restoration task sample to the high-resolution image to obtain a corresponding blurred image, downsampling using bicubic interpolation to obtain a low-resolution image of size 64×64, and finally adding Gaussian noise and Poisson noise to the low-resolution image to produce a low-resolution-super-resolution image pair.

Images