Astronomical image super-resolution reconstruction method and device, equipment and storage medium

Abstract

The invention discloses an astronomical image super-resolution reconstruction method and device, equipment and a storage medium. The method comprises the following steps: constructing a super-resolution reconstruction network; inputting an input image of the original astronomical image into the super-resolution reconstruction network to extract global information and local information, and fusing the global information and the local information to obtain a plurality of enhanced feature maps; performing multi-scale feature extraction and feature fusion on shallow feature maps in the plurality of enhanced feature maps to obtain an optimized feature map; performing convolution and up-sampling on the optimized feature map and the deep feature maps in the plurality of enhanced feature maps to generate a super-resolution astronomical image; calculating a loss value based on the original astronomical image and the super-resolution astronomical image, and updating network parameters until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model; and performing image reconstruction on the shot low-resolution astronomical image through the super-resolution reconstruction model to obtain a super-resolution astronomical image. The astronomical image reconstruction effect is improved.

Description

Technical Field

[0001] The present application relates to the field of image reconstruction technology, and in particular to a method, device, equipment and storage medium for super-resolution reconstruction of astronomical images. Background Art

[0002] Generally speaking, the higher the resolution of an astronomical image, the richer the information that can be extracted. However, the resolution of astronomical images is limited by the technical level of the sensor and the shooting distance. The method of directly improving the image quality by upgrading the hardware has problems such as high cost, high process level and long time period. Image super-resolution reconstruction technology is a typical low-level computer vision task. Its goal is to use the information in the image to improve the resolution of the image. Usually, a single low-resolution image is used to reconstruct a high-resolution image, so that the reconstructed image has richer visual information. Therefore, improving the quality of optical remote sensing images with the help of super-resolution reconstruction algorithms has become a hot topic in the field of remote sensing. It has the advantages of economy, convenience and efficiency.

[0003] In recent years, thanks to the development of computer vision in the field of remote sensing, image super-resolution reconstruction methods based on deep learning have made great progress. However, existing super-resolution reconstruction methods based on deep learning do not fully utilize shallow features of different resolutions, thus ignoring the recovery of weak sources in the image, resulting in poor astronomical image reconstruction results. Summary of the invention

[0004] The present application provides a method, apparatus, device and storage medium for super-resolution reconstruction of astronomical images, which are used to improve the technical problem that existing super-resolution reconstruction methods do not fully utilize shallow features of different resolutions, thereby ignoring the recovery of weak sources in the image, resulting in poor astronomical image reconstruction effect.

[0005] In view of this, the first aspect of the present application provides a method for super-resolution reconstruction of astronomical images, comprising:

[0006] Constructing a super-resolution reconstruction network, wherein the super-resolution reconstruction network includes a plurality of shallow feature perception modules, a local optimization module and a progressive generation module;

[0007] Acquire an input image of an original astronomical image, extract global information and local information of the input image in sequence through a plurality of shallow feature perception modules, and fuse the global information and the local information to obtain a plurality of enhanced feature maps;

[0008] Performing multi-scale feature extraction on the shallow feature maps in the multiple enhanced feature maps through the local optimization module, and performing feature fusion on the extracted multi-scale features to obtain an optimized feature map;

[0009] The optimized feature map and the deep feature map in the multiple enhanced feature maps are convolved and up-sampled by the progressive generation module to generate a super-resolution astronomical image;

[0010] Calculating a loss value based on the original astronomical image and the super-resolution astronomical image, and updating a network parameter of the super-resolution reconstruction network by the loss value until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model;

[0011] A low-resolution astronomical image taken by a mobile device is obtained, and the low-resolution astronomical image is reconstructed using the super-resolution reconstruction model to obtain a super-resolution astronomical image.

[0012] Optionally, the shallow feature perception module includes a global and local feature extraction module and a residual dense connection module, and the global and local feature extraction module includes a first feature extraction branch, a second feature extraction branch and a multi-layer perceptron;

[0013] Each of the shallow feature perception modules extracts global information and local information of the input image and fuses the global information and the local information to obtain an enhanced feature map, including:

[0014] Extracting global information and local information of the input image through the first feature extraction branch, and fusing the global information and the local information to obtain feature fusion information;

[0015] The second feature extraction branch sequentially performs convolution processing and channel attention feature extraction on the input image to obtain a channel attention feature;

[0016] Performing feature extraction on the fused feature fusion information, the channel attention features and the input image through the multi-layer perceptron to obtain a first fused feature map;

[0017] The first fusion feature map is subjected to residual dense connection through the residual dense connection module to enhance feature transfer and obtain an enhanced feature map.

[0018] Optionally, performing multi-scale feature extraction on the shallow feature map in the multiple enhanced feature maps by the local optimization module, and performing feature fusion on the extracted multi-scale features to obtain the optimized feature map, includes:

[0019] Dividing the shallow feature maps in the multiple enhanced feature maps into multiple groups of enhanced feature sub-maps along the channel dimension through the local optimization module, performing convolution processing and feature fusion of different convolution kernel sizes on each group of enhanced feature sub-maps, and obtaining a second fused feature map;

[0020] Extracting attention vectors of feature maps of different scales from the second fused feature map; recalibrating the attention vector to obtain recalibration weights of multi-scale channels; weighting the second fused feature map by the recalibration weights to obtain a refined feature map with rich multi-scale feature information;

[0021] Performing three convolution processes on the second fused feature map in parallel to obtain a first new feature map, a second new feature map and a third new feature map respectively; extracting a spatial attention weight map through the first new feature map and the second new feature map; fusing the third new feature map with the spatial attention weight map to obtain a spatial attention feature map;

[0022] The spatial attention feature map and the refined feature map are element-by-element summed and convolved to obtain a final optimized feature map.

[0023] Optionally, the progressive generation module is composed of a color conversion module and a plurality of pyramid dual attention modules stacked in sequence;

[0024] The step of performing convolution and upsampling processing on the optimized feature map and the deep feature map in the plurality of enhanced feature maps by the progressive generation module to generate a super-resolution astronomical image comprises:

[0025] Converting a deep feature map in the plurality of enhanced feature maps into an RGB image by the color conversion module;

[0026] The deep feature map after color conversion is convolved and up-sampled through the pyramid dual attention module, and then fused with the optimized feature map and convolved and up-sampled to generate a super-resolution astronomical image.

[0027] Optionally, the calculating the loss value based on the original astronomical image and the super-resolution astronomical image includes:

[0028] Calculating a wavelength mean square error through the real wavelength information of the original astronomical image and the wavelength information of the super-resolution astronomical image to obtain a wavelength loss;

[0029] Calculating the mean square error or absolute error of pixels at the same position in the original astronomical image and the super-resolution astronomical image, or calculating the similarity between the original astronomical image and the super-resolution astronomical image in brightness, contrast and structure to obtain an image reconstruction loss;

[0030] The wavelength loss and the image reconstruction loss are weightedly summed to obtain a final loss value.

[0031] Optionally, the method further includes:

[0032] The wavelength information of the super-resolution astronomical image is extracted through a classification network.

[0033] Optionally, the method further includes:

[0034] Uploading the low-resolution astronomical image and the super-resolution astronomical image to a cloud album;

[0035] The super-resolution reconstruction model deployed on the remote server is updated using the astronomical images in the cloud album, and the updated model parameters are sent to the mobile device deployed with the super-resolution reconstruction model for local model update, and the astronomical image super-resolution reconstruction is performed using the updated super-resolution reconstruction model.

[0036] A second aspect of the present application provides an astronomical image super-resolution reconstruction device, comprising:

[0037] A network construction unit, used to construct a super-resolution reconstruction network, wherein the super-resolution reconstruction network includes a plurality of shallow feature perception modules, a local optimization module and a progressive generation module;

[0038] A network training unit is used to obtain an input image of an original astronomical image, extract global information and local information of the input image in sequence through a plurality of shallow feature perception modules, and fuse the global information and the local information to obtain a plurality of enhanced feature maps; perform multi-scale feature extraction on shallow feature maps in the plurality of enhanced feature maps through the local optimization module, and perform feature fusion on the extracted multi-scale features to obtain an optimized feature map; perform convolution and up-sampling processing on the optimized feature map and deep feature maps in the plurality of enhanced feature maps through the progressive generation module to generate a super-resolution astronomical image; calculate a loss value based on the original astronomical image and the super-resolution astronomical image, and update the network parameters of the super-resolution reconstruction network through the loss value until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model;

[0039] The image reconstruction unit is used to obtain a low-resolution astronomical image taken by a mobile device, and reconstruct the low-resolution astronomical image through the super-resolution reconstruction model to obtain a super-resolution astronomical image.

[0040] A third aspect of the present application provides an electronic device, the device comprising a processor and a memory;

[0041] The memory is used to store program code and transmit the program code to the processor;

[0042] The processor is used to execute the astronomical image super-resolution reconstruction method described in any one of the first aspects according to the instructions in the program code.

[0043] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the method for super-resolution reconstruction of astronomical images described in any one of the first aspects is implemented.

[0044] It can be seen from the above technical solutions that this application has the following advantages:

[0045] The astronomical image super-resolution reconstruction method provided by the present application extracts global and local information of the astronomical image through a shallow feature perception module, and optimizes the design according to the imaging characteristics of the astronomical image, and uses a local optimization module to optimize the shallow feature error part containing more high-frequency information, and finally the progressive generation module fuses the optimized features during the generation process to obtain a reconstructed high-resolution image, making full use of the features of astronomical images with different resolutions, effectively restoring the weak sources in the astronomical image, avoiding feature loss and image distortion, so that the super-resolution reconstruction network better adapts to the special needs of astronomical images and improves the reconstruction effect;

[0046] Furthermore, wavelength information constraints are introduced during the training process to preserve the details of astronomical images and further improve the reconstruction effect of astronomical images;

[0047] This application reduces computational complexity and resource usage by optimizing the model structure and parameter configuration, greatly improving the processing speed while maintaining a small number of parameters, and is suitable for fast reasoning on mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram of a flow chart of a method for super-resolution reconstruction of astronomical images provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a super-resolution reconstruction network provided in an embodiment of the present application;

[0051] Figure 3 A structural diagram of a global and local feature extraction module provided in an embodiment of the present application;

[0052] Figure 4 A structural schematic diagram of a residual dense connection module provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of the structure of an attention module provided in an embodiment of the present application;

[0054] Figure 6 A design diagram for realizing super-resolution reconstruction of astronomical images based on an embedded processing solution provided in an embodiment of the present application;

[0055] Figure 7 A design diagram for realizing super-resolution reconstruction of astronomical images based on a remote server online processing solution provided in an embodiment of the present application;

[0056] Figure 8 Another schematic diagram of a flow chart of an astronomical image super-resolution reconstruction method provided in an embodiment of the present application;

[0057] Fig. 9 A schematic diagram of the structure of an astronomical image super-resolution reconstruction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] For easier understanding, please refer to Figure 1 , the embodiment of the present application provides a method for super-resolution reconstruction of astronomical images, comprising:

[0060] Step 110: construct a super-resolution reconstruction network, which includes multiple shallow feature perception modules, local optimization modules and progressive generation modules.

[0061] The super-resolution reconstruction network constructed in the embodiment of the present application includes multiple shallow feature perception modules, local optimization modules and progressive generation modules. Please refer to Figure 2 A super-resolution reconstruction network is provided. Figure 2The super-resolution reconstruction network in is composed of 6 shallow feature perception modules (SPBlocks) stacked in sequence, namely, the first shallow feature perception module (SPBlock1), the second shallow feature perception module (SPBlock2), the third shallow feature perception module (SPBlock3), the fourth shallow feature perception module (SPBlock4), the fifth shallow feature perception module (SPBlock5) and the sixth shallow feature perception module (SPBlock6) are connected in series in sequence, and the input end of the first shallow feature perception module is used to acquire astronomical images. Among them, each shallow feature perception module is composed of a global and local feature extraction module and a residual dense connection module. The global and local feature extraction module is used to capture global and local features, while the residual dense connection module enhances feature transfer through residual dense connections, thereby maximizing the retention of the original features of the image and improving the fidelity of the reconstruction results. The structure of the global and local feature extraction module is shown in the figure. Figure 3 As shown in Figure 1, the global and local feature extraction module includes two feature extraction branches. The first feature extraction branch consists of a split layer, an anchored stripe self-attention mechanism layer (Anchored Stripe Attention), a window self-attention layer (WindowAttention V2) and a concatenate layer (feature concatenation layer). The second feature extraction branch consists of two convolutional layers and a channel attention layer (Channel Attention). The features extracted by the first feature extraction branch, the features extracted by the second feature extraction branch and the original input are concatenated and input into a multi-layer perceptron (MLP) for processing, and then input into a residual dense connection module for feature extraction. The structure of the residual dense connection module (RDB) is shown in Figure 1. Figure 4 As shown in the figure, it consists of three convolutional layers and a fusion layer stacked together. The output ends of the three convolutional layers are respectively connected to the ReLU activation function. The convolutional layers are cross-connected. The fusion layer consists of a concat layer and a 1×1 convolutional layer. The output of the fusion layer is summed with the input of the first convolutional layer as the final output of the residual dense connection module.

[0062] Among them, the local optimization module consists of an attention module (PAD) and a convolutional layer. The structure of PAD is as follows: Figure 5 As shown in FIG. 1 , there are two local optimization modules, namely the first local optimization module and the second local optimization module. The progressive generation module includes a color conversion module (ToRGB module) and multiple pyramid dual attention modules (PGBlock). Multiple pyramid dual attention modules are stacked in sequence. Figure 2The progressive generation module in is composed of the first pyramid dual attention module (PGBlock1, the ToRGB module can be embedded in the input of PGBlock1 as a submodule in PGBlock1), the second pyramid dual attention module (PGBlock2), the third pyramid dual attention module (PGBlock3), the fourth pyramid dual attention module (PGBlock4), the fifth pyramid dual attention module (PGBlock5) and the sixth pyramid dual attention module (PGBlock6) connected in series in sequence. Each pyramid dual attention module contains two layers of CNN and an upsampling layer. Among them, the first shallow feature perception module is connected to the sixth pyramid dual attention module (PGBlock6) through the first local optimization module, the second shallow feature perception module is connected to the fifth pyramid dual attention module (PGBlock5) through the second local optimization module, and the sixth shallow feature perception module is connected to the first pyramid dual attention module (PGBlock5), so as to utilize the shallow features and deep features of the astronomical image extracted by the shallow feature perception module, and directly input the deep features into the progressive generation module, and then use the local optimization module to optimize the erroneous parts of the shallow features containing more high-frequency information, and finally the progressive generation module fuses the deep features with the optimized features during the generation process to obtain the reconstructed high-resolution image.

[0063] Step 120: Obtain an input image of an original astronomical image, extract global information and local information of the input image in sequence through multiple shallow feature perception modules, and fuse the global information and the local information to obtain multiple enhanced feature maps.

[0064] After obtaining the high-resolution original astronomical image, it is subjected to resolution reduction processing (such as using the nearest neighbor method or bilinear interpolation method to process the original astronomical image) to generate a low-resolution astronomical image, and the low-resolution astronomical image is input as the input image into the above-constructed super-resolution reconstruction network, and the global information and local information of the input image are extracted in turn through multiple shallow feature perception modules, and the global information and the local information are fused to obtain multiple enhanced feature maps. Among them, each shallow feature perception module extracts the global information and local information of the input image through the first feature extraction branch, and fuses the global information and the local information to obtain feature fusion information; the input image is sequentially subjected to convolution processing and channel attention feature extraction through the second feature extraction branch to obtain channel attention features; the fused feature fusion information, channel attention features and input image are subjected to feature extraction through a multi-layer perceptron to obtain a first fused feature map; the first fused feature map is subjected to residual dense connection through the residual dense connection module to enhance feature transfer and obtain an enhanced feature map. It should be noted that the enhanced feature maps extracted by the first shallow feature perception module and the second shallow feature perception module are shallow feature maps; the enhanced feature maps extracted by the third shallow feature perception module, the fourth shallow feature perception module and the fifth shallow feature perception module are middle-level feature maps; and the enhanced feature map extracted by the sixth shallow feature perception module is a deep feature map.

[0065] In the embodiment of the present application, a low-resolution astronomical image of 64×64×3 is used as an example to illustrate the feature extraction process of the astronomical image. The low-resolution astronomical image is input into the super-resolution reconstruction network constructed above. The global and local feature extraction module in the first shallow feature perception module (SPBlock1) extracts features of the input image (i.e., the low-resolution astronomical image) in parallel through two feature extraction branches. The split layer in the first feature extraction branch divides the input image along the channel to obtain two groups of feature maps. Feature extraction is performed on one group of feature maps through the Anchored Stripe Attention layer, and feature extraction is performed on the other group of feature maps through the Window Attention V2 layer to obtain global information and local information. The extracted global information and local information are fused through the Concatenate layer, and the number of output channels is expanded to 32, so the feature dimension becomes 64×64×32. The feature extraction process of the Anchored Stripe Attention layer and the Window AttentionV2 layer belongs to the prior art, and its specific feature extraction process will not be repeated here.

[0066] The second feature extraction branch changes the dimension of the input image to 64×64×32 through two 3×3 convolutional layers, and then enhances important features through the Channel Attention layer, while the channel dimension remains unchanged.

[0067] Finally, the features extracted by the two feature extraction branches and the input image are fused by short-circuiting and input into the multi-layer perceptron for processing to obtain the first fused feature map with a dimension of 64×64×32.

[0068] The residual dense connection module receives the first fused feature map from the first global and local feature extraction module, and uses the convolution operation of the dense connection structure to accumulate the multi-layer features to obtain the final enhanced feature map. The output channel is still 32 and the output dimension is 64×64×32.

[0069] The feature extraction process from SPBlock2 to SPBlock6 is similar to that of SPBlock1, except that the input images are different (the input image of SPBlock1 is a low-resolution astronomical image, and the input image of SPBlock2 to SPBlock6 is the output of the previous shallow feature perception module) and the feature dimension changes are slightly different. When the input image passes through SPBlock2 to SPBlock5, the size is halved each time, and the channels are doubled. For example: the feature map dimension of SPBlock2 output becomes 32×32×64; the feature map dimension of SPBlock3 output becomes 16×16×128; the feature map dimension of SPBlock4 output becomes 8×8×256; the feature map dimension of SPBlock5 output becomes 4×4×512; when passing through SPBlock6, the output feature map size is halved, and the channels remain unchanged, that is, 2×2×512.

[0070] Step 130: extract multi-scale features from shallow feature maps in the multiple enhanced feature maps through a local optimization module, and perform feature fusion on the extracted multi-scale features to obtain an optimized feature map.

[0071] After receiving the shallow enhanced feature map (i.e., shallow feature map) output by the shallow feature perception module, the local optimization module divides the shallow enhanced feature map into S groups of enhanced feature sub-maps from the channel dimension, represented as [X0, X1, X2, …, X s-1 ], and for each set of enhanced feature subgraphs X i Convolution processing with different convolution kernel sizes is performed on (i=0,1,2,...,s-1) to extract multi-scale context information and obtain multiple groups of convolution feature maps. The amount of calculation can be reduced by grouping convolution; element-wise summation is performed on each group of convolution feature maps to obtain the second fused feature map (F).

[0072] After obtaining the second fused feature map, two parts of feature extraction are performed on it in parallel, one of which is as follows: extracting the attention vectors of feature maps of different scales from the second fused feature map; recalibrating the attention vectors to obtain the recalibrated weights of the multi-scale channels; weighting the second fused feature map by the recalibrated weights to obtain a refined feature map with rich multi-scale feature information. Specifically, the attention of feature maps of different scales is extracted from the second fused feature map by the SEWeight module (SE Weight module) to obtain the attention vector of the channel dimension; the attention vector of the channel dimension is recalibrated by the Softmax layer to obtain the recalibrated weights of the multi-scale channels; an element-wise product operation is performed on the recalibrated weights and the corresponding second fused feature map to obtain a refined feature map (Y) with rich multi-scale feature information; it should be noted that the SEWeight module is an existing network structure, and its specific structure will not be described here.

[0073] The feature extraction process of the other part is: extract the spatial attention map from the second fused feature map through convolution layers and matrix multiplication, that is, perform three convolutions on the second fused feature map in parallel to obtain the first new feature map, the second new feature map and the third new feature map respectively; extract the spatial attention weight map through the first new feature map and the second new feature map; fuse the third new feature map with the spatial attention weight map to obtain the spatial attention feature map. Specifically, the second fused feature map is convolved three times in parallel to obtain the first new feature map B, the second new feature map D and the third new feature map E respectively; the first new feature map B is reshaped to realize the vectorization operation of the first new feature map B, and the vector features obtained after the vectorization of the first new feature map B are transposed and matrix multiplication is performed with the second new feature map D to obtain a pixel similarity matrix; then the pixel similarity matrix is ​​processed through the softmax layer to obtain a spatial attention weight map; the third new feature map E is vectorized and then matrix multiplication and vectorization are performed with the spatial attention weight map, and finally element-by-element sum operation is performed with the second fused feature map F to obtain the final spatial attention feature map (L).

[0074] The local optimization module sums the spatial attention feature map and the refined feature map element by element to obtain the final optimized feature map.

[0075] It should be noted that the processing processes of the first local optimization module and the second local optimization module are similar, except that the input images are different. Therefore, the specific processing processes of the two local optimization modules will not be described separately in the embodiment of the present application.

[0076] Step 140: Perform convolution and upsampling processing on the optimized feature map and the deep feature maps in the plurality of enhanced feature maps through a progressive generation module to generate a super-resolution astronomical image.

[0077] The progressive generation module first converts the deep enhanced feature map output by the sixth shallow feature perception module into an RGB image through the ToRGB module, and then performs convolution and Upsampling operations on it through PGBlock1 to improve the resolution, and then fuses it with the next layer and inputs it into the next PGBlock (PGBlock2) for convolution and Upsampling operations. After receiving the output feature map of PGBlock4 and the optimized feature map output by the second local optimization module, PGBlock5 fuses the two received feature maps and then performs convolution and upsampling operations; after receiving the output feature map of PGBlock5 and the optimized feature map output by the first local optimization module, PGBlock6 fuses the two received feature maps and then performs convolution and upsampling operations to finally generate a super-resolution astronomical image.

[0078] The super-resolution reconstruction network in this application effectively reduces the number of model parameters and calculations by grouping and layering convolutions, thereby improving computational efficiency; residual connections and dense connections are used to enhance feature transfer and gradient flow. This design not only improves the training efficiency of the network, but also reduces the difficulty of training deep networks; by introducing multi-scale feature extraction and attention mechanisms, the network can more effectively utilize limited parameter resources and capture more useful information, thereby improving overall performance.

[0079] Step 150: Calculate a loss value based on the original astronomical image and the super-resolution astronomical image, and update the network parameters of the super-resolution reconstruction network according to the loss value until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model.

[0080] The wavelength mean square error is calculated through the wavelength information of the original astronomical image and the wavelength information of the super-resolution astronomical image to obtain the wavelength loss; the pixel mean square error or absolute error at the same position in the original astronomical image and the super-resolution astronomical image is calculated, or the similarity between the original astronomical image and the super-resolution astronomical image in brightness, contrast and structure is calculated to obtain the image reconstruction loss; the wavelength loss and image reconstruction loss are weightedly summed to obtain the final loss value.

[0081] When the original astronomical image is collected, the real wavelength information of the original astronomical image is obtained and added to the label. The wavelength information of the generated super-resolution astronomical image can be extracted through the classification network, wherein when training the classification network, the astronomical image is used as the input image, and the real wavelength information of the astronomical image is used as the training target to train the classification network, and the wavelength information of the generated super-resolution astronomical image is extracted through the trained classification network; then the wavelength loss is calculated by comparing the wavelength information predicted by the classification network with the real wavelength information. Specifically, the wavelength loss can be obtained by calculating the mean square error of the wavelength information of the original astronomical image and the wavelength information of the super-resolution astronomical image, that is:

[0082]

[0083] Where n is the number of samples of low-resolution astronomical images, y i is the real wavelength information corresponding to the i-th low-resolution astronomical image (i.e., the wavelength information of the original astronomical image i), is the wavelength information of the i-th super-resolution astronomical image generated.

[0084] It is also possible to calculate the mean square error or absolute error of pixels at the same position in the original astronomical image and the super-resolution astronomical image, or calculate the similarity in brightness, contrast and structure between the original astronomical image and the super-resolution astronomical image to obtain the image reconstruction loss; perform weighted summation of the wavelength loss and the image reconstruction loss to obtain the final loss value; update the network parameters of the super-resolution reconstruction network (the network parameters include the weights and biases between each network layer) by the calculated final loss value until the super-resolution reconstruction network converges (such as reaching the maximum number of training iterations, the training error converges to a certain value, or the training error is lower than a preset error threshold), and obtain a trained super-resolution reconstruction model. This application combines the wavelength loss with the astronomical image reconstruction loss to update the parameters of the super-resolution reconstruction network, so that the reconstruction network can learn and retain the wavelength characteristics in the astronomical image, which helps to improve the reconstruction accuracy and effect.

[0085] Step 160: Obtain a low-resolution astronomical image taken by a mobile device, reconstruct the low-resolution astronomical image using a super-resolution reconstruction model, and obtain a super-resolution astronomical image.

[0086] In one embodiment, the trained super-resolution reconstruction model can be converted into a format suitable for mobile devices, such as TensorFlow Lite, ONNX or Core ML; then, the converted super-resolution reconstruction model is deployed on the mobile device, and its computing resources (such as CPU, GPU or NPU) are used for local real-time reasoning to achieve super-resolution astronomical image reconstruction. This method has the advantages of low latency, high privacy and offline processing, and is very suitable for application scenarios that require instant feedback.

[0087] In another embodiment, the trained super-resolution reconstruction model may be deployed to a remote server to enable rapid super-resolution astronomical reconstruction using the server's high-performance computing resources.

[0088] This super-resolution reconstruction model can be applied to mobile devices through embedded processing or online processing on a remote server, enabling fast and efficient post-processing of captured astronomical images.

[0089] Please refer to Figure 6 In the embedded processing solution, the mobile device collects astronomical images through the deployed camera module. After capturing the low-resolution astronomical image, the low-resolution astronomical image is reconstructed through the super-resolution reconstruction model deployed on the mobile device to obtain the super-resolution astronomical image.

[0090] Please refer to Figure 7 In the remote server online processing solution, the mobile device can upload the captured low-resolution astronomical images to the remote server. The remote server uses high-performance computing resources and deployed super-resolution reconstruction models to quickly reconstruct super-resolution astronomical images, and then transmits the processed high-resolution astronomical images back to the mobile device for users to view and analyze. This method can make full use of the powerful computing power of the server and is suitable for scenarios that require high-precision and large-scale data processing. The server can centrally manage and update the model to ensure that the latest version is used.

[0091] In another embodiment, the collected low-resolution astronomical images and the generated super-resolution astronomical images can be uploaded to a cloud album; then, the super-resolution reconstruction model deployed on the remote server is updated using the astronomical images in the cloud album, and the updated model parameters are sent to a mobile device deployed with the super-resolution reconstruction model for local model update, and the astronomical image super-resolution reconstruction is performed using the updated super-resolution reconstruction model.

[0092] Please refer to Figure 8After capturing a low-resolution astronomical image, the mobile device can ask the user whether to save it to the cloud album; if the user chooses to save it to the cloud album, the collected low-resolution astronomical image will be uploaded to the cloud album; if the user chooses not to save it to the cloud album, it will be saved to the mobile device album, and then the super-resolution reconstruction model deployed on the mobile device can be selected for super-resolution astronomical image reconstruction, or the captured low-resolution astronomical image can be uploaded to the remote server for super-resolution astronomical image reconstruction to obtain super-resolution astronomical images; the generated super-resolution astronomical images can be further backed up to the cloud album. The super-resolution reconstruction model deployed on the remote server can also be updated and trained through the astronomical images in the cloud album, and the updated network parameters can be sent to the mobile device to update the super-resolution reconstruction model in the mobile device, thereby improving the accuracy of astronomical image reconstruction.

[0093] This application extracts global and local information of astronomical images through a shallow feature perception module, and optimizes the design according to the imaging characteristics of astronomical images. It uses a local optimization module to optimize the erroneous parts of shallow features containing more high-frequency information. Finally, the progressive generation module fuses the optimized features during the generation process to obtain a reconstructed high-resolution image. It makes full use of astronomical image features of different resolutions, effectively restores weak sources in astronomical images, avoids feature loss and image distortion, and enables the super-resolution reconstruction network to better adapt to the special needs of astronomical images and improve the reconstruction effect. It further introduces wavelength information constraints during the training process to retain astronomical image details and further improve the reconstruction effect. This application optimizes the model structure and parameter configuration, reduces computational complexity and resource usage, and greatly improves the processing speed while maintaining a small number of parameters. It is suitable for fast reasoning on mobile devices. This application supports two methods: embedded processing and remote server online processing. Users can choose local fast processing or cloud high-precision processing according to their needs.

[0094] Please refer to Fig. 9 The present application also provides an astronomical image super-resolution reconstruction device, including:

[0095] A network construction unit 310 is used to construct a super-resolution reconstruction network, wherein the super-resolution reconstruction network includes a plurality of shallow feature perception modules, a local optimization module and a progressive generation module;

[0096] The network training unit 320 is used to obtain an input image of an original astronomical image, extract global information and local information of the input image in sequence through multiple shallow feature perception modules, and fuse the global information and the local information to obtain multiple enhanced feature maps; extract multi-scale features from shallow feature maps in the multiple enhanced feature maps through a local optimization module, and fuse the extracted multi-scale features to obtain an optimized feature map; perform convolution and up-sampling processing on the optimized feature map and deep feature maps in the multiple enhanced feature maps through a progressive generation module to generate a super-resolution astronomical image; calculate a loss value based on the original astronomical image and the super-resolution astronomical image, and update the network parameters of the super-resolution reconstruction network through the loss value until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model;

[0097] The image reconstruction unit 330 is used to obtain a low-resolution astronomical image taken by a mobile device, and reconstruct the low-resolution astronomical image using a super-resolution reconstruction model to obtain a super-resolution astronomical image.

[0098] The embodiment of the present application also provides an electronic device, the device comprising a processor and a memory;

[0099] The memory is used to store the program code and transmit the program code to the processor;

[0100] The processor is used to execute the astronomical image super-resolution reconstruction method in the aforementioned method embodiment according to the instructions in the program code.

[0101] An embodiment of the present application also provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and when the program code is executed by a processor, the astronomical image super-resolution reconstruction method in the aforementioned method embodiment is implemented.

[0102] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0103] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0104] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0107] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0109] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A super-resolution reconstruction method for astronomical images, characterized in that: include: Constructing a super-resolution reconstruction network, wherein the super-resolution reconstruction network includes a plurality of shallow feature perception modules, a local optimization module and a progressive generation module; Acquire an input image of an original astronomical image, extract global information and local information of the input image in sequence through a plurality of shallow feature perception modules, and fuse the global information and the local information to obtain a plurality of enhanced feature maps; Performing multi-scale feature extraction on the shallow feature maps in the multiple enhanced feature maps through the local optimization module, and performing feature fusion on the extracted multi-scale features to obtain an optimized feature map; The optimized feature map and the deep feature map in the multiple enhanced feature maps are convolved and up-sampled by the progressive generation module to generate a super-resolution astronomical image; Calculating a loss value based on the original astronomical image and the super-resolution astronomical image, and updating a network parameter of the super-resolution reconstruction network by the loss value until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model; A low-resolution astronomical image taken by a mobile device is obtained, and the low-resolution astronomical image is reconstructed using the super-resolution reconstruction model to obtain a super-resolution astronomical image.

2. The astronomical image super-resolution reconstruction method according to claim 1, characterized in that: The shallow feature perception module includes a global and local feature extraction module and a residual dense connection module, and the global and local feature extraction module includes a first feature extraction branch, a second feature extraction branch and a multi-layer perceptron; Each of the shallow feature perception modules extracts global information and local information of the input image and fuses the global information and the local information to obtain an enhanced feature map, including: Extracting global information and local information of the input image through the first feature extraction branch, and fusing the global information and the local information to obtain feature fusion information; The second feature extraction branch sequentially performs convolution processing and channel attention feature extraction on the input image to obtain a channel attention feature; Performing feature extraction on the fused feature fusion information, the channel attention features and the input image through the multi-layer perceptron to obtain a first fused feature map; The first fusion feature map is subjected to residual dense connection through the residual dense connection module to enhance feature transfer and obtain an enhanced feature map.

3. The astronomical image super-resolution reconstruction method according to claim 1, characterized in that: The extracting multi-scale features of the shallow feature maps in the multiple enhanced feature maps by the local optimization module and fusing the extracted multi-scale features to obtain the optimized feature map comprises: Dividing the shallow feature maps in the multiple enhanced feature maps into multiple groups of enhanced feature sub-maps along the channel dimension through the local optimization module, performing convolution processing and feature fusion of different convolution kernel sizes on each group of enhanced feature sub-maps, and obtaining a second fused feature map; Extracting attention vectors of feature maps of different scales from the second fused feature map; recalibrating the attention vector to obtain recalibration weights of multi-scale channels; weighting the second fused feature map by the recalibration weights to obtain a refined feature map with rich multi-scale feature information; Performing three convolution processes on the second fused feature map in parallel to obtain a first new feature map, a second new feature map and a third new feature map respectively; extracting a spatial attention weight map through the first new feature map and the second new feature map; fusing the third new feature map with the spatial attention weight map to obtain a spatial attention feature map; The spatial attention feature map and the refined feature map are element-by-element summed and convolved to obtain a final optimized feature map.

4. The astronomical image super-resolution reconstruction method according to claim 1, characterized in that: The progressive generation module is composed of a color conversion module and a plurality of pyramid dual attention modules stacked in sequence; The step of performing convolution and upsampling processing on the optimized feature map and the deep feature map in the plurality of enhanced feature maps by the progressive generation module to generate a super-resolution astronomical image comprises: Converting a deep feature map in the plurality of enhanced feature maps into an RGB image by the color conversion module; The deep feature map after color conversion is convolved and up-sampled through the pyramid dual attention module, and then fused with the optimized feature map and convolved and up-sampled to generate a super-resolution astronomical image.

5. The astronomical image super-resolution reconstruction method according to claim 1, characterized in that: The calculating the loss value based on the original astronomical image and the super-resolution astronomical image comprises: Calculating a wavelength mean square error through the real wavelength information of the original astronomical image and the wavelength information of the super-resolution astronomical image to obtain a wavelength loss; Calculating the mean square error or absolute error of pixels at the same position in the original astronomical image and the super-resolution astronomical image, or calculating the similarity between the original astronomical image and the super-resolution astronomical image in brightness, contrast and structure to obtain an image reconstruction loss; The wavelength loss and the image reconstruction loss are weightedly summed to obtain a final loss value.

6. The astronomical image super-resolution reconstruction method according to claim 5, characterized in that: The method further comprises: The wavelength information of the super-resolution astronomical image is extracted through a classification network.

7. The astronomical image super-resolution reconstruction method according to claim 1, characterized in that: The method further comprises: Uploading the low-resolution astronomical image and the super-resolution astronomical image to a cloud album; The super-resolution reconstruction model deployed on the remote server is updated using the astronomical images in the cloud album, and the updated model parameters are sent to the mobile device deployed with the super-resolution reconstruction model for local model update, and the astronomical image super-resolution reconstruction is performed using the updated super-resolution reconstruction model.

8. An astronomical image super-resolution reconstruction device, characterized in that: include: A network construction unit, used to construct a super-resolution reconstruction network, wherein the super-resolution reconstruction network includes a plurality of shallow feature perception modules, a local optimization module and a progressive generation module; A network training unit is used to obtain an input image of an original astronomical image, extract global information and local information of the input image in sequence through a plurality of shallow feature perception modules, and fuse the global information and the local information to obtain a plurality of enhanced feature maps; perform multi-scale feature extraction on shallow feature maps in the plurality of enhanced feature maps through the local optimization module, and perform feature fusion on the extracted multi-scale features to obtain an optimized feature map; perform convolution and up-sampling processing on the optimized feature map and deep feature maps in the plurality of enhanced feature maps through the progressive generation module to generate a super-resolution astronomical image; calculate a loss value based on the original astronomical image and the super-resolution astronomical image, and update the network parameters of the super-resolution reconstruction network through the loss value until the super-resolution reconstruction network converges to obtain a trained super-resolution reconstruction model; The image reconstruction unit is used to obtain a low-resolution astronomical image taken by a mobile device, and reconstruct the low-resolution astronomical image through the super-resolution reconstruction model to obtain a super-resolution astronomical image.

9. An electronic device, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the astronomical image super-resolution reconstruction method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and when the program codes are executed by a processor, the method for super-resolution reconstruction of astronomical images according to any one of claims 1 to 7 is implemented.

Images