Super-resolution reconstruction method for remote sensing images based on convolution scale uncertainty

Abstract

This invention discloses a super-resolution reconstruction method for remote sensing images based on convolutional scale uncertainty, comprising the following steps: 1) inputting a low-resolution remote sensing image into two convolutional modules of different scales to obtain two first shallow feature vectors of different scales; 2) obtaining two second shallow feature vectors as outputs; 3) extracting deep feature vectors; 4) obtaining updated deep feature vectors; 5) fusing the two updated deep feature vectors and passing them through a convolutional layer to obtain the residual vector required for super-resolution; 6) performing bilinear interpolation on the input low-resolution remote sensing image and adding the residual vector to obtain the final super-resolution reconstruction result. This invention's super-resolution reconstruction method for remote sensing images based on convolutional scale uncertainty adds a convolutional scale uncertainty module during the super-resolution reconstruction process, resulting in a clearer image with better detail recovery.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a super-resolution reconstruction method for remote sensing images based on convolution scale uncertainty. Background Technology

[0002] Remote sensing images are typically obtained by sensors mounted on aircraft at low, high, or outer altitudes, capturing the reflection of optical signals from the Earth's surface. These images are then processed using a series of image processing techniques. As a common method of Earth observation, remote sensing images provide rich information about the Earth's surface and can be effectively applied in agriculture, military, surveying, geological monitoring, climate early warning, forest fire prevention, and many other fields. However, due to limitations in detector performance and the technical means of remote sensing, remote sensing images are often limited by resolution, making it difficult to clearly and accurately identify surface targets, thus hindering interpretation and application. Researchers have conducted extensive research to address this problem, typically employing deep learning-based methods. Currently, most deep learning-based methods do not explicitly distinguish between detailed regions requiring more attention and flat regions requiring less attention, treating them with the same computational load. This results in a severe imbalance between computational load and demand: detailed regions requiring excessive computation receive insufficient attention, making it difficult to restore them effectively; while flat regions requiring less computation incur additional computational overhead, wasting resources without contributing to the super-resolution reconstruction results. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a super-resolution reconstruction method for remote sensing images based on convolution scale uncertainty, which is aimed at the limitations of detector performance and the low resolution remote sensing images obtained by remote sensing techniques. The invention proposes a convolution scale uncertainty module, which utilizes the asymmetric differences between features caused by convolution operations at different scales to add an additional attention mechanism in texture-rich regions, thereby obtaining a result image with better detail recovery and clearer image during the super-resolution reconstruction process.

[0004] A method for super-resolution reconstruction of remote sensing images based on convolution scale uncertainty includes the following steps:

[0005] 1) Inputting low-resolution remote sensing images into two convolutional modules of different scales yields two first shallow feature vectors of different scales.

[0006] 2) Input two shallow feature vectors into the convolutional scale uncertainty module to obtain two second shallow feature vectors as outputs. The processing steps of the convolutional scale uncertainty module include obtaining the uncertainty difference between the two input vectors, generating uncertainty attention weights based on this uncertainty difference, and using these uncertainty attention weights to process the input vectors to obtain the output vectors.

[0007] 3) Concatenate the first shallow feature vector with the second shallow feature vector at another scale, and extract the corresponding deep feature vectors.

[0008] 4) Input the two deep feature vectors into the convolutional scale uncertainty module to obtain the updated deep feature vectors.

[0009] 5) The two updated deep feature vectors are fused together, and the residual vector required for super-resolution is obtained through a convolutional layer.

[0010] 6) After bilinear interpolation of the input low-resolution remote sensing image, the residual vector is added to obtain the final super-resolution reconstruction result.

[0011] As one preferred embodiment, the method for generating uncertainty attention weights involves taking the maximum and average vectors between uncertain difference channels, concatenating their features, and then performing convolution and layer normalization operations on the concatenated vectors to obtain the uncertainty attention weights.

[0012] As one preferred embodiment, step 4) further includes a loop operation step of using the updated deep feature vector as input to the convolution scale uncertainty module again to obtain a re-updated deep feature vector.

[0013] As one preferred embodiment, the cyclic operation step is repeated 3-4 times.

[0014] As one of the preferred solutions, the information fusion processing in step 5) includes concatenating the two updated deep feature vectors and processing them by the residual network module to obtain low-resolution fusion features, and then upsampling them to high-resolution fusion features after super-resolution amplification by the up-projection module.

[0015] The advantages and beneficial effects of this invention are as follows:

[0016] The remote sensing image super-resolution reconstruction method based on convolutional scale uncertainty of this invention adds a convolutional scale uncertainty module to the super-resolution reconstruction process. Utilizing the asymmetric differences between features caused by convolution operations at different scales, it adds an additional attention mechanism to texture-rich regions, thereby obtaining a clearer and more detailed image during super-resolution reconstruction. The method first inputs low-resolution remote sensing images into convolutional modules at scale 1 and scale 2 respectively for shallow feature extraction, obtaining shallow features at scale 1 and scale 2. Next, the shallow features are input into the convolutional scale uncertainty module, using the differences between scale 1 and scale 2 features to obtain attention weights, which are then used to process the shallow features at scale 1 and scale 2 before being output to the next module. Finally, the shallow features at scale 1 and scale 2 processed by the convolutional scale uncertainty module are concatenated and input into deep feature extraction modules at different scales to obtain the corresponding high-dimensional representations of the deep features. The deep feature vectors are then input into the convolutional scale uncertainty module to obtain processed deep vectors. These processed deep vectors are then concatenated with the input deep feature vectors and input into the convolutional scale uncertainty module again. This process alternates with the deep feature extraction module to obtain deep feature vectors at scales one and two with rich information. The deep feature vectors at scales one and two are then input into the feature fusion module to extract the residual vectors required for super-resolution reconstruction. Finally, the residual vectors are added to the bilinear interpolation result of the original low-resolution remote sensing image to obtain the final super-resolution reconstruction result. Attached Figure Description

[0017] Figure 1 This is a flowchart of the remote sensing image super-resolution reconstruction method based on convolution scale uncertainty according to the present invention;

[0018] Figure 2 This is a schematic diagram of the convolution scale uncertainty module;

[0019] Figure 3 This is a schematic diagram of residual network module 1;

[0020] Figure 4 This is a schematic diagram of residual network module 2;

[0021] Figure 5 This is a schematic diagram of the fusion module;

[0022] Figure 6 This is a schematic diagram of the upper projection module;

[0023] Figure 7 A schematic diagram illustrating the results of extracting the uncertainty differences between scales from shallow features;

[0024] Figure 8 A schematic diagram of the uncertainty attention weighting results generated from shallow feature vectors.

[0025] Figure 9 Original low-resolution remote sensing Figure 2 The result of bilinear interpolation;

[0026] Figure 10 The result is a 2x super-resolution reconstruction using this method.

[0027] For those skilled in the art, other related figures can be obtained from the above figures without any creative effort. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solution of the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0029] method:

[0030] 1. Convolutional modules for scale 1 and scale 2

[0031] First, the low-resolution remote sensing image is input into two convolutional modules of different scales to obtain shallow feature vectors at scale one and scale two, as shown below:

[0032]

[0033] in, These are the shallow feature vectors extracted by the scale 1 and scale 2 convolution modules, respectively, where I is the input low-resolution remote sensing image; con v1 For a convolution operation of size 3*3, con v2 The convolution operation is 5x5 in size. Different scales refer to the different sizes of the convolution operations used in the residual network modules. In scale one, the convolution operation is 3x3, and in scale two, it is 5x5. The terms "scale one" and "scale two" below refer to the difference in the size of the convolution operations used in the corresponding modules. In scale one, all convolution operations are 3x3, and in scale two, all are 5x5. The purpose of using two different scales is that the 3x3 and 5x5 convolution operations process different features. 3x3 focuses on processing small texture details in the image, while 5x5 focuses on processing boundary information at intermediate and large scales in the image.

[0034] 2. Convolution Scale Uncertainty Module

[0035] The convolutional scale uncertainty module utilizes the uncertainty differences between features caused by convolution operations at different scales. It adds more attention mechanisms in texture-rich regions and reduces attention mechanisms in flat regions to achieve better detail recovery and clearer images during super-resolution reconstruction. The specific structure is shown in the attached figure. Figure 2 As shown. First, the shallow feature vectors of scale one and scale two are... The inputs are subtracted and their absolute values ​​are taken from the inputs in the convolutional scale uncertainty module to extract the uncertainty differences between scales, as shown below:

[0036]

[0037] in, To account for the uncertainty differences between the extracted scales, the absolute value of abs(·) is taken. Then, for each scale... The operations for retrieving the maximum value and average value between channels are as follows:

[0038]

[0039] in, The differences in uncertainty between scales The average and maximum vectors between channels are calculated, with AVE(·) and MAX(·) representing the average and maximum values ​​between channels, respectively. Then, the average and maximum vectors obtained above are concatenated. Next, convolution and layer normalization operations are performed on the concatenated vectors to generate uncertain attention weights, as shown below:

[0040]

[0041] in, Uncertainty attention weights are generated for shallow feature vectors. `norm(·)` is a layer normalization operation, `conv1` is a 3x3 convolution operation, and `(·,·)` is a vector concatenation operation. The uncertainty attention weights are extracted based on the mean and maximum differences between the two scale operations, thus requiring a smaller 3x3 convolution operation to focus on small texture details. Finally, the uncertainty attention weights are applied to the shallow feature vectors respectively. The data is processed and then output to the corresponding branch for the next step, as shown below:

[0042]

[0043] Here, This is a shallow feature vector of scale one after processing with uncertainty attention weights. This represents the shallow feature vector at scale 2 after processing with uncertainty attention weights, and · represents the dot product operation. In subsequent steps, The input will be fed into the branch of scale two for the next operation. The input will be fed into the branch of scale one for the next operation. The interleaving is to fully fuse the feature vectors at the scale level, so that the corresponding feature vectors contain richer information.

[0044] 3. Deep Feature Extraction

[0045] In this step, the shallow feature vectors of scale one are first... Shallow feature vectors of scale 2 after uncertainty attention weighting Perform vector concatenation, and combine the shallow feature vectors of scale two. Shallow feature vectors of scale one after uncertainty attention weighting Vector concatenation is performed. Then, these concatenations are input into the scale-1 deep feature vector extraction module and the scale-2 deep feature vector extraction module, respectively, for deep feature vector extraction, as shown below:

[0046]

[0047] in, These are the deep feature vectors at scale one and scale two, respectively. `resblock1` is the deep feature extraction module at scale one, using a classic residual network structure, as shown in the attached diagram. Figure 3 As shown in the figure. resblock2 is a scale-two deep feature extraction module, which adopts the classic residual network module. The specific structure is shown in the attached figure. Figure 4 As shown.

[0048] 4. The uncertainty attention module and the deep feature vector extraction module are executed alternately.

[0049] After extracting the deep feature vectors, they are input into the convolutional scale uncertainty module to generate deep feature vectors after uncertainty weight processing. Then, these vectors are concatenated and input again into the convolutional scale uncertainty module to generate new deep feature vectors after uncertainty weight processing. This process is repeated three times. The purpose of this repetition is to ensure that the final feature vectors fully encompass multi-scale features, allowing for more complete fusion of information across scales. The resulting deep feature vectors at scales one and two, after sufficient uncertainty weight processing, can fully express the imbalanced feature information contained in the image itself. The specific repetitive processing flow is detailed in the appendix. Figure 1 As shown. The deep feature vector after processing by the first convolutional scale uncertainty module can be calculated by the following formula:

[0050]

[0051] in, The absolute value operation of abs(·) is performed to extract the inter-scale uncertainty difference from the first convolution scale uncertainty module. The uncertainty attention weights are generated for the first deep feature vector, norm(·) is the layer normalization operation, conv1 is the 3*3 convolution operation, and (·,·) is the vector concatenation operation. This represents the scale-1 deep feature vector after the first convolutional scale uncertainty module. is the deep feature vector at scale 2 after processing by the first convolution scale uncertainty module, and · represents the dot product operation.

[0052] Next, the second deep feature vector extraction module is executed as follows:

[0053]

[0054] in, These are the scale 1 deep feature vector and scale 2 deep feature vector obtained from the second deep feature vector extraction module, respectively. resblock1 and resblock2 are the attached... Figure 3 Appendix Figure 4 The residual network module shown.

[0055] Then, the deep feature vector processed by the second convolutional scale uncertainty module can be calculated by the following formula:

[0056]

[0057] in, The absolute value operation of abs(·) is performed to extract the inter-scale uncertainty difference from the second convolutional scale uncertainty module. Uncertainty attention weights are generated for the second deep feature vector, norm(·) is the layer normalization operation, conv1 is the 3*3 convolution operation, and (·,·) is the vector concatenation operation. This represents the scale-1 deep feature vector after the second convolutional scale uncertainty module. is the deep feature vector of scale 2 after processing by the second convolution scale uncertainty module, and · represents the dot product operation.

[0058] Next, the third layer feature vector extraction module is executed, as shown below:

[0059]

[0060] in, These are the scale 1 and scale 2 deep feature vectors obtained from the third deep feature vector extraction module, respectively. resblock1 and resblock2 are the attached... Figure 3 Appendix Figure 4The residual network module is shown. The deep feature vector after processing by the third convolutional scale uncertainty module can be calculated by the following formula:

[0061]

[0062]

[0063] in, The absolute value operation of abs(·) is performed to extract the inter-scale uncertainty difference from the third convolution scale uncertainty module. Uncertainty attention weights are generated for the third deep feature vector, norm(·) is the layer normalization operation, conv1 is the 3*3 convolution operation, and (·,·) is the vector concatenation operation. This represents the scale-1 deep feature vector following the third convolutional scale uncertainty module. is the deep feature vector of scale 2 after processing by the third convolution scale uncertainty module, and · represents the dot product operation.

[0064] 5. Feature Fusion Module

[0065] By alternately executing the uncertainty attention module and the deep feature vector extraction module, deep feature vectors at scales one and two, after sufficient processing of uncertainty weights, are obtained. Then, feature fusion is performed, fully fusing the information from the scale-one and scale-two processed deep feature vectors. Next, an upprojection module is used to increase the resolution of the fused features. Finally, a final convolutional layer is used to process the data, obtaining the residual vector required for the final super-resolution.

[0066] In detail, After vector concatenation, the vectors are processed through one resblock1 layer, and then passed through a convolutional layer to obtain a fully fused feature vector, as shown below:

[0067]

[0068] Among them, F mix To obtain the fully fused feature vector, conv1(·) is a 3*3 convolution operation, U p Projection(·) is the top projection module shown in the attached figure, and resblock1(·) is the residual network module shown in the attached figure. Using a smaller 3x3 convolution operation for extraction allows for full attention to small texture details.

[0069] 6. Super-resolution reconstruction module

[0070] Finally, the input low-resolution remote sensing image is bilinearly interpolated and then the fused residual vector is added to obtain the final super-resolution reconstruction result, as shown below:

[0071] OUT = bic(I) + F mix

[0072] Where OUT represents the final super-resolution reconstructed image, bic() is the bilinear interpolation operation, and I represents the low-resolution remote sensing image.

[0073] The present invention has been described above by way of example. It should be noted that any simple modifications, alterations or other equivalent substitutions that can be made by those skilled in the art without creative effort without departing from the core of the present invention fall within the protection scope of the present invention.

Claims

1. A method for super-resolution reconstruction of remote sensing images based on convolution scale uncertainty, characterized in that, Includes the following steps, 1) Inputting low-resolution remote sensing images into two convolutional modules of different scales yields two first shallow feature vectors of different scales. 2) Input two shallow feature vectors into the convolutional scale uncertainty module to obtain two second shallow feature vectors as outputs. The processing steps of the convolutional scale uncertainty module include obtaining the uncertainty difference between the two input vectors, generating uncertainty attention weights based on this uncertainty difference, and using these uncertainty attention weights to process the input vectors to obtain the output vectors. 3) Concatenate the first shallow feature vector with the second shallow feature vector at another scale, and extract the corresponding deep feature vectors. 4) Input the two deep feature vectors into the convolutional scale uncertainty module to obtain the updated deep feature vectors. 5) The two updated deep feature vectors are fused together, and the residual vector required for super-resolution is obtained through a convolutional layer. 6) After performing bilinear interpolation on the input low-resolution remote sensing image, add the aforementioned residual vector to obtain the final super-resolution reconstruction result. The method for generating uncertainty attention weights involves concatenating the maximum and average vectors between uncertain difference channels, and then performing convolution and layer normalization operations on the concatenated vectors to obtain the uncertainty attention weights.

2. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty as described in claim 1, characterized in that, Step 4) further includes a loop operation step of using the updated deep feature vector as input to the convolution scale uncertainty module again to obtain the updated deep feature vector.

3. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty as described in claim 2, characterized in that, The cyclic operation steps are repeated 3 to 4 times.

4. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty as described in claim 1, characterized in that, The information fusion processing in step 5) includes concatenating the two updated deep feature vectors and processing them by the residual network module to obtain low-resolution fusion features, and then upsampling them to high-resolution fusion features after super-resolution amplification by the upprojection module.

Images