Remote sensing image super-division reconstruction method based on convolution scale uncertainty

Abstract

The invention discloses a remote sensing image super-division reconstruction method based on convolution scale uncertainty, which comprises the following steps: 1) inputting a low-resolution remote sensing image into two convolution modules with different scales to obtain two first shallow feature vectors with different scales, 2) obtaining the output of two second shallow feature vectors, 3) extracting a deep feature vector, and 4) extracting a deep feature vector; the method comprises the following steps of: (1) updating deep feature vectors, (2) updating the deep feature vectors, (3) updating the deep feature vectors, (4) obtaining updated deep feature vectors, (5) carrying out information fusion processing on the two updated deep feature vectors, and obtaining a residual vector required by super-resolution through a convolution layer, and (6) carrying out bilinear interpolation on an input low-resolution remote sensing image, and adding the residual vector to obtain a final super-resolution reconstruction result. According to the remote sensing image super-resolution reconstruction method based on convolution scale uncertainty, a convolution scale uncertainty module is added in the super-resolution reconstruction process, and a clearer result image with better detail recovery is obtained in the super-resolution reconstruction process.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a remote sensing image super-resolution reconstruction method based on convolution scale uncertainty. Background Art

[0002] Remote sensing images are usually obtained by sensors carried on aircraft at low altitude, high altitude or space to obtain the reflection information of optical signals on the earth's surface, and then obtained through a series of image processing methods. As a common means of ground perception, remote sensing images have obtained rich surface observation information and can be effectively applied in agriculture, military, surveying and mapping, geological detection, climate warning, forest fire prevention and other fields. However, due to the performance limitations of the detector itself and the technical means of remote perception, remote sensing images are usually limited by resolution, making it difficult to clearly and accurately identify surface targets, which hinders the corresponding interpretation and application work. Researchers have carried out a lot of research on this issue, and usually use deep learning-based methods to solve this problem. At present, most deep learning-based methods do not explicitly distinguish between areas with rich details that require more attention and flat areas that do not require too much attention, and keep them with the same amount of calculation, resulting in a serious imbalance between the amount of calculation and the amount of demand, so that: the attention to the detailed areas that require too much calculation is too little, making it difficult to restore the detailed areas well; the flat areas that do not require too much calculation increase the extra amount of calculation, resulting in a waste of resources without having a beneficial effect on the super-resolution reconstruction results. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art, and to provide a remote sensing image super-resolution reconstruction method based on convolution scale uncertainty for low-resolution remote sensing images obtained by means of technical means of remote perception due to the limited performance of the detector itself. A convolution scale uncertainty module is proposed, which utilizes the asymmetric differences between features caused by convolution operations of different scales, and adds additional attention mechanisms in texture-rich areas, so as to obtain a result image with better detail restoration and clearer result in the super-resolution reconstruction process.

[0004] A remote sensing image super-resolution reconstruction method based on convolution scale uncertainty comprises the following steps:

[0005] 1) Input the low-resolution remote sensing image into two convolution modules of different scales to obtain the first shallow feature vectors of two different scales.

[0006] 2) inputting the two shallow feature vectors into the convolutional scale uncertainty module and obtaining the output of two second shallow feature vectors, wherein the processing step of the convolutional scale uncertainty module includes obtaining the uncertainty difference between the two input vectors and generating uncertainty attention weights according to the uncertainty difference, and using the uncertainty attention weights to process the input vectors respectively to obtain output vectors,

[0007] 3) respectively concatenate the first shallow feature vector with the second shallow feature vector of another scale and extract the corresponding deep feature vector,

[0008] 4) Inputting the two deep feature vectors into the convolutional scale uncertainty module to obtain the updated deep feature vector,

[0009] 5) The two updated deep feature vectors are fused and the residual vector required for super-resolution is obtained through a convolution layer.

[0010] 6) Perform bilinear interpolation on the input low-resolution remote sensing image and add the residual vector to obtain the final super-resolution reconstruction result.

[0011] As one of the preferred schemes, the method for generating uncertainty attention weights is to obtain the maximum value vector and the average value vector between uncertain difference channels, then perform feature splicing on them, and perform convolution and layer normalization operations on the spliced ​​vectors in sequence to obtain the generated uncertainty attention weights.

[0012] As one of the preferred solutions, the step 4) also includes a cyclic operation step of using the updated deep feature vector as an input of the convolutional scale uncertainty module again to obtain a re-updated deep feature vector.

[0013] As one of the preferred embodiments, the cyclic operation step is repeated 3-4 times.

[0014] As one of the preferred schemes, the information fusion processing in the step 5) includes splicing the two updated deep feature vectors and processing them by the residual network module to obtain a low-resolution fusion feature, and upsampling it through the upper projection module to a high-resolution fusion feature after super-resolution amplification.

[0015] The advantages and beneficial effects of the present invention are:

[0016] The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty of the present invention adds a convolution scale uncertainty module in the super-resolution reconstruction process, and uses the asymmetric differences between features caused by convolution operations of different scales to add additional attention mechanisms in texture-rich areas, thereby obtaining a result image with better detail recovery and clearer results in the super-resolution reconstruction process. The method first inputs the low-resolution remote sensing image into the convolution modules of scale one and scale two respectively, extracts shallow features, and obtains shallow features at scale one and scale two; then, the shallow features are input into the convolution scale uncertainty module, and the difference between the features of scale one and scale two is used to obtain the attention weight to process the shallow features of scale one and scale two and output them to the next module; then, the shallow features of scale one and scale two processed by the convolution scale uncertainty module are spliced ​​and respectively input into the deep feature extraction modules of different scales to obtain the deep features of the corresponding high-dimensional expression. layer feature vector; the deep feature vector is input into the convolution scale uncertainty module again to obtain the processed deep vector, which is concatenated with the input deep feature vector respectively, and then input into the convolution scale uncertainty module again to be processed alternately with the deep feature extraction module to obtain the deep feature vectors of scale one and scale two with rich information; the deep feature vectors of scale one and scale two are input into the feature fusion module to extract the residual vector required for super-resolution reconstruction; finally, the residual vector is added to the bilinear interpolation result of the original low-resolution remote sensing image to obtain the final super-resolution reconstruction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of the remote sensing image super-resolution reconstruction method based on convolution scale uncertainty of the present invention;

[0018] Figure 2 Schematic diagram of the convolutional scale uncertainty module;

[0019] Figure 3 Schematic diagram of residual network module 1;

[0020] Figure 4 Schematic diagram of residual network module 2;

[0021] Figure 5 It is a schematic diagram of the fusion module;

[0022] Figure 6 Schematic diagram of the upper projection module;

[0023] Figure 7 Schematic diagram of the uncertainty differences between scales extracted from shallow features;

[0024] Figure 8 Schematic diagram of the uncertainty attention weight results generated by shallow feature vectors.

[0025] Fig. 9 Original low-resolution remote sensing Figure 2 times the bilinear interpolation result;

[0026] Fig.10 This is the reconstruction result of 2x super-resolution using this method.

[0027] For ordinary technicians in this field, other relevant drawings can be obtained based on the above drawings without any creative work. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0029] method:

[0030] 1. Convolutional modules of scale 1 and scale 2

[0031] First, the low-resolution remote sensing image is input into two convolution modules of different scales to obtain shallow feature vectors of scale one and scale two, as shown below:

[0032]

[0033] in, are the shallow feature vectors extracted by the scale-1 and scale-2 convolution modules, respectively, and I is the input low-resolution remote sensing image; con v1 is a convolution operation of size 3*3, con v2 The convolution operation is 5*5 in size. Different scales refer to the different sizes of convolution operations used in the residual network module. The convolution operation in scale one is 3*3 in size, and the convolution operation in scale two is 5*5 in size. The scale one and scale two mentioned below refer to the difference in the size of convolution operations used in the corresponding modules. The convolution operations in scale one are all 3*3 in size, and the convolution operations in scale two are all 5*5 in size. The purpose of using two different scales is that the features processed by the convolution operations of 3*3 and 5*5 are different. 3*3 focuses on the processing of small texture details in the image, and 5*5 focuses on processing the boundary information of the intermediate scale and large scale in the image.

[0034] 2. Convolutional Scale Uncertainty Module

[0035] The convolution scale uncertainty module uses the uncertainty differences between features caused by convolution operations of different scales to increase more attention mechanisms in texture-rich areas and reduce attention mechanisms in flat areas to achieve better detail restoration and clearer result images in the super-resolution reconstruction process. The specific structure is shown in the attached figure. Figure 2 First, the shallow feature vectors of scale one and scale two are The input is sent to the convolutional scale uncertainty module for subtraction and absolute value operation to extract the uncertainty difference between scales, as shown below:

[0036]

[0037] in, is the uncertainty difference between the extracted scales, and abs(·) takes the absolute value operation. Take the maximum value and average value between channels as follows:

[0038]

[0039] in, The uncertainty differences between scales are The average value vector and maximum value vector between channels are AVE(·) and MAX(·), respectively, which are the average value and maximum value operations between channels. Then, the average value vector and maximum value vector solved above are concatenated. Next, the concatenated vectors are convolved and layer normalized in turn to generate uncertainty attention weights, as shown below:

[0040]

[0041] in, The uncertainty attention weights generated for the shallow feature vectors, norm(·) is the layer normalization operation, conv1 is a 3*3 convolution operation, (·,·) is a vector concatenation operation, where the uncertainty attention weights are extracted based on the mean and maximum differences of the two scale operations, so a smaller 3*3 convolution operation is required to extract and focus on small texture detail information. Finally, the uncertainty attention weights are used to analyze the shallow feature vectors. Process it and then output it to the corresponding branch for the next step, as shown below:

[0042]

[0043] Here, is the shallow feature vector of scale 1 after uncertainty attention weight processing, is the shallow feature vector of scale 2 after uncertainty attention weight processing, and is the dot product operation. In the subsequent steps, It will be input into the branch of scale 2 for the next step. It will be input into the branch of scale one for the next step. The purpose of interleaving is to fully integrate the feature vectors at the scale level, and the corresponding feature vectors will contain richer information.

[0044] 3. Deep feature extraction

[0045] In this step, we first transform the shallow feature vector of scale 1 into The shallow feature vector of scale 2 after processing with uncertainty attention weights Perform vector concatenation and convert the shallow feature vector of scale 2 The shallow feature vector of scale 1 after processing with uncertainty attention weights Then, they are input into the scale one deep feature vector extraction module and the scale two deep feature vector extraction module respectively to extract deep feature vectors, as shown below:

[0046]

[0047] in, are the deep feature vectors of scale 1 and scale 2 respectively. resblock1 is the deep feature extraction module of scale 1, and the network structure adopted is the classic residual network module. The specific structure is shown in the attached figure. Figure 3 As shown in the figure, resblock2 is a deep feature extraction module of scale 2. The network structure adopted is the classic residual network module. The specific structure is shown in the figure. Figure 4 shown.

[0048] 4. The uncertainty attention module and the deep feature vector extraction module are executed alternately

[0049] After the deep feature vector is extracted, it is input into the convolution scale uncertainty module to generate a deep feature vector after uncertainty weight processing. Then, it is concatenated and input into the convolution scale uncertainty module again to generate a new deep feature vector after uncertainty weight processing. This process is repeated 3 times. The purpose of repeating this process is to make the final feature vector fully contain multi-scale features and make the information fusion between scales more sufficient. The deep feature vectors of scales 1 and 2 that have been fully processed with uncertainty weights can fully express the unbalanced feature information contained in the image. The specific repeated processing process is shown in the attached. Figure 1 As shown. The deep feature vector after the first convolution scale uncertainty module processing can be calculated by the following formula:

[0050]

[0051] in, It is the uncertainty difference between scales extracted by the first convolutional scale uncertainty module, and abs(·) takes the absolute value operation. is the uncertainty attention weight generated for the first deep feature vector, norm(·) is the layer normalization operation, conv1 is the convolution operation of size 3*3, and (·,·) is the vector concatenation operation; is the deep feature vector of scale 1 after the first convolutional scale uncertainty module, is the deep feature vector of scale 2 after the first convolutional scale uncertainty module processing, and is the dot product operation.

[0052] Next, the second deep feature vector extraction module is executed as follows:

[0053]

[0054] in, They are the deep feature vectors of scale 1 and scale 2 obtained by the second deep feature vector extraction module, resblock1 and resblock2 are attached Figure 3 , Attachment Figure 4 The residual network module shown.

[0055] Then, the deep feature vector after the second convolutional scale uncertainty module can be calculated as follows:

[0056]

[0057] in, The uncertainty difference between scales extracted by the second convolutional scale uncertainty module, abs(·) takes the absolute value operation. ; is the uncertainty attention weight generated for the second deep feature vector, norm(·) is the layer normalization operation, conv1 is the convolution operation of size 3*3, and (·,·) is the vector concatenation operation; is the deep feature vector of scale one after the second convolutional scale uncertainty module, is the deep feature vector of scale 2 after the second convolutional scale uncertainty module processing, and is the dot product operation.

[0058] Next, the third layer feature vector extraction module is executed as follows:

[0059]

[0060] in, They are the deep feature vectors of scale 1 and scale 2 obtained by the third deep feature vector extraction module, resblock1 and resblock2 are attached Figure 3 , Attachment Figure 4The residual network module shown. The deep feature vector after the third convolution scale uncertainty module processing can be calculated as follows:

[0061]

[0062]

[0063] in, It is the uncertainty difference between scales extracted by the third convolutional scale uncertainty module, and abs(·) takes the absolute value operation. is the uncertainty attention weight generated for the third deep feature vector, norm(·) is the layer normalization operation, conv1 is the convolution operation of size 3*3, and (·,·) is the vector concatenation operation; is the deep feature vector of scale one after the third convolutional scale uncertainty module, is the deep feature vector of scale 2 after being processed by the third convolutional scale uncertainty module, and is the dot product operation.

[0064] 5. Feature Fusion Module

[0065] By alternately executing the above uncertainty attention module and the deep feature vector extraction module, we obtain the deep feature vectors of scale 1 and scale 2 after the uncertainty weights are fully processed. Then, feature fusion is performed to fully fuse the deep feature vectors processed at scales 1 and 2. Then, the resolution of the features after information fusion is increased by the up-projection module. Finally, the last convolutional layer is used for processing to obtain the residual vector required for the final super-resolution.

[0066] In detail, After the vectors are concatenated, they are processed in a resblock1 and then go through a convolutional layer to obtain a fully fused feature vector, as shown below:

[0067]

[0068] Among them, F mix is the fully fused feature vector, conv1(·) is a convolution operation of size 3*3, and U p Projection(·) is the upper projection module shown in the figure, and resblock1(·) is the residual network module shown in the figure. A smaller 3*3 convolution operation is used for extraction, which can fully focus on small texture detail information.

[0069] 6. Super-resolution reconstruction module

[0070] Finally, the input low-resolution remote sensing image is bilinearly interpolated and added to the fused residual vector to obtain the final super-resolution reconstruction result, as shown below:

[0071] OUT=bic(I)+F mix

[0072] Among them, OUT is the final super-resolution reconstruction result image, bic() is the bilinear interpolation operation, and I is the low-resolution remote sensing image.

[0073] The present invention is described above by way of example. It should be noted that, without departing from the core of the present invention, any simple deformation, modification or other equivalent replacement that can be made by those skilled in the art without inventive effort falls within the protection scope of the present invention.

Claims

1. A remote sensing image super-resolution reconstruction method based on convolution scale uncertainty, characterized in that: The following steps are included: 1) Input the low-resolution remote sensing image into two convolution modules of different scales to obtain the first shallow feature vectors of two different scales. 2) inputting the two shallow feature vectors into the convolutional scale uncertainty module and obtaining the output of two second shallow feature vectors, wherein the processing step of the convolutional scale uncertainty module includes obtaining the uncertainty difference between the two input vectors and generating uncertainty attention weights according to the uncertainty difference, and using the uncertainty attention weights to process the input vectors respectively to obtain output vectors, 3) respectively concatenate the first shallow feature vector with the second shallow feature vector of another scale and extract the corresponding deep feature vector, 4) Inputting the two deep feature vectors into the convolutional scale uncertainty module to obtain the updated deep feature vector, 5) The two updated deep feature vectors are fused and the residual vector required for super-resolution is obtained through a convolution layer. 6) Perform bilinear interpolation on the input low-resolution remote sensing image and add the residual vector to obtain the final super-resolution reconstruction result.

2. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty according to claim 1, characterized in that: The method for generating uncertainty attention weights is to obtain the maximum value vector and the average value vector between uncertain difference channels, then perform feature splicing on them, and perform convolution and layer normalization operations on the spliced ​​vectors in sequence to obtain the generated uncertainty attention weights.

3. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty according to claim 1, characterized in that: The step 4) also includes a cyclic operation step of using the updated deep feature vector as an input to the convolutional scale uncertainty module again to obtain a deep feature vector that is updated again.

4. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty according to claim 3, characterized in that: The number of cycles of the cyclic operation step is 3-4 times.

5. The remote sensing image super-resolution reconstruction method based on convolution scale uncertainty according to claim 1, characterized in that: The information fusion processing in the step 5) includes concatenating the two updated deep feature vectors and processing them by the residual network module to obtain a low-resolution fusion feature, and upsampling it by the upper projection module to a high-resolution fusion feature after super-resolution amplification.

Images