Remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution

Abstract

This invention belongs to the field of computer vision technology, specifically relating to an arbitrary-scale super-resolution method for remote sensing images based on dynamic-scale frequency-domain convolution. The method includes constructing and training an arbitrary-scale super-resolution network model, obtaining the image to be processed as input to the trained arbitrary-scale super-resolution network model to obtain a reconstructed image. The arbitrary-scale super-resolution network model includes a feature extraction module, a dynamic-scale frequency-domain feature learning module, a cross-scale non-local upsampling module, and an MFGLIF module. This invention improves the network's ability to recover missing structures and texture details, as well as its ability to handle discontinuous mappings between coordinates and signal values.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution. Background Technology

[0002] Arbitrary scale super-resolution of remote sensing images is a key image processing technology that aims to reconstruct high-resolution images at arbitrary magnification from low-resolution images, thereby improving the accuracy of spatial resolution and ground feature details to meet the needs of different application scenarios.

[0003] Existing research methods have achieved significant visual effects in arbitrary-scale super-resolution of remote sensing images based on deep learning. Current methods typically employ discrete or continuous representations, dynamically predicting filter weights or constructing implicit neural functions to map query coordinates to signal values ​​based on scale factors. However, these methods generally use static convolutions and scale factors in predicting filters or constructing implicit neural functions, neglecting the dynamic feature extraction requirements at different scale factors and the inherent characteristics of RSI. Furthermore, as the scale factor increases, local features extracted from the input image suffer from severe loss of structural and texture details, and the network frequently faces challenges such as shape distortion and checkerboard artifacts. Therefore, exploring new strategies to achieve dynamic scale feature extraction to supplement local features is crucial.

[0004] Learning continuous image representations enables the reconstruction of high-resolution images of arbitrary scales from low-resolution inputs. Local ensembles and implicit neural functions are used to map query coordinates to signal values ​​based on latent codes in a single feature space. However, local ensembles utilize a finite receptive field to calculate set weights, typically calculated by the area of ​​the rectangle between the query coordinates and each nearest neighbor coordinate. This ignores the loss of visual information due to neglecting coordinates and local features, and fails to adequately consider the integration of globally relevant features of the image. Furthermore, due to the strong randomness of RSI texture distribution, predicting signal values ​​using only local latent codes in a single feature space often leads to discontinuities. Therefore, how to fully utilize global information and combine continuous feature representations in multiple feature spaces to enhance the representationality of latent codes and effectively handle discontinuous coordinate-signal value mappings has become a pressing problem. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency-domain convolution. The method includes constructing and training an arbitrary-scale super-resolution network model, and obtaining the image to be processed as input to the trained arbitrary-scale super-resolution network model to obtain a reconstructed image. The arbitrary-scale super-resolution network model includes a feature extraction module, a dynamic-scale frequency-domain feature learning module, a cross-scale nonlocal upsampling module, and an MFGLIF module.

[0006] The training process for arbitrary-scale super-resolution network models includes:

[0007] S1. Input the LR remote sensing image into the feature extraction module to obtain shallow features; the feature extraction module includes a 3×3 convolutional layer and a ReLU activation function layer;

[0008] S2. Input the shallow features into the dynamic scale frequency domain feature learning module to obtain multi-level features; the multi-level features include first-level features, second-level features and third-level features;

[0009] S3. Input the multi-level features into the cross-scale nonlocal upsampling module to obtain multi-scale features; the multi-scale features include first-scale features, second-scale features and third-scale features;

[0010] S4. Input the multi-scale features into the MFGLIF module to obtain the reconstructed high-resolution remote sensing image.

[0011] The beneficial effects of this invention are:

[0012] This invention achieves dynamic scale feature extraction through a novel Scale-Aware Frequency Domain Block (SAFB) and leverages the combined advantages of spatial and frequency domain features to improve the network's ability to recover missing structures and texture details. Considering that local integration in continuous representations only utilizes a limited receptive field, a Cross-Scale Nonlocal Upsampling Module (CSNUB) is introduced to enhance the network's ability to utilize nonlocal feature information. Simultaneously, a Multi-Feature Global-Local Implicit Function (MFGLIF) is introduced to effectively enhance latent code representation and improve the network's ability to handle discontinuous mappings between coordinates and signal values. Attached Figure Description

[0013] Figure 1 This is the training process for the arbitrary-scale super-resolution network model of the present invention;

[0014] Figure 2 This is a schematic diagram of the overall network model structure of a remote sensing image arbitrary-scale super-resolution method based on dynamic scale frequency domain convolution according to the present invention.

[0015] Figure 3 This is a schematic diagram of the iterative branch structure in the dynamic scale frequency domain feature learning module of the present invention;

[0016] Figure 4 This is the dynamic scale-aware convolutional graph of the present invention;

[0017] Figure 5 This is a diagram of the frequency domain adaptive filter of the present invention;

[0018] Figure 6 This is a diagram of the cross-scale nonlocal upsampling module of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention provides a method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution, including constructing and training an arbitrary-scale super-resolution network model, obtaining the image to be processed as input to the trained arbitrary-scale super-resolution network model to obtain the reconstructed image; the arbitrary-scale super-resolution network model includes a feature extraction module, a dynamic-scale frequency domain feature learning module, a cross-scale nonlocal upsampling module, and an MFGLIF module.

[0021] like Figure 1 , Figure 2 As shown, the training process of an arbitrary-scale super-resolution network model includes:

[0022] S1. Input the LR remote sensing image into the feature extraction module to obtain shallow features; the feature extraction module includes a 3×3 convolutional layer and a ReLU activation function layer.

[0023] Specifically, step S1 can be represented as follows:

[0024] F0 = ReLU(Conv3(I) LR ))

[0025] Where F0 represents shallow features, I LR represents an LR remote sensing image, Conv3(·) represents a 3×3 convolutional layer, and ReLU(·) represents a ReLU activation function layer.

[0026] S2. Input the shallow features into the dynamic scale frequency domain feature learning module to obtain multi-level features; the multi-level features include first-level features, second-level features and third-level features.

[0027] Specifically, such as Figure 2 As shown, the Dynamic Scale Frequency Domain Feature Learning Module (DSFFLM) includes three iterative branches. The first iterative branch includes a first SAFM unit, a second SAFM unit, and a third SAFM unit cascaded in sequence; the second iterative branch includes a fourth SAFM unit, a fifth SAFM unit, and a sixth SAFM unit cascaded in sequence; and the third iterative branch includes a seventh SAFM unit, an eighth SAFM unit, and a ninth SAFM unit cascaded in sequence. The nine SAFM units have the same structure, each including a 1×1 convolutional layer and a scale-aware frequency domain block (SAFB).

[0028] In step S2, the shallow features and scale factor r are input into the dynamic scale frequency domain feature learning module. The specific processing steps include:

[0029] S21. Shallow features are input into the first SAFM unit to obtain the first SAFM output. Combine shallow features with the first SAFM output After concatenation, the data is input into the second SAFM unit to obtain the second SAFM output. Shallow features, first SAFM output Second SAFM output After splicing, the first-level features are obtained by inputting them into the third SAFM unit.

[0030] S22. Shallow features and first-level features After concatenation, the input is given to the fourth SAFM unit to obtain the fourth SAFM output. Combine shallow features with the fourth SAFM output After concatenation, the input is given to the fifth SAFM unit to obtain the fifth SAFM output. Shallow features, fourth SAFM output and the fifth SAFM output After splicing, the second-level features are obtained by inputting them into the sixth SAFM unit.

[0031] S23. Shallow features and second-level features After concatenation, the input is given to the seventh SAFM unit to obtain the seventh SAFM output. Combine shallow features with the seventh SAFM output After concatenation, the data is input into the eighth SAFM unit to obtain the eighth SAFM output. Shallow features, seventh SAFM output and the eighth SAFM output After splicing, the third-level features are obtained by inputting them into the ninth SAFM unit.

[0032] All levels of features have the same size.

[0033] Specifically, such as Figure 3 As shown, the scale-aware frequency domain block includes channel dilated convolutional layers, channel compressed convolutional layers, dynamic scale-aware convolutional blocks, ESA channel attention layers, and frequency-domain adaptive filters; the processing of each SAFM unit includes:

[0034] S211. Pass the input features through a 1×1 convolutional layer to obtain convolutional features;

[0035] S212. The convolutional features are passed through a channel dilated convolutional layer with a kernel size of 3×3 to obtain dilated features; the dilated features are passed through a channel compressed convolutional layer with a kernel size of 3×3 to obtain compressed features; the compressed features are passed through a dynamic scale-aware convolutional block to obtain dynamic scale features; the dynamic scale features are passed through an ESA channel attention layer to obtain spatial features.

[0036] S213. Convolutional features are processed by a frequency domain adaptive filter to obtain frequency domain features;

[0037] S214. The output feature is obtained by adding the convolutional feature and the spatial feature and then concatenating them with the frequency domain feature.

[0038] Specifically, the processing procedure of the SAFM unit can be expressed as follows:

[0039] f spa =ESA(DSA(Conv 3,folding (Conv 3,unfolding (Conv1(H i,j )))))

[0040] f fre =FAF(Conv1(H i,j ))

[0041]

[0042] Among them, f spa Describing spatial characteristics, f fre Represents frequency domain characteristics, H represents the output feature of the SAFM unit of the iterative branch at layers i = 1, 2, 3, where j = 1, 2, 3 is the feature that fuses spatial and frequency domain information. i,j This represents the input features of the SAFM unit j = 1, 2, 3 of the iterative branches in layers i = 1, 2, 3, where Conv1(·) represents a 1×1 convolutional layer. 3,unfolding (·) indicates a channel dilated convolutional layer, Conv 3,folding (·) represents a channel compression convolutional layer, DSA(·) represents a dynamic scale-aware convolutional block, ESA(·) represents an ESA channel attention layer, and FAF(·) represents a frequency domain adaptive filter.

[0043] Specifically, such as Figure 4 As shown, the process of obtaining dynamic scale features from compressed features through a dynamic scale-aware convolutional block (DSA-Conv) includes:

[0044] S221. Perform global average pooling on the compressed features to obtain pooled features. After passing the pooled features through a fully connected layer and a ReLU activation function layer, concatenate them with the scale factor r to obtain scaled concatenated features.

[0045] S222. The scale-stitched features are sequentially passed through a fully connected layer, a ReLU activation function layer, a ReLU activation function layer, and a Sigmoid activation function layer to obtain k scale-aware attention weights w1, w2, ..., w k ;

[0046] S223. Scale-aware convolutional kernels are obtained by fusing k prior convolutional kernels of the same size using k scale-aware attention weights.

[0047] S224. Scale-aware convolution kernels are used to process compressed features to obtain dynamic scale features.

[0048] Specifically, such as Figure 5 As shown, the process of obtaining frequency domain features from convolutional features through a frequency domain adaptive filter includes:

[0049] S231. Perform Fourier transform decomposition on the convolutional features to obtain phase P and amplitude A;

[0050] S232. The phase P is processed by a depthwise separable convolution with a kernel size of 3×3, a ReLU activation function, a point convolution with a kernel size of 1×1, and a Sigmoid activation function to obtain the phase mapping weights; the phase is multiplied by the phase mapping weights to obtain the weighted phase information;

[0051] S234. The amplitude A is processed by a depthwise separable convolution with a kernel size of 3×3, a ReLU activation function, a point convolution with a kernel size of 1×1, and a Sigmoid activation function to obtain the amplitude mapping weights; the amplitude is multiplied by the amplitude mapping weights to obtain the weighted amplitude information;

[0052] S235. The frequency domain features are obtained by performing an inverse Fourier transform based on the weighted phase information and weighted amplitude information.

[0053] S3. Input the multi-level features into the cross-scale nonlocal upsampling module to obtain multi-scale features; the multi-scale features include first-scale features, second-scale features and third-scale features.

[0054] Specifically, the cross-scale nonlocal upsampling module (CSNUB) includes nonlocal attention modules at three scales; level features are input into the nonlocal attention modules to supplement global features, and in step 3, multi-level features are input into the cross-scale nonlocal upsampling module to obtain multi-scale features including:

[0055] S31. Input the first-level features into the first-scale non-local attention module to obtain the first-scale features;

[0056] S32. Input the second-level features into the non-local attention module of the second scale to obtain the second-scale features;

[0057] S33. Input the third-level features into the third-scale nonlocal attention module to obtain the third-scale features.

[0058] Specifically, such as Figure 6 As shown, the processing steps of the nonlocal attention module at each scale include:

[0059] S311. Input the input features into the query network δ respectively. q Sum Value Network δ v We obtain the query feature Q and the value feature V; the input features are downsampled using a 3×3 convolution and then input into the key network δ. k Obtain the key feature K;

[0060] S312. After sampling and shaping the key features, the attention weights are calculated by performing operations with the query features, which can be expressed as follows:

[0061]

[0062] Among them, W att The attention weights are represented by T, which represents the matrix transpose.

[0063] S313. After sampling and shaping the logarithmic feature V, it is then compared with the attention weight W. att Attention features are obtained through fusion;

[0064] S314. Concatenate the attention features with the input features to obtain the nonlocal feature F′. i ;

[0065] S315. For nonlocal features F′ i After interpolation and upsampling, the output features are obtained through 3×3 convolution and pixel attention operations, which can be represented as follows:

[0066]

[0067] Among them, F′ i This represents the non-local feature corresponding to the i-th level feature output by the i-th iterative branch. Indicates r i Double upsampling, PA(·) denotes pixel attention operation, and Conv3(·) denotes a 3×3 convolutional layer. Indicates the i-th amplification r i Scale features of multiples.

[0068] S4. Input the multi-scale features into the MFGLIF module (multi-feature global-local implicit function module) to obtain the reconstructed high-resolution remote sensing image.

[0069] Specifically, the local latent code in the multi-feature space is obtained from multi-scale features using the Multi-Feature Global-Local Implicit Function (MFGLIF). After obtaining the enhanced latent code, the coordinate domain is mapped to the signal domain using the implicit function to obtain the reconstructed clear image, which can be represented as follows:

[0070] s=f(L(x,Z1),L(x,Z2),…,L(x,Z N );r)

[0071] Where s represents the predicted RGB value, L(·,Z) t Z represents a local implicit neural function. i This represents the set of the four nearest neighbor latent codes for the query coordinate x in the i-th scale feature, typically {z 00 ,z 01 ,z 10 ,z 11} represents the latent codes for the lower left, upper left, lower right, and lower right coordinates of the query coordinate x in the i-th scale feature, i.e., Z i ={z 00 ,z 01 ,z 10 ,z 11 f(·) is a multi-feature global implicit function, N=3.

[0072] Step S4 specifically includes:

[0073] S41. Obtain the local latent codes of the first-scale features, second-scale features, and third-scale features respectively through local implicit neural functions;

[0074] S42. Input the scale factor into two fully connected layers and one softmax activation function to obtain scale-aware weights;

[0075] S43. The enhanced latent code is obtained by weighting and concatenating the local latent code with the scale-aware weight;

[0076] The process is represented as

[0077] {π1,π2,…,π N} = softmax(FC(FC(r)))

[0078] Z = Concat(π1*L(x,Z1),…π N *L(x,Z N ))

[0079]

[0080] Where Z represents the enhanced latent code, π i Let (u,v) represent the scale weights of the i-th feature space, where (u,v) is the ratio of x to z. 00The position offset is normalized to [0,1].

[0081] S44. The enhanced latent code is input into a global implicit neural function to achieve the mapping between the coordinate domain and the signal domain, thereby obtaining the reconstructed high-resolution remote sensing image.

[0082] S5. Calculate the loss and train the model parameters based on the reconstructed high-resolution remote sensing images until the model parameters converge.

[0083] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "setting," "connection," "fixing," "rotation," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal connection of two components or the interaction between two components. Unless otherwise explicitly limited, those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution, characterized in that, Construct and train an arbitrary-scale super-resolution network model, and obtain the image to be processed by inputting the trained arbitrary-scale super-resolution network model to obtain the reconstructed image; the arbitrary-scale super-resolution network model includes a feature extraction module, a dynamic scale frequency domain feature learning module, a cross-scale nonlocal upsampling module, and an MFGLIF module. The training process for arbitrary-scale super-resolution network models includes: S1. Input the LR remote sensing image into the feature extraction module to obtain shallow features; the feature extraction module includes a 3×3 convolutional layer and a ReLU activation function layer; S2. Input the shallow features into the dynamic scale frequency domain feature learning module to obtain multi-level features; the multi-level features include first-level features, second-level features and third-level features; The dynamic scale-aware frequency domain feature learning module includes three iterative branches. The first iterative branch includes a first SAFM unit, a second SAFM unit, and a third SAFM unit cascaded in sequence. The second iterative branch includes a fourth SAFM unit, a fifth SAFM unit, and a sixth SAFM unit cascaded in sequence. The third iterative branch includes a seventh SAFM unit, an eighth SAFM unit, and a ninth SAFM unit cascaded in sequence. The nine SAFM units have the same structure, each including a 1×1 convolutional layer and a scale-aware frequency domain block. Step S2 specifically includes the following processing steps: S21. Shallow features are input into the first SAFM unit to obtain the first SAFM output. Combine shallow features with the first SAFM output After concatenation, the data is input into the second SAFM unit to obtain the second SAFM output. Shallow features, first SAFM output Second SAFM output After splicing, the first-level features are obtained by inputting them into the third SAFM unit. S22. Shallow features and first-level features After concatenation, the input is given to the fourth SAFM unit to obtain the fourth SAFM output. Combine shallow features with the fourth SAFM output After concatenation, the input is given to the fifth SAFM unit to obtain the fifth SAFM output. Shallow features, fourth SAFM output and the fifth SAFM output After splicing, the second-level features are obtained by inputting them into the sixth SAFM unit. S23. Shallow features and second-level features After concatenation, the input is given to the seventh SAFM unit to obtain the seventh SAFM output. Combine shallow features with the seventh SAFM output After concatenation, the data is input into the eighth SAFM unit to obtain the eighth SAFM output. Shallow features, seventh SAFM output and the eighth SAFM output After splicing, the third-level features are obtained by inputting them into the ninth SAFM unit. S3. Input the multi-level features into the cross-scale nonlocal upsampling module to obtain multi-scale features; the multi-scale features include first-scale features, second-scale features and third-scale features; The cross-scale nonlocal upsampling module includes three scales of nonlocal attention modules; step 3 inputs multi-level features into the cross-scale nonlocal upsampling module to obtain multi-scale features including: S31. Input the first-level features into the first-scale non-local attention module to obtain the first-scale features; S32. Input the second-level features into the non-local attention module of the second scale to obtain the second-scale features; S33. Input the third-level features into the third-scale nonlocal attention module to obtain the third-scale features; S4. Input the multi-scale features into the MFGLIF module to obtain the reconstructed high-resolution remote sensing image; Step S4 specifically includes: S41. Obtain the local latent codes of the first-scale features, second-scale features, and third-scale features respectively through local implicit neural functions; S42. Input the scale factor into two fully connected layers and one softmax activation function to obtain scale-aware weights; S43. The enhanced latent code is obtained by weighting and concatenating the local latent code with the scale-aware weight; S44. Input the enhanced latent code into a global implicit neural function to obtain a reconstructed high-resolution remote sensing image.

2. The method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution according to claim 1, characterized in that, The scale-aware frequency domain block includes channel dilated convolutional layers, channel compressed convolutional layers, dynamic scale-aware convolutional blocks, ESA channel attention layers, and frequency domain adaptive filters. The processing procedure for each SAFM unit includes: S211. Pass the input features through a 1×1 convolutional layer to obtain convolutional features; S212. The convolutional features are passed through a channel dilated convolutional layer with a kernel size of 3×3 to obtain dilated features; the dilated features are passed through a channel compressed convolutional layer with a kernel size of 3×3 to obtain compressed features; the compressed features are passed through a dynamic scale-aware convolutional block to obtain dynamic scale features; the dynamic scale features are passed through an ESA channel attention layer to obtain spatial features. S213. Convolutional features are processed by a frequency domain adaptive filter to obtain frequency domain features; S214. The output feature is obtained by adding the convolutional feature and the spatial feature and then concatenating them with the frequency domain feature.

3. The method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution according to claim 2, characterized in that, The process of obtaining dynamic scale features from compressed features through dynamic scale-aware convolutional blocks includes: S221. Perform global average pooling on the compressed features to obtain pooled features. After passing the pooled features through a fully connected layer and a ReLU activation function layer, concatenate them with a scaling factor to obtain scaled concatenated features. S222. The scale-stitched features are sequentially passed through a fully connected layer, a ReLU activation function layer, another fully connected layer, a ReLU activation function layer, and a Sigmoid activation function layer to obtain k scale-aware attention weights; S223. K scale-aware attention weights are used to fuse k prior convolutional kernels to obtain a scale-aware convolutional kernel; S224. Scale-aware convolution kernels are used to process compressed features to obtain dynamic scale features.

4. The method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution according to claim 2, characterized in that, The process of obtaining frequency domain features from convolutional features through a frequency domain adaptive filter includes: S231. Perform Fourier transform decomposition on the convolutional features to obtain the phase and amplitude; S232. The phase is processed by a depthwise separable convolution with a kernel size of 3×3, a ReLU activation function, a pointwise convolution with a kernel size of 1×1, and a Sigmoid activation function to obtain phase mapping weights; the phase is multiplied by the phase mapping weights to obtain weighted phase information; S234. The amplitude is processed by a depthwise separable convolution with a kernel size of 3×3, a ReLU activation function, a point convolution with a kernel size of 1×1, and a Sigmoid activation function to obtain the amplitude mapping weights; the amplitude is multiplied by the amplitude mapping weights to obtain the weighted amplitude information; S235. The frequency domain features are obtained by performing an inverse Fourier transform based on the weighted phase information and weighted amplitude information.

5. The method for arbitrary-scale super-resolution of remote sensing images based on dynamic-scale frequency domain convolution according to claim 1, characterized in that, The processing steps for each scale of the nonlocal attention module include: S311. Input the input features into the query network and value network respectively to obtain query features and value features; input the input features through a 3×3 convolution and then into the key network to obtain key features; S312. After sampling and shaping the key features, the attention weights are obtained by performing operations with the query features; S313. After sampling and shaping the value features, they are fused with the attention weights to obtain the attention features; S314. Concatenate the attention features with the input features to obtain nonlocal features; S315. Interpolate and upsample the non-local features, and then obtain the output features through 3×3 convolution and pixel attention operations.

Images