Remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a remote sensing image arbitrary-scale super-resolution method based on dynamic scale frequency domain convolution, which comprises the following steps: constructing and training an arbitrary-scale super-resolution network model, acquiring an image to be processed, and inputting the image to the trained arbitrary-scale super-resolution network model to obtain a reconstructed image; the arbitrary-scale super-resolution network model comprises a feature extraction module, a dynamic scale frequency domain feature learning module, a cross-scale non-local up-sampling module and an MFGLIF module; according to the method, the capability of recovering missing structures and texture details by a network and the capability of processing discontinuous mapping of coordinates and signal values can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to an arbitrary-scale super-resolution method for remote sensing images based on dynamic-scale frequency-domain convolution. Background Art

[0002] Arbitrary-scale super-resolution of remote sensing images is a key image processing technology that aims to reconstruct high-resolution images of arbitrary magnification from low-resolution images, improve the accuracy of spatial resolution and ground object details, and meet the needs of different application scenarios.

[0003] Among the existing research methods, remote sensing image arbitrary scale super-resolution based on deep learning has achieved remarkable visual effects. Existing methods are usually based on discrete representation or continuous representation implementation methods, dynamically predicting filter weights according to scale factors or constructing implicit neural functions to achieve the mapping between query coordinates and signal values. However, these methods generally use static convolution, predicting filters or using scale factors in implicit neural functions, ignoring the dynamic feature extraction requirements of different scale factors and the inherent characteristics of RSI. In addition, as the scale factor increases, the local features extracted from the input image will have serious problems of missing structure and texture details, and the network often faces challenges such as shape distortion and checkerboard artifacts. Therefore, it is particularly important to explore new strategies to achieve dynamic scale feature extraction to supplement local features.

[0004] Learning continuous image representation can reconstruct high-resolution images of any scale from low-resolution inputs, and realize the mapping of query coordinates and signal values ​​based on single feature space latent codes by using local integration and implicit neural functions. However, local integration uses a limited receptive field to calculate the set weights, which are usually calculated by the rectangular area between the query coordinates and each nearest neighbor coordinate. It ignores the loss of visual information of coordinates and local features and does not fully consider the global correlation feature integration of the image. In addition, due to the strong randomness of RSI texture distribution, predicting signal values ​​using only local latent codes in a single feature space usually leads to discontinuity. Therefore, how to make full use of global information and combine continuous feature representations in multiple feature spaces to enhance the latent code representation to effectively handle the discontinuity of coordinate and signal value mapping has become an urgent problem to be solved. Summary of the invention

[0005] To solve the above problems, the present invention provides a remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution, including constructing and training an arbitrary scale super-resolution network model, obtaining an arbitrary scale super-resolution network model trained by inputting an image to be processed to obtain a reconstructed image; the arbitrary scale super-resolution network model includes a feature extraction module, a dynamic scale frequency domain feature learning module, a cross-scale non-local upsampling module and an MFGLIF module;

[0006] The training process of the arbitrary scale super-resolution network model includes:

[0007] S1. Inputting the LR remote sensing image into a feature extraction module to obtain shallow features; the feature extraction module includes a 3×3 convolution layer and a ReLU activation function layer;

[0008] S2. Inputting shallow features into a dynamic scale frequency domain feature learning module to obtain multi-level features; the multi-level features include first-level features, second-level features, and third-level features;

[0009] S3. Input the multi-level features into the cross-scale non-local upsampling module to obtain multi-scale features; the multi-scale features include first scale features, second scale features and third scale features;

[0010] S4. Input the multi-scale features into the MFGLIF module to obtain the reconstructed high-resolution remote sensing image.

[0011] Beneficial effects of the present invention:

[0012] The present invention realizes dynamic scale feature extraction through a new scale-aware frequency domain block (SAFB), and utilizes the comprehensive advantages of spatial and frequency domain features to improve the network's ability to restore missing structure and texture details; considering that local integration in continuous representation only utilizes a limited receptive field, a cross-scale non-local upsampling module (CSNUB) is introduced to enhance the network's ability to utilize non-local feature information; at the same time, a multi-feature global-local implicit function (MFGLIF) is introduced to effectively enhance the latent code representation and improve the network's ability to handle discontinuous mapping between coordinates and signal values. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 The training process of the arbitrary scale super-resolution network model of the present invention;

[0014] Figure 2 A schematic diagram of the overall structure of a network model of a remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to the present invention;

[0015] Figure 3 It is a schematic diagram of a layer of iterative branch structure in the dynamic scale frequency domain feature learning module of the present invention;

[0016] Figure 4 The dynamic scale-aware convolutional graph of the present invention;

[0017] Figure 5 is a frequency domain adaptive filter diagram of the present invention;

[0018] Figure 6 This is a diagram of the cross-scale non-local upsampling module of the present invention. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The present invention provides a remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution, comprising constructing and training an arbitrary scale super-resolution network model, obtaining an arbitrary scale super-resolution network model trained by inputting an image to be processed to obtain a reconstructed image; the arbitrary scale super-resolution network model comprises a feature extraction module, a dynamic scale frequency domain feature learning module, a cross-scale non-local upsampling module and an MFGLIF module.

[0021] like Figure 1 , Figure 2 As shown in Figure 1, the training process of the arbitrary scale super-resolution network model includes:

[0022] S1. Input the LR remote sensing image into a feature extraction module to obtain shallow features; the feature extraction module includes a 3×3 convolution layer and a ReLU activation function layer.

[0023] Specifically, step S1 can be expressed as

[0024] F0 = ReLU(Conv3(I LR ))

[0025] Among them, F0 represents the shallow feature, I LR represents the LR remote sensing image, Conv3(·) represents the 3×3 convolutional layer, and ReLU(·) represents the ReLU activation function layer.

[0026] S2. Input the shallow features into the dynamic scale frequency domain feature learning module to obtain multi-level features; the multi-level features include first-level features, second-level features and third-level features.

[0027] Specifically, Figure 2 As shown, the dynamic scale frequency domain feature learning module (DSFFLM) includes three layers of iterative branches, the first layer of iterative branches includes the first SAFM unit, the second SAFM unit and the third SAFM unit cascaded in sequence; the second layer of iterative branches includes the fourth SAFM unit, the fifth SAFM unit and the sixth SAFM unit cascaded in sequence; the third layer of iterative branches includes the seventh SAFM unit, the eighth SAFM unit and the ninth SAFM unit cascaded in sequence; the structures of the 9 SAFM units are the same, all including a 1×1 convolution layer and a scale-aware frequency domain block (SAFB);

[0028] In step S2, the shallow features and the scale factor r are input into the dynamic scale frequency domain feature learning module. The specific processing process includes:

[0029] S21. Input shallow features into the first SAFM unit to obtain the first SAFM output Combine shallow features with the first SAFM output After splicing, input the second SAFM unit to obtain the second SAFM output Output shallow features and first SAFM and the second SAFM output After splicing, input the third SAFM unit to obtain the first level feature

[0030] S22. Shallow features and first-level features After splicing, input the fourth SAFM unit to obtain the fourth SAFM output Combine shallow features with the fourth SAFM output After splicing, input the fifth SAFM unit to obtain the fifth SAFM output Output shallow features and the fourth SAFM and the fifth SAFM output After splicing, input the sixth SAFM unit to obtain the second level feature

[0031] S23. Shallow features and second-level features After splicing, input the seventh SAFM unit to obtain the seventh SAFM output Combine shallow features with the seventh SAFM output After splicing, input into the eighth SAFM unit to obtain the eighth SAFM output Output shallow features and the seventh SAFM and the eighth SAFM output After splicing, input the ninth SAFM unit to obtain the third level feature

[0032] All level features are the same size.

[0033] Specifically, Figure 3 As shown in the figure, the scale-aware frequency domain block includes a channel expansion convolution layer, a channel compression convolution layer, a dynamic scale-aware convolution block, an ESA channel attention layer, and a frequency domain adaptive filter; the processing process of each SAFM unit includes:

[0034] S211. The input feature is passed through a 1×1 convolution layer to obtain a convolution feature;

[0035] S212. The convolutional features are passed through a channel expansion convolution layer with a convolution kernel size of 3×3 to obtain expansion features; the expansion features are passed through a channel compression convolution layer with a convolution kernel size of 3×3 to obtain compression features; the compression features are passed through a dynamic scale-aware convolution block to obtain dynamic scale features; the dynamic scale features are passed through an ESA channel attention layer to obtain spatial domain features;

[0036] S213. The convolution feature is passed through a frequency domain adaptive filter to obtain a frequency domain feature;

[0037] S214. Add the convolutional features and the spatial domain features and concatenate them with the frequency domain features to obtain the output features.

[0038] Specifically, the processing of the SAFM unit can be expressed as

[0039] f spa =ESA(DSA(Conv 3,folding (Conv 3,unfolding (Conv1(H i,j )))))

[0040] f fre =FAF(Conv1(H i,j ))

[0041]

[0042] Among them, f spa represents the spatial characteristics, f fre represents the frequency domain characteristics, represents the output features of the j=1, 2, 3 SAFM units of the i=1, 2, 3 iterative branches, i.e., the features of the fusion of spatial and frequency domain information, H i,j represents the input features of the j=1, 2, 3 SAFM units in the iterative branch of the i=1, 2, 3 layers, Conv1(·) represents a 1×1 convolutional layer, Conv 3,unfolding (·) represents the channel expansion convolution layer, Conv 3,folding (·) denotes a channel-squeezed convolutional layer, DSA(·) denotes a dynamic scale-aware convolutional block, ESA(·) denotes an ESA channel attention layer, and FAF(·) denotes a frequency-domain adaptive filter.

[0043] Specifically, Figure 4 As shown in FIG. 1 , the process of obtaining dynamic scale features by the compressed features through the dynamic scale-aware convolution block (DSA-Conv) includes:

[0044] S221. Perform global average pooling on the compressed features to obtain pooled features, and concatenate the pooled features with the scale factor r after passing through the fully connected layer and the ReLU activation function layer to obtain scale concatenated features;

[0045] S222. The scale-concatenated features are sequentially passed through a fully connected layer, a ReLU activation function, a fully connected layer, a ReLU activation function layer, and a Sigmoid activation function layer to obtain k scale-aware attention weights w1, w2, …, w k ;

[0046] S223. Using k scale-aware attention weights to fuse k prior convolution kernels of the same size to obtain a scale-aware convolution kernel;

[0047] S224. Use scale-aware convolution kernel to process compressed features to obtain dynamic scale features.

[0048] Specifically, Figure 5 As shown in FIG. 1 , the process of obtaining frequency domain features through the frequency domain adaptive filter of the convolution features includes:

[0049] S231. Perform Fourier transform decomposition on the convolution feature to obtain phase P and amplitude A;

[0050] S232. The phase P is subjected to a depthwise separable convolution with a convolution kernel size of 3×3, a ReLU activation function, a point convolution with a convolution kernel size of 1×1, and a Sigmoid activation function to obtain a phase mapping weight; the phase is multiplied by the phase mapping weight to obtain weighted phase information;

[0051] S234. The amplitude A is subjected to a depthwise separable convolution with a convolution kernel size of 3×3, a ReLU activation function, a point convolution with a convolution kernel size of 1×1, and a Sigmoid activation function to obtain an amplitude mapping weight; the amplitude is multiplied by the amplitude mapping weight to obtain weighted amplitude information;

[0052] S235. Perform inverse Fourier transform based on the weighted phase information and weighted amplitude information to obtain frequency domain features.

[0053] S3. Input the multi-level features into a cross-scale non-local upsampling module to obtain multi-scale features; the multi-scale features include first-scale features, second-scale features and third-scale features.

[0054] Specifically, the cross-scale non-local upsampling module (CSNUB) includes three-scale non-local attention modules; the level features are input into the non-local attention module to supplement the global features. In step 3, the multi-level features are input into the cross-scale non-local upsampling module, and the multi-scale features obtained include:

[0055] S31. Input the first-level feature into the non-local attention module of the first scale to obtain the first-scale feature;

[0056] S32. Input the second level feature into the non-local attention module of the second scale to obtain the second scale feature;

[0057] S33. Input the third-level features into the non-local attention module of the third scale to obtain the third-scale features.

[0058] Specifically, Figure 6 As shown in Figure 2, the processing of the non-local attention module at each scale includes:

[0059] S311. Input the input features into the query network δ q Sum value network δ v Get the query feature Q and value feature V; downsample the input feature through 3×3 convolution and input it into the key network δ k Get the key feature K;

[0060] S312. After sampling and shaping the key features, they are operated with the query features to obtain the attention weight, which can be expressed as

[0061]

[0062] Among them, W att represents the attention weight, T represents the matrix transpose;

[0063] S313. Sampling and shaping the value feature V and the attention weight W att Fusion obtains attention features;

[0064] S314. Concatenate the attention feature with the input feature to obtain the non-local feature F′ i ;

[0065] S315. For non-local features F′ i Interpolation upsampling is performed, and then the output features are obtained through 3×3 convolution and pixel attention operation, which can be expressed as

[0066]

[0067] Among them, F′ i represents the non-local feature corresponding to the i-th level feature output by the i-th layer iterative branch, Represents r i times upsampling, PA(·) represents pixel attention operation, Conv3(·) represents 3×3 convolutional layer, represents the i-th magnification r i times the scale characteristics.

[0068] S4. Input the multi-scale features into the MFGLIF module (multi-feature global-local implicit function module) to obtain the reconstructed high-resolution remote sensing image.

[0069] Specifically, the local latent code of the multi-feature space is obtained from the multi-scale features through the multi-feature global-local implicit function (MFGLIF). After the enhanced latent code is obtained, the implicit function is used to map the coordinate domain and the signal domain to obtain the reconstructed clear image, which can be expressed as

[0070] s=f(L(x,Z1),L(x,Z2),…,L(x,Z N );r)

[0071] Among them, s represents the predicted RGB value, L(·,Z t ) represents the local implicit neural function, Z i represents the set of four nearest neighbor latent codes of the query coordinate x in the i-th scale feature, usually {z 00 ,z 01 ,z 10 ,z 11} represents the lower left, upper left, lower right and lower right latent codes of the query coordinate x in the i-th scale feature, i.e., Z i ={z 00 ,z 01 ,z 10 ,z 11}, f(·) is a multi-feature global implicit function, N=3.

[0072] Step S4 specifically includes:

[0073] S41. Obtaining local latent codes of the first scale feature, the second scale feature, and the third scale feature respectively through a local implicit neural function;

[0074] S42. Input the scale factor into two fully connected layers and a softmax activation function to obtain scale-aware weights;

[0075] S43. weighted concatenation of the local latent code and the scale-aware weight to obtain an enhanced latent code;

[0076] The process is represented as

[0077] {π1,π2,…,π N}=softmax(FC(FC(r)))

[0078] Z=Concat(π1*L(x,Z1),…π N *L(x,Z N ))

[0079]

[0080] Among them, Z represents the enhanced latent code, π i represents the scale weight of the i-th feature space, (u,v) is the ratio of x to z 00The position offset is normalized to [0,1];

[0081] S44. The enhanced latent code is input into a global implicit neural function to achieve the mapping between the coordinate domain and the signal domain, and obtain the reconstructed high-resolution remote sensing image.

[0082] S5. Calculate the loss training model parameters according to the reconstructed high-resolution remote sensing image until the model parameters converge.

[0083] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0084] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution, characterized in that: Constructing and training an arbitrary scale super-resolution network model, obtaining an arbitrary scale super-resolution network model trained after inputting an image to be processed to obtain a reconstructed image; the arbitrary scale super-resolution network model includes a feature extraction module, a dynamic scale frequency domain feature learning module, a cross-scale non-local upsampling module and an MFGLIF module; The training process of the arbitrary scale super-resolution network model includes: S1. Inputting the LR remote sensing image into a feature extraction module to obtain shallow features; the feature extraction module includes a 3×3 convolution layer and a ReLU activation function layer; S2. Inputting shallow features into a dynamic scale frequency domain feature learning module to obtain multi-level features; the multi-level features include first-level features, second-level features, and third-level features; S3. Input the multi-level features into the cross-scale non-local upsampling module to obtain multi-scale features; the multi-scale features include first scale features, second scale features and third scale features; S4. Input the multi-scale features into the MFGLIF module to obtain the reconstructed high-resolution remote sensing image.

2. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 1 is characterized in that: The dynamic scale frequency domain feature learning module includes three layers of iterative branches, the first layer of iterative branches includes a first SAFM unit, a second SAFM unit and a third SAFM unit cascaded in sequence; the second layer of iterative branches includes a fourth SAFM unit, a fifth SAFM unit and a sixth SAFM unit cascaded in sequence; the third layer of iterative branches includes a seventh SAFM unit, an eighth SAFM unit and a ninth SAFM unit cascaded in sequence; the 9 SAFM units have the same structure, all including a 1×1 convolution layer and a scale-aware frequency domain block; The specific processing process of step S2 includes: S21. Input shallow features into the first SAFM unit to obtain the first SAFM output Combine shallow features with the first SAFM output After splicing, input the second SAFM unit to obtain the second SAFM output Output shallow features and the first SAFM and the second SAFM output After splicing, input the third SAFM unit to obtain the first level feature S22. Shallow features and first-level features After splicing, input the fourth SAFM unit to obtain the fourth SAFM output Combine shallow features with the fourth SAFM output After splicing, input the fifth SAFM unit to obtain the fifth SAFM output Output shallow features and the fourth SAFM and the fifth SAFM output After splicing, input the sixth SAFM unit to obtain the second level feature S23. Shallow features and second-level features After splicing, input the seventh SAFM unit to obtain the seventh SAFM output Combine shallow features with the seventh SAFM output After splicing, input into the eighth SAFM unit to obtain the eighth SAFM output Output shallow features and the seventh SAFM and the eighth SAFM output After splicing, input the ninth SAFM unit to obtain the third level feature 3. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 2 is characterized in that: The scale-aware frequency domain block includes a channel dilation convolution layer, a channel compression convolution layer, a dynamic scale-aware convolution block, an ESA channel attention layer, and a frequency domain adaptive filter; The processing of each SAFM unit includes: S211. The input feature is passed through a 1×1 convolution layer to obtain a convolution feature; S212. The convolutional features are passed through a channel expansion convolution layer with a convolution kernel size of 3×3 to obtain expansion features; the expansion features are passed through a channel compression convolution layer with a convolution kernel size of 3×3 to obtain compression features; the compression features are passed through a dynamic scale-aware convolution block to obtain dynamic scale features; the dynamic scale features are passed through an ESA channel attention layer to obtain spatial domain features; S213. The convolution feature is passed through a frequency domain adaptive filter to obtain a frequency domain feature; S214. Add the convolutional features and the spatial domain features and concatenate them with the frequency domain features to obtain the output features.

4. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 3 is characterized in that: The process of obtaining dynamic scale features by compressed features through the dynamic scale-aware convolution block includes: S221. Perform global average pooling on the compressed features to obtain pooled features, and concatenate the pooled features with the scale factor after passing through the fully connected layer and the ReLU activation function layer to obtain scale concatenated features; S222. The scale-concatenated features are sequentially passed through a fully connected layer, a ReLU activation function layer, a fully connected layer, a ReLU activation function layer, and a Sigmoid activation function layer to obtain k scale-aware attention weights; S223. Using k scale-aware attention weights to fuse k prior convolution kernels to obtain a scale-aware convolution kernel; S224. Use scale-aware convolution kernel to process compressed features to obtain dynamic scale features.

5. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 3 is characterized in that: The process of obtaining frequency domain features through frequency domain adaptive filters through convolution features includes: S231. Perform Fourier transform decomposition on the convolution feature to obtain phase and amplitude; S232. The phase is subjected to a depthwise separable convolution with a convolution kernel size of 3×3, a ReLU activation function, a point convolution with a convolution kernel size of 1×1, and a Sigmoid activation function to obtain a phase mapping weight; the phase is multiplied by the phase mapping weight to obtain weighted phase information; S234. The amplitude is subjected to a depthwise separable convolution with a convolution kernel size of 3×3, a ReLU activation function, a point convolution with a convolution kernel size of 1×1, and a Sigmoid activation function to obtain an amplitude mapping weight; the amplitude is multiplied by the amplitude mapping weight to obtain weighted amplitude information; S235. Perform inverse Fourier transform based on the weighted phase information and weighted amplitude information to obtain frequency domain features.

6. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 1 is characterized in that: The cross-scale non-local upsampling module includes three-scale non-local attention modules; Step 3: Input the multi-level features into the cross-scale non-local upsampling module to obtain the multi-scale features including: S31. Input the first-level feature into the non-local attention module of the first scale to obtain the first-scale feature; S32. Input the second level feature into the non-local attention module of the second scale to obtain the second scale feature; S33. Input the third-level features into the non-local attention module of the third scale to obtain the third-scale features.

7. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 1 is characterized in that: The processing of the non-local attention module at each scale includes: S311. Input the input features into the query network and the value network respectively to obtain query features and value features; input the input features into the key network after 3×3 convolution to obtain key features; S312. After sampling and shaping the key feature, perform calculation on the query feature to obtain the attention weight; S313. After sampling and shaping the value feature, it is fused with the attention weight to obtain the attention feature; S314. Concatenate the attention feature with the input feature to obtain a non-local feature; S315. Perform interpolation upsampling on the non-local features, and then obtain the output features through 3×3 convolution and pixel attention operations.

8. The remote sensing image arbitrary scale super-resolution method based on dynamic scale frequency domain convolution according to claim 1, characterized in that: Step S4 specifically includes: S41. Obtaining local latent codes of the first scale feature, the second scale feature, and the third scale feature respectively through a local implicit neural function; S42. Input the scale factor into two fully connected layers and a softmax activation function to obtain scale-aware weights; S43. weighted concatenation of the local latent code and the scale-aware weight to obtain an enhanced latent code; S44. The enhanced latent code is input into a global implicit neural function to obtain a reconstructed high-resolution remote sensing image.

Images