Layered attention enhancement remote sensing image super-resolution system based on state space model

Abstract

The invention discloses a hierarchical attention enhancement remote sensing image super-resolution system based on a state space model, and relates to the field of computer vision and remote sensing image processing. According to the system, a low-resolution remote sensing image, a primary feature extraction module, a multistage state space model feature extraction module, a first adder module, an image reconstruction module and a high-resolution remote sensing image interact in sequence; the primary feature extraction module and the first adder module interact front and back; and the multi-level state space model feature extraction module, the hierarchical feature attention enhancement module and the first adder module interact in sequence. According to the method, the problem that shallow-layer high-frequency information is forgotten during multi-layer feature extraction in a traditional super-resolution method can be relieved, and then hierarchical features are enhanced; and the limited global-local consistency feature learning capability of the state space model is compensated, and competitive calculation complexity is achieved while the super-resolution performance of the remote sensing image is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and remote sensing image processing, and in particular to a remote sensing image super-resolution system based on a state-space model hierarchical attention enhancement; that is, a system and method based on a deep learning state-space model that utilizes hierarchical attention enhancement and spatial-frequency fusion. Background Art

[0002] Agriculture, meteorology, geography, military and other fields have benefited from remote sensing imaging technology. Since high-resolution remote sensing images can record high-quality earth observation details in detail, they are widely used in various earth observation tasks, such as land cover classification, object detection and change detection. Therefore, in order to realize the application and efficient analysis of remote sensing images, it is crucial to obtain high-resolution remote sensing images. [See references: [1] J. Li et al., "ConvFormerSR: Fusing Transformers and Convolutional Neural Networks for Cross-Sensor Remote Sensing Imagery Super-Resolution," in IEEE Transactions on Geoscience and Remote Sensing, vol. 62, pp. 1-15, 2024; [2] Y. Xiao, Q. Yuan, K. Jiang, J. He, C.-W. Lin and L. Zhang, "TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution," in IEEE Transactions on Image Processing, vol. 33, pp. 738-752, 2024].

[0003] However, factors such as sensor noise, optical distortion, and environmental interference still have an adverse effect on the quality of remote sensing images, making it difficult to obtain high-resolution remote sensing images. Compared with hardware improvements, which will increase costs and be time-consuming, super-resolution technology (SR) provides a more flexible and economical solution to the problem of restoring high-quality images from low-quality images. Traditional methods include interpolation-based methods, reconstruction-based methods, and learning-based prior methods. These methods can produce certain effects, but it is difficult to produce accurate results and involves laborious optimization processes. Therefore, the currently popular super-resolution is based on deep learning, which learns the mapping between LR and HR image spaces to more efficiently and time-savingly predict the high-frequency information missing in low-resolution images.

[0004] However, most current super-resolution methods based on state-space models are applied to the field of natural images. Compared with natural images, remote sensing images are different in that the scenes of remote sensing images are complex and diverse, and the features of ground objects have significant scale variations and directional differences. Existing super-resolution restoration methods for remote sensing images based on state-space models face the following challenges: ① Insufficient utilization of hierarchical information: The layer-by-layer feature extraction method of traditional SR models will lead to the forgetting of shallow high-frequency information, affecting the ability to reconstruct details. ② Insufficient extraction of local-global correlation features: The sequential scanning method of state-space models in visual tasks will inhibit the modeling ability of non-causal image structures and affect the learning of global information.

[0005] Therefore, there is an urgent need to design a super-resolution restoration method with higher fine-grained accuracy to greatly improve the quality of satellite images. Summary of the Invention

[0006] The present invention aims to address the existing technical issues outlined above by providing a remote sensing image super-resolution system based on hierarchical attention enhancement using a state-space model. This system effectively improves the image quality of high-magnification low-resolution remote sensing images by combining hierarchical attention enhancement with spatial-frequency domain fusion and employing an improved state-space model architecture.

[0007] The object of the present invention is achieved like this:

[0008] The present invention solves the deficiencies of the prior art through the following technical solutions:

[0009] Multi-level state-space model feature extraction module: Through multi-level state-space model feature extraction, the state-space model feature extraction module stacks a space-frequency state-space model block consisting of a normalization layer, a residual connection with a learnable factor, an adder, a visual state-space model module, and a space-frequency information fusion module. By utilizing the long-distance modeling capability of the state-space model that only requires linear complexity, it improves super-resolution performance without excessively increasing computational complexity, thereby alleviating the lack of local-global information extraction.

[0010] Hierarchical attention enhancement module: Through the adaptive aggregation of multi-level features, it alleviates the forgetting problem of shallow high-frequency features and enhances hierarchical feature information.

[0011] The technical idea is:

[0012] First, a primary feature extraction module is used to transform the input low-resolution remote sensing image, mapping it from a low-dimensional space to a higher-dimensional space to obtain primary features. Then, a state-space model feature extraction module is used to extract multi-level state-space model features. Then, a hierarchical feature attention enhancement module is used to adaptively aggregate high-frequency features lost during deep feature extraction. Then, the multi-level state-space model extracted features, primary features, and hierarchical attention enhancement features are added together through a first adder module to obtain deep features. Finally, an image reconstruction module is used to convolve and upsample the deep features to obtain a high-resolution image result. The present invention effectively improves the feature extraction capability of the remote sensing image super-resolution network through step-by-step state-space model feature extraction and hierarchical attention enhancement, thereby improving the super-resolution restoration quality of low-resolution remote sensing images.

[0013] The specific plan is:

[0014] 1. State-space model-based hierarchical attention-enhanced remote sensing image super-resolution system (referred to as the system)

[0015] This system includes a primary feature extraction module, a multi-level state space model feature extraction module, a hierarchical feature attention enhancement module, a first adder module, and an image reconstruction module;

[0016] The low-resolution remote sensing image, the primary feature extraction module, the multi-level state space model feature extraction module, the first adder module, the image reconstruction module and the high-resolution remote sensing image interact in sequence;

[0017] The primary feature extraction module and the first adder module interact with each other;

[0018] The multi-level state space model feature extraction module, the hierarchical feature attention enhancement module and the first adder module interact in sequence.

[0019] 2. State-space model-based hierarchical attention-enhanced remote sensing image super-resolution method (referred to as method)

[0020] This method comprises the following steps:

[0021] ① Primary feature extraction

[0022] The primary feature extraction module receives the input low-resolution remote sensing image and uses a single-layer convolutional neural network to generate the primary feature representation F0 of the low-resolution remote sensing image;

[0023] ② Multi-level state space model feature extraction

[0024] A. 1st state space model feature extraction

[0025] By stacking multiple space-frequency state-space model blocks, the received primary feature F0 is sequentially passed through space-frequency state-space model blocks 1, 2...L, and then passed through the second adder module to obtain the first-level state-space model extracted feature F1;

[0026] B. Second state space model feature extraction

[0027] Using multiple stacked space-frequency state-space model blocks and adder modules with the same structure as in the first state-space model feature extraction to generate the second-level state-space model extraction feature F2;

[0028] C. Feature extraction of the third state space model

[0029] Using multiple stacked space-frequency state-space model blocks and adder modules with the same structure as in the second state-space model feature extraction to generate the third-level state-space model extraction feature F3;

[0030] D. Follow this procedure and extract the feature F from the final N-1 level state space model. N-1 After the N-th level state space model feature extraction, the N-th level state space model extraction feature F is generated N ;

[0031] ③ Hierarchical feature attention enhancement

[0032] By receiving the state space model feature extraction module at each level, the state space model extracts features F1, F2...F N , first channel-level splicing and then adaptively selecting the level information that needs to be enhanced F E ;

[0033] ④Adder

[0034] By extracting the primary feature F0 and the multi-level state space model feature F N and hierarchical attention-enhanced features FE Add together to get the deep feature F D ;

[0035] ⑤High-resolution remote sensing image generation

[0036] The image reconstruction module receives the deep features F D ,Two layers of convolution operations and one layer of upsampling operations are used to generate high-quality ,remote sensing images.

[0037] Working principle:

[0038] The implementation of the present invention is generally divided into three stages:

[0039] In the first stage, the primary feature extraction module is used to extract primary features from the known low-resolution remote sensing images;

[0040] In the second stage, a multi-level state space model is used for feature extraction, hierarchical feature attention enhancement, and adder operation to perform multi-level learning and hierarchical information enhancement on primary features to obtain deep features.

[0041] In the third stage, the image reconstruction module is used to obtain high-resolution remote sensing image restoration results through two layers of convolution operations and one layer of upsampling operations.

[0042] Compared with the prior art, the present invention has the following advantages and positive effects:

[0043] The present invention can alleviate the problem of forgetting shallow high-frequency information when extracting features from multiple layers in traditional super-resolution methods, thereby enhancing hierarchical features; and make up for the limited global-local consistency feature learning ability of the state-space model, while improving the super-resolution performance of remote sensing images and having competitive computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the structural block diagram of this system. Figure 1 middle:

[0045] 10—Initial feature extraction module (IFE);

[0046] 20—Multi-level State Space Model Feature Extraction Module (SSMFE),

[0047] 21—1st state space model feature extraction (SSMFE1),

[0048] 22—Second State Space Model Feature Extraction (SSMFE2),

[0049] …

[0050] 2N-Nth State Space Model Feature Extraction (SSMFE N ),

[0051] N is set to 6;

[0052] 30-Hierarchical Feature Attention Enhancement Module (HFAE);

[0053] 40—1st adder module (Add1);

[0054] 50—Image reconstruction module (IR);

[0055] A—low-resolution remote sensing image;

[0056] B—High-resolution remote sensing imagery.

[0057] Figure 2 This is the structural block diagram of the state space model feature extraction module (SSMFE) of this system. Figure 2 middle:

[0058] 201 — 1st space-frequency state-space model block (SFSSMB1),

[0059] 202 — Second Space-Frequency State-Space Model Block (SFSSMB2),

[0060] 203 — 3rd Space-Frequency State-Space Model Block (SFSSMB3),

[0061] …

[0062] 20L—Lth Space-Frequency State-Space Model Block (SFSSMB L ),

[0063] 211—the second adder module (Add2),

[0064] L is set to 6.

[0065] Figure 3 This is the structural block diagram of the space-frequency state-space model block (SFSSMB) of this system. Figure 3 middle:

[0066] 2001 — 1st normalization layer (Norm1),

[0067] 2002—Visual State Space Model (VSSM),

[0068] 2003 — residual connections with learnable factor γ1,

[0069] 2004—3rd adder module (Add3),

[0070] 2005 — 2nd normalization layer (Norm2),

[0071] 2006—Spatial Frequency Information Interaction Module (SFIIM),

[0072] 2007—Residual connections with learnable factor γ2,

[0073] 2008—4th adder module (Add4).

[0074] Figure 4 This is the structural block diagram of the hierarchical feature attention enhancement module (HFAE) of this system. Figure 4 middle:

[0075] 301—State space model characteristics F1, F2...F at each level N ,

[0076] 302—channel splicing,

[0077] 303—1st data reorganization,

[0078] 304—Second data reorganization,

[0079] 305—1st multiplier,

[0080] 306—normalization function,

[0081] 307—Second multiplier,

[0082] 308—3rd data reorganization,

[0083] 309—5th adder.

[0084] 310—4th data reorganization. DETAILED DESCRIPTION

[0085] The following is a detailed description with reference to the accompanying drawings and embodiments:

[0086] 1. System

[0087] 1. Overall

[0088] like Figure 1 ,This system includes a primary feature extraction module 10, a multi-level state space model feature extraction module 20, a hierarchical feature attention enhancement module 30, a first adder module 40, and an image reconstruction module 50;

[0089] The interactive relationship is:

[0090] The low-resolution remote sensing image A, the primary feature extraction module 10, the multi-level state space model feature extraction module 20, the first adder module 40, the image reconstruction module 50 and the high-resolution remote sensing image B interact in sequence;

[0091] The primary feature extraction module 10 and the first adder module 40 interact with each other;

[0092] The multi-level state space model feature extraction module 20, the hierarchical feature attention enhancement module 30 and the first adder module 40 interact in sequence.

[0093] In detail: the primary feature extraction module 10 has an input terminal and an output terminal, the multi-level state space model feature extraction module 20 has an input terminal and N output terminals, the hierarchical feature attention enhancement module 30 has N input terminals and one output terminal, the first adder module 40 has three input terminals and one output terminal, and the image reconstruction module 50 has an input terminal and an output terminal; the input terminal of the primary feature extraction module 10 is the input terminal of the system - the low-resolution remote sensing image A, and the output terminal is the input terminal of the multi-level state space model feature extraction module 20, and the input terminal of the hierarchical feature attention enhancement module 30 is the state space model extraction features F1, F2...F of each level in the multi-level state space model feature extraction module 20. N , the output is the hierarchical enhanced feature F E The input end of the first adder module 40 is the primary feature F0, the Nth level state space model extraction feature F N and hierarchical enhancement features F E , the output end is the input end of the image reconstruction module 50, and the output end of the image reconstruction module is the high-resolution remote sensing image B.

[0094] 2. Functional modules

[0095] 1) Primary feature extraction module 10

[0096] like Figure 1 , the primary feature extraction module 10 performs convolution operation on the input low-resolution remote sensing image A to generate the primary feature F0.

[0097] Its workflow is: scan each element of the input data block from left to right and from top to bottom until the end, receive one element of the input data block each time and the elements within the domain range specified by the convolution definition in sequence, and perform convolution and operation with multiple groups of convolution kernels of the same size contained in the convolution layer unit, and the output results are arranged in the same order.

[0098] The relationship between the primary feature extraction module 10 and other modules:

[0099] The primary feature extraction module 10 receives the input low-resolution remote sensing image A, obtains the primary representation F0 of the low-resolution remote sensing image through convolution operation, and sends it to the multi-level state space model feature extraction module 20 and the first adder module 40.

[0100] 2) Multi-level state space model feature extraction module 20

[0101] like Figure 1The multi-level state space model feature extraction module 20 includes the first, second...N state space model feature extraction modules 21, 22...2N that interact in sequence, where N is set to 6, and obtains the state space model features F1, F2...F at each level in sequence. N ;

[0102] like Figure 2 Each state-space model feature extraction module includes the first, second, ..., L space-frequency state-space model blocks 201, 202, ..., 20L that interact in sequence, and a second adder 211, where L is set to 6; if feature F0 is input, the feature F after each space-frequency state-space model block is obtained. 01 、F 02 ...F 0L , and then after the adder operation, the first-level state space model extraction feature F1 is finally output;

[0103] like Figure 3 Each space-frequency state space model block includes a first normalization layer 2001, a visual state space model 2002, a third adder module 2004, a second normalization layer 2005, a space-frequency information interaction module 2006 and a fourth adder module 2008 that interact in sequence;

[0104] The residual connection 2003 with the learnable factor γ1 and the third adder module 2004 interact with each other;

[0105] The residual connection 2007 with the learnable factor γ2 and the fourth adder module 2008 interact with each other;

[0106] The first normalization layer 2001 and the residual connection 2003 with the learnable factor γ1 receive the same input end, the output end of the first normalization layer 2001 is the input end of the visual state space model 2002, the output end of the visual state space model 2002 and the output end of the residual connection 2003 with the learnable factor γ1 are the input end of the third adder 2004, the output end of the third adder 2004 is the input end of the second normalization layer 2005 and the residual connection 2007 with the learnable factor γ2, the output end of the second normalization layer 2005 is the input end of the spatial-frequency information interaction module 2006, the output end of the spatial-frequency information interaction module 2006 and the output end of the residual connection 2007 with the learnable factor γ2 are the input end of the fourth adder module 2008, and the output end of the fourth adder is the final output.

[0107] (1) The workflow of the space-frequency state-space model block is:

[0108] Input feature F0, first through the normalization operation 2001 and the visual state space module 2002 to obtain the feature F learned by the state space model M, then the input feature F0 is connected to the feature F through the learnable factor γ1 residual 2003 M The two are input to the third adder 2004 to obtain the intermediate feature F IM Then, the space-frequency feature F is obtained through the normalization operation 2005 and the space-frequency information interaction module 2006. SF , and then the intermediate feature F IM Through the learnable factor γ2 residual connection 2007 and the space-frequency feature F SF After the fourth adder module 2008, the final output is F 01 .

[0109] (2) The workflow of the visual state space model is:

[0110] The visual state space model includes two branches, one of which consists of a linear layer, a convolution operation, an activation function, a selective scan, and a layer normalization; the other branch consists of a linear layer, an activation function, and finally the two branches are multiplied and passed through a linear layer again to obtain the visual state space model learning feature F M .

[0111] (3) The workflow of the spatial frequency information interaction module is:

[0112] The spatial-frequency information interaction module includes two-branch processing in two domains, namely the spatial domain and the frequency domain. Specifically, the spatial domain consists of two sub-branches: the convolution sub-branch and the linear sub-branch extract spatial features and then perform element-by-element multiplication; the frequency domain includes the interactive fast Fourier transform, convolution operation, channel attention, residual connection and inverse fast Fourier transform. Finally, the features of the two domains are spliced through the channel and then subjected to a convolution operation to obtain the space-frequency feature F. SF .

[0113] 3) Hierarchical feature attention enhancement module 30

[0114] like Figure 4 The hierarchical feature attention enhancement module 30 includes a state space model extraction feature set 301 of each level that interacts sequentially, a channel splicing 302, a first data reorganization 303 and a second data reorganization 304, a first multiplier 305, a normalization function 306, a second multiplier 307, a third data reorganization 308, a fifth adder 309 and a fourth data reorganization 310;

[0115] The channel splicing 302 and the fifth adder 309 interact with each other;

[0116] The second data reassembly 304 and the second multiplier 307 are operated in a sequential manner;

[0117] Its workflow is:

[0118] The hierarchical feature attention enhancement module 30 receives the features F1, F2, ... F extracted from the state space model at each level. N 301, first through channel splicing 302 to obtain feature X, then through the first data reorganization 303 and the second data reorganization 304, respectively generate matrices M1 and M2; after the matrices M1 and M2 pass through the first multiplier 305, the similarity matrix S is obtained, and the similarity matrix S is normalized 306 to obtain the attention matrix A, and then it and the matrix M2 pass through the second multiplier 307 and the third data reorganization 308 to obtain the feature F M ; Then feature F M The feature X first passes through the fifth adder 309 and then passes through the fourth data reorganization 310 to obtain the final output F E , generate some high-frequency information lost in the layer-by-layer feature extraction process, and realize hierarchical feature enhancement;

[0119] The first and second data reorganizations are to arrange the input four-dimensional feature map into a two-dimensional matrix and flatten the spatial dimension; the third data reorganization rearranges the processed two-dimensional matrix into a four-dimensional feature map to facilitate addition operation with the original four-dimensional feature; the fourth data reorganization arranges the four-dimensional feature map into a three-dimensional feature map to facilitate processing by the subsequent adder module.

[0120] 4) First adder module 40

[0121] The first adder 40 adds the primary extracted feature F0 and the Nth level state space model extracted feature F N and hierarchical attention-enhanced features F E Add element by element to get the deep feature F D .

[0122] 5) Image reconstruction module 50

[0123] The image reconstruction module 50 receives the deep features F D Two layers of convolution operations and one layer of upsampling operations are used to generate high-resolution remote sensing image output.

[0124] 2. Methods:

[0125] 1. Step 1

[0126] The primary feature extraction module 10 performs convolution operation on the input low-resolution remote sensing image A to generate the primary feature F0, which is expressed as follows:

[0127] F0=Conv(A),

[0128] Among them, Conv represents a 3×3 convolution operation.

[0129] 2. Step ②

[0130] The N-level state space model feature extraction module 20 includes an N-level state space model feature extraction module, which receives the output of the previous level state space model feature extraction module, and generates new features after the state space model feature extraction module again, until the output feature F of the final N-level state space model feature extraction module is N , the above process is expressed by mathematical formula:

[0131] F N =H SSMFE1 (……H SSMFE2 (H SSMFEN (F0))),

[0132] Among them, H SSMFE1 represents the first-level state space model feature extraction, H SSMFE2 represents the second-level state space model feature extraction,

[0133] H SSMFEN Represents the feature extraction of the N-th level state space model;

[0134] The state-space model feature extraction module includes L-level space-frequency state-space model blocks 201, 202, ..., 20L, and a second adder module 211. L is set to 6. If the input feature F0 is expressed as follows:

[0135] F 0L =H SFMBL (……H SFMB2 (H SFMB1 (F0))),

[0136] F1=F 0L +F0,

[0137] Among them, F 0L is the feature after passing through the L-level space-frequency state-space model block, and F1 is the output feature of the first-level state-space model feature extraction module.

[0138] The space-frequency state space model block includes a first normalization layer 2001, a visual state space model 2002, a third adder module 2004, a second normalization layer 2005, a space-frequency information interaction module 2006 and a fourth adder module 2008, which interact in sequence;

[0139] The residual connection 2003 with the learnable factor γ1 and the third adder module 2004 interact with each other;

[0140] The residual connection 2007 with the learnable factor γ2 and the 4th adder module 2008 interact with each other.

[0141] If the feature F0 is input, the whole process can be expressed as follows:

[0142] F IM =H VSSM (LN(F0))+γ1×F0,

[0143] Among them, LN represents the layer normalization operation, H VSSM represents the visual state space module and γ1 is the learnable factor.

[0144] F 01 =H SFIIM (LN(F IM ))+γ2×F IM ,

[0145] Among them, H SFIIM represents the spatial frequency information interaction module, and γ2 is a learnable factor.

[0146] The spatial frequency information interaction module 2006 includes spatial domain and frequency domain processing.

[0147] Specifically, the spatial domain consists of two sub-branches. The convolution sub-branch and the linear sub-branch extract channel features respectively and then perform element-by-element multiplication, which can be expressed as:

[0148]

[0149] Among them, F S is the spatial domain feature obtained after spatial domain operation, F Conv Represents the convolution feature obtained after the convolution sub-branch, which consists of two 1×1 convolutions and one 3×3 convolution. Linear represents the linear features obtained after the linear sub-branch, Represents element-wise multiplication.

[0150] The frequency domain includes fast Fourier transform, channel attention, residual connection and inverse fast Fourier transform, and the operation is expressed as:

[0151] F F =IFFT(H CA (FFT(Y) Amp )+FFT(Y) Amp , H CA (FFT(Y) Pha )+FFT(Y) Pha ),

[0152] Among them, Y represents the input feature, F F is the frequency domain feature obtained after frequency domain operation, FFT and IFFT represent fast Fourier transform and inverse fast Fourier transform respectively, H CA represents the channel attention operation, FFT(Y) AmpRepresents the amplitude component after Fourier transform, FFT(Y) Pha Represents the phase component after Fourier transform. Channel attention mainly consists of convolution operation, maximum pooling operation and element-by-element multiplication.

[0153] Finally, the features of the two domains are concatenated through channels and then passed through two convolutional layers to obtain the space-frequency feature F SF , expressed as:

[0154] F SF =DWConv(Conv(Concat(F S ,F F ))),

[0155] Among them, Conv represents a 1×1 convolution operation, DWConv represents a 3×3 depth-separable convolution operation, and Concat represents a channel concatenation operation.

[0156] 3. Step 3

[0157] The hierarchical feature attention enhancement module receives the N-level state space model to extract features F1, F2...F N After that, it is first spliced into feature X through the channel, and then undergoes two data reorganization operations to generate matrices M1 and M2 respectively, which can be expressed as follows:

[0158] X=Concat(F1、F2……F N )

[0159] M1=Reshape(X),

[0160] M2=Reshape(X)=M1 T ,

[0161] Among them, Reshape represents the data reorganization operation, which changes the feature dimension distribution, and T represents the matrix transpose operation.

[0162] Next, the matrices M1 and M2 are multiplied to obtain the similarity matrix S. The maximum value of each row in the similarity matrix S is found and expanded to the size of the original similarity matrix to obtain the maximum value matrix M. max , then the maximum value matrix M max Subtracting the similarity matrix S element by element to obtain the dissimilarity measure matrix D, which is expressed by the mathematical formula:

[0163]

[0164] D=M max –S,

[0165] in, Represents element-wise multiplication.

[0166] Next, the matrix D is normalized to obtain the attention matrix A, which is then multiplied by the matrix M2 and reorganized to obtain the feature F. M ; Then feature F M The weight of the learnable factor δ is added to the feature X, and then the final output F is obtained through data reorganization operation. E , which is the enhanced hierarchical feature information, the mathematical formula is expressed as:

[0167] A=Softmax(D),

[0168]

[0169] F E =Reshape(δF M +X),

[0170] Among them, Softmax represents the normalized exponential function, It is element-by-element multiplication, and Conv is a 3×3 convolution operation.

[0171] 4. Step 4

[0172] The first adder 40 adds the output feature F of the multi-level state space model feature extraction module N , the output features of the hierarchical feature attention enhancement module E Add it to the primary feature F0 to get the deep feature F D , expressed as a mathematical formula:

[0173] F D =F N +F E +F0.

[0174] 5. Step 5

[0175] Image reconstruction 50 receives deep features F D Finally, two layers of convolution operations are used, and one layer of pixel rearrangement and upsampling are performed to generate a high-resolution remote sensing image output, which can be expressed as follows:

[0176] B=Conv(PixelShuffle(Conv(F D ))),

[0177] Among them, Conv is a 3×3 convolution operation, and PixelShuffle is pixel reordering upsampling.

[0178] 3. Experimental Evaluation

[0179] To demonstrate the effectiveness of the embodiments of the present invention, the inventors conducted an experimental evaluation using the PyTorch deep learning framework. The experimental platform was configured with an Nvidia GeForce GTX 3090 graphics card. N was set to 6, and L was set to 6. Data augmentation techniques (including horizontal flipping and random rotations of 90°, 180°, and 270°) were used to enhance the generalization capability of the model. The performance of the test system was measured using the PSNR and SSIM metrics. A total of 500,000 iterations were performed during training; the Adam method was used for training, with the relevant parameters set to β1 = 0.9 and β2 = 0.999; the initial learning rate was 0.0002; and the learning rate was reduced to half at the 250,000th, 400,000th, 450,000th, and 475,000th iterations, respectively. The loss function for network training was L1 loss.

[0180] The training data set of the present invention is the AID data set, which is cropped to a size of 600×600 pixels; the test data sets are the DIOR, DOTA and UCMerced data sets, the DIOR data set is cropped to a size of 512×512 pixels, the DOTA data set is cropped to a size of 512×512 pixels, and the UCMerced data set is cropped to a size of 256×256 pixels.

[0181] The results of the comparison of the super-resolution performance of remote sensing images of the present method and similar methods are shown in Table 1. As can be seen from the results in Table 1, the method of the present invention has a significant effect in improving the super-resolution quality of remote sensing images.

[0182] Table 1

[0183]

[0184]

[0185] Those skilled in the art may make various modifications and variations to the embodiments of the present invention. If such modifications and variations are within the scope of the claims and equivalents, such modifications and variations are also within the scope of protection of the present invention. Any content not described in detail in this specification is prior art known to those skilled in the art.

Claims

1. A hierarchical attention-enhanced remote sensing image super-resolution system based on a state-space model, characterized by: It includes a primary feature extraction module (10), a multi-level state space model feature extraction module (20), a hierarchical feature attention enhancement module (30), a first adder module (40) and an image reconstruction module (50); The low-resolution remote sensing image (A), the primary feature extraction module (10), the multi-level state space model feature extraction module (20), the first adder module (40), the image reconstruction module (50) and the high-resolution remote sensing image (B) interact in sequence; The primary feature extraction module (10) and the first adder module (40) interact with each other; The multi-level state space model feature extraction module (20), the hierarchical feature attention enhancement module (30) and the first adder module (40) interact with each other in sequence.

2. The state-space model-based hierarchical attention-enhanced remote sensing image super-resolution system according to claim 1, characterized in that: The primary feature extraction module (10) performs a convolution operation on the input low-resolution remote sensing image (A) to generate a primary feature F0; Its workflow is: Scan each element of the input data block from left to right and from top to bottom until the end, and receive one element of the input data block each time and the elements within the domain range specified by the convolution definition in sequence, and perform convolution and operation with multiple groups of convolution kernels of the same size contained in the convolution layer unit, and the output results are arranged in the same order.

3. The state-space model-based hierarchical attention-enhanced remote sensing image super-resolution system according to claim 1, characterized in that: The multi-level state space model feature extraction module (20) includes the first, second, ... N state space model feature extraction modules (21, 22 ... 2N) that interact in sequence, N is set to 6, and the state space model features F1, F2 ... F at each level are obtained in sequence. N ; The state space model feature extraction module comprises the first, second, ... L space-frequency state space model blocks (201, 202 ... 20L) interacting in sequence and the second adder module (211). The module input feature interacts with the second adder module (211) in sequence, L is set to 6, and finally the space-frequency state space model extraction feature F is obtained in sequence. 01 、F 02 ...F 0L , F1; Each space-frequency state space model block includes a first normalization layer (2001), a visual state space model (2002), a third adder module (2004), a second normalization layer (2005), a space-frequency information interaction module (2006) and a fourth adder module (2008) that interact in sequence; The residual connection (2003) with a learnable factor γ1 and the third adder module (2004) interact with each other; The residual connection with a learnable factor γ2 (2007) and the 4th adder module (2008) interact forward and backward; The first normalization layer (2001) and the residual connection (2003) with a learnable factor γ1 receive the same input end, the output end of the first normalization layer (2001) is the input end of the visual state space model (2002), the output end of the visual state space model (2002) and the output end of the residual connection (2003) with a learnable factor γ1 are the input end of the third adder (2004), and the output end of the third adder module (2004) is The second normalization layer (2005) and the residual connection (2007) with the learnable factor γ2 are input terminals, the output terminal of the second normalization layer (2005) is the input terminal of the spatial frequency information interaction module (2006), the output terminal of the spatial frequency information interaction module (2006) and the output terminal of the residual connection (2007) with the learnable factor γ2 are input terminals of the fourth adder module (2008), and the output terminal of the fourth adder is the final output; The visual state space model (2002) includes two branches: one branch consists of a linear layer, a convolution operation, an activation function, a selective scan, and a normalization layer; the other branch consists of a linear layer, an activation function, and finally the two branches are multiplied and passed through a linear layer again to obtain the visual state space model learning feature F. M . The spatial-frequency information interaction module (2006) includes two-branch two-domain processing, namely the spatial domain and the frequency domain; the spatial domain consists of two sub-branches, the convolution sub-branch and the linear sub-branch respectively extract spatial features and then perform element-by-element multiplication; the frequency domain includes the interactive fast Fourier transform, convolution operation, channel attention, residual connection and inverse fast Fourier transform in sequence; finally, the features of the two domains are spliced through the channel and then subjected to a convolution operation to obtain the space-frequency feature F SF .

4. The state-space model-based hierarchical attention-enhanced remote sensing image super-resolution system according to claim 1, characterized in that: The hierarchical feature attention enhancement module (30) includes sequentially interacting state space model extraction feature sets (301), channel splicing (302), first data reorganization (303) and second data reorganization (304), a first multiplier (305), a normalization function (306), a second multiplier (307), a third data reorganization (308), a fifth adder (309) and a fourth data reorganization (310); The channel splicing (302) and the fifth adder (309) interact with each other; The second data reorganization (304) and the second multiplier (307) interact with each other; Its workflow is: The hierarchical feature attention enhancement module (30) receives the state space model extraction features F1, F2...F at each level N (301), first through channel splicing (302) to obtain feature X, then through the first data reorganization (303) and the second data reorganization (304), respectively generate matrices M1 and M2; after the matrices M1 and M2 pass through the first multiplier (305), the similarity matrix S is obtained, and the similarity matrix S is normalized (306) to obtain the attention matrix A, and then the matrix M2 passes through the second multiplier (307) and the third data reorganization (308) in sequence to obtain the feature F M ; Then feature F M The feature X is first added through the fifth adder (309), and then through the fourth data reorganization (310) to obtain the final output F E , generate some high-frequency information lost in the layer-by-layer feature extraction process, and realize hierarchical feature enhancement; The first and second data reorganizations are to arrange the input four-dimensional feature map into a two-dimensional matrix and flatten the spatial dimension; the third data reorganization rearranges the processed two-dimensional matrix into a four-dimensional feature map to facilitate addition operation with the original four-dimensional feature; the fourth data reorganization arranges the four-dimensional feature map into a three-dimensional feature map to facilitate processing by the subsequent adder module.

5. The state-space model-based hierarchical attention-enhanced remote sensing image super-resolution system according to claim 1, characterized in that: The first adder module (40) is configured by combining the primary extraction feature F0 and the Nth level state space model extraction feature F N and hierarchical attention-enhanced features F E Add element by element to get the deep feature F D .

6. The state-space model-based hierarchical attention-enhanced remote sensing image super-resolution system according to claim 1, characterized in that: The image reconstruction module (50) receives the deep features F D Two layers of convolution operations and one layer of upsampling operations are used to generate high-resolution remote sensing image output.

7. The method for enhancing remote sensing image super-resolution based on state-space model hierarchical attention according to claim 1, 2, 3, 4, 5 or 6, characterized in that: ① Primary feature extraction The primary feature extraction module receives the input low-resolution remote sensing image and uses a single-layer convolutional neural network to generate the primary feature representation F0 of the low-resolution remote sensing image; ② Multi-level state space model feature extraction A. 1st state space model feature extraction By stacking a plurality of space-frequency state-space model blocks, the received primary feature F0 is sequentially passed through space-frequency state-space model blocks (201, 202...20L) of 1, 2..., and 201, and then passed through a second adder module (211), thereby obtaining a first-level state-space model extracted feature F1; B. Second state space model feature extraction Using multiple stacked space-frequency state-space model blocks and adder modules with the same structure as in the first state-space model feature extraction to generate the second-level state-space model extraction feature F2; C. Feature extraction of the third state space model Using multiple stacked space-frequency state-space model blocks and adder modules with the same structure as in the first state-space model feature extraction to generate the third-level state-space model extraction feature F3; D. Follow this procedure and extract the feature F from the final N-1 level state space model. N-1 After the N-th level state space model feature extraction module, the N-th level state space model extraction feature F is generated N ; ③ Hierarchical feature attention enhancement By receiving the state space model feature extraction module at each level, the state space model extracts features F1, F2...F N , first channel-level splicing and then adaptively selecting the level information that needs to be enhanced F E ; ④Adder By extracting the primary feature F0 and the multi-level state space model feature F N and hierarchical attention-enhanced features F E Add together to get the deep feature F D ; ⑤High-resolution remote sensing image generation The image reconstruction module receives the deep features F D ,Two layers of convolution operations and one layer of upsampling operations are used to generate high-quality ,remote sensing images.

Images