Remote sensing image surface cover classification method and system based on Mamba

Abstract

The present invention discloses a remote sensing image surface cover classification method and system based on Mamba. By acquiring remote sensing image data and performing preprocessing, a remote sensing image classification network based on Mamba is constructed. By integrating the global modeling advantages of the Mamba architecture with the local fine representation characteristics of the KAG attention module, a feature extraction module with multi-scale collaborative processing capabilities is constructed, which can effectively cope with the limitations of the limited receptive field of traditional CNN and the high computational complexity of the TF network. By adopting a dual-branch parallel structure to achieve multi-granularity analysis of features, the efficiency of fine-grained feature recognition is improved while ensuring the global context association capability. In view of the inter-class imbalance characteristics of remote sensing images, a joint optimization strategy of two loss functions is adopted to effectively balance the learning weights of different land object categories. The method proposed in the present invention significantly improves the surface cover classification effect in complex remote sensing scenes and has important engineering application value.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing, and in particular to a remote sensing image surface cover classification method and system based on Mamba. Background Art

[0002] With the rapid development of remote sensing technology, the ability to acquire high-resolution remote sensing imagery has significantly improved. Land cover classification, as a core task in remote sensing image interpretation, has important applications in areas such as land resources surveys, ecological and environmental monitoring, and urban planning. Traditional land cover classification methods rely primarily on manually designed features (such as texture and spectral indices) combined with traditional machine learning models (such as support vector machines and random forests). However, these methods have limited feature representation capabilities and are unable to cope with the nonlinear distribution of complex land features and the subtle differences in high-resolution imagery.

[0003] With breakthroughs in deep learning technology, models based on convolutional neural networks (CNNs) and Transformers have gradually become the mainstream methods for remote sensing image classification. CNNs extract spatial features through local receptive fields, but their global modeling capabilities are insufficient, making it difficult to effectively capture the long-range dependencies between widely distributed objects in remote sensing imagery. Transformers, on the other hand, achieve global context modeling through a self-attention mechanism, but their computational complexity increases quadratically with image size, resulting in significant computational resource consumption and memory bottlenecks when processing high-resolution remote sensing imagery.

[0004] As a novel state-space model, the Mamba architecture, with its selective state mechanism and linear computational complexity, offers new insights into modeling long-range dependencies in high-resolution remote sensing image classification. The Kolmogorov-Arnold neural network (KAN) enhances local feature modeling through learnable activation functions, effectively capturing edge details of objects. The current research challenge lies in synergizing the global modeling advantages of Mamba with the local, refined representation capabilities of KAN to achieve accurate modeling of object boundaries while preserving the global information captured by remote sensing images. This is crucial for improving the accuracy of land cover classification.

[0005] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art. Summary of the Invention

[0006] Therefore, the purpose of the present invention is to provide a remote sensing image surface cover classification method based on Mamba, which can more efficiently and accurately identify various types of land objects through global modeling and local refinement, effectively improve the land object classification accuracy, and is suitable for surface cover classification in remote sensing scenarios.

[0007] In order to achieve the above object, the present invention also provides a remote sensing image surface cover classification method based on Mamba, comprising the following steps:

[0008] S1. Obtain original remote sensing images;

[0009] S2. Inputting the original remote sensing image into the trained Mamba remote sensing image classification network, wherein the Mamba remote sensing image classification network uses the Mamba-KAN module as a core module to form a symmetric encoder-decoder network; the core module connects the visual Mamba module and the KAG attention module in parallel; and the Mamba remote sensing image classification network is trained using the established sample set;

[0010] S3. Use the Mamba remote sensing image classification network to output the classified remote sensing image surface cover classification results.

[0011] Further preferably, the process of establishing the sample set includes the following steps:

[0012] Obtain original remote sensing image data and preprocess it;

[0013] Use ArcGIS software to outline the surface features, make the original image data and label data conform to the preset size, and obtain the pre-processed remote sensing image data;

[0014] The preprocessed images and corresponding labels are divided into training sets, test sets, and validation sets according to preset ratios.

[0015] Further preferably, the Mamba remote sensing image classification network includes:

[0016] Set the encoder to divide the slices by patch partition and project the slices into a high-dimensional space. Use the Mamba-KAN module to perform three downsampling to extract feature information.

[0017] At each horizontal stage, skip connections are used to fuse high-level semantic features with shallow features. Finally, the feature map is linearly interpolated and amplified by 4 times, and then the image pixels are segmented into multiple types of ground objects through softmax and argmax functions.

[0018] Further preferably, the training process of the Mamba remote sensing image classification network includes the following steps:

[0019] S201, input the preprocessed remote sensing image data into the surface cover classification network. During training, select the SGDoptimizer function as the parameter optimizer, use the cross entropy loss and Lovász-Softmax loss as the loss function, the initial learning rate is 0.01, the number of training iterations is 100, and the step size is 4;

[0020] S202, extracting features through the Mamba-KAN module of the encoder in the land cover classification network, and fusing the feature map generated by upsampling with the feature map on the left side by the decoder. During this process, each Mamba-KAN module obtains global and local features of the image;

[0021] S203, using a 1×1 convolution to perform channel adjustment, adjusting the number of channels in the final feature layer to the number of categories;

[0022] S204, after another four-fold upsampling, the feature map Adjust to original image resolution;

[0023] S205, through the softmax function and argmax function, the upsampled feature map Processing is performed to generate the final surface cover result map, which is the same size as the original Figure 1 The calculation formula (1) is as follows:

[0024] Formula (1)

[0025] Among them, F represents the feature map after upsampling, Indicates the proportion, It means that the number of channels is first expanded using the Linear layer, and then the height and width are enlarged by the rearrange function with a magnification ratio of 4 to achieve upsampling.

[0026] Further preferably, the step of connecting the visual Mamba module and the KAG attention module in parallel includes:

[0027] S210, extracting features from the input remote sensing image data to obtain input features F;

[0028] S211, inputting the input feature F into the visual Mamba module, performing dual-branch parallel processing, and obtaining a first feature element and a second feature element respectively;

[0029] S212: Input the input feature F into the KAG attention module, divide it into multiple sub-features, and normalize each sub-feature and then fuse them to obtain the third feature element;

[0030] S213, multiply the first feature element, the second feature element and the third feature element, perform feature mapping through a linear layer, and add them to the original input feature Get the final output.

[0031] Further preferably, in S211, inputting the input feature F into the visual Mamba module and performing dual-branch parallel processing includes:

[0032] Normalize the input feature F to generate feature F';

[0033] The first branch generates a feature map through a linear layer, and the calculation formula (2) is as follows:

[0034] Formula (2)

[0035] is the first characteristic element;

[0036] After the second branch adjusts the channel dimension through the linear layer, it is processed by 3×3 depthwise separable convolution and SiLU activation function, and input into the SS2D module to achieve global modeling across channels and spatial dimensions. The calculation formula (3) is as follows:

[0037] Formula (3)

[0038] is the second characteristic element, Indicates the execution of separable convolution operation; LN indicates layer normalization operation.

[0039] Further preferably, in S212, the input feature F is input into the KAG attention module, divided into multiple sub-features, and each sub-feature is normalized and then fused to obtain a third feature element, including:

[0040] Input features Evenly divided into i sub-features along the channel dimension ;

[0041] The importance of each sub-feature channel is evaluated through the efficient channel attention ECA module;

[0042] , Evaluate the importance of each sub-feature;

[0043] Perform Tanh normalization on each sub-feature to generate a constrained feature in the range [-1, 1];

[0044] Perform polynomial fusion on normalized features based on Gram polynomial to generate enhanced features ;

[0045] ;

[0046] ;

[0047] ;

[0048] in, and are all learnable parameters, It's about A polynomial of degree ; bias function use function; Represents the result after Gram polynomial fusion;

[0049] Calculate attention weights based on spatial global average features ;

[0050] Concatenate the original features and multiply them Forming the final third characteristic element.

[0051] Further preferably, the attention weight is calculated based on the spatial global average feature The following formula (4) is used to calculate:

[0052] Formula (4)

[0053] in, H is the overall height of the feature map, h represents the height of each feature, W is the overall width of the feature map, and w is the width of each feature.

[0054] The present invention also provides a Mamba-based remote sensing image surface cover classification system, comprising: a data acquisition module, a sample library, and a Mamba remote sensing image classification network;

[0055] The data acquisition module is used to acquire original remote sensing images;

[0056] The sample library is used to train the constructed Mamba remote sensing image classification network;

[0057] The Mamba remote sensing image classification network uses the Mamba-KAN module as a core module to form a symmetric encoder-decoder network; the core module connects the visual Mamba module and the KAG attention module in parallel; and uses the trained network parameters to output a classified remote sensing image surface cover classification result based on the input original remote sensing image.

[0058] The Mamba-based remote sensing image surface cover classification method and system disclosed in this application constructs the Mamba-KAN module by combining Mamba and Kolmogorov-Arnold neural networks, so that the model has powerful global modeling and local refined representation capabilities, effectively addressing the bottlenecks of limited receptive field of traditional convolutional neural networks and high computational complexity of Transformer models. This module realizes multi-scale collaborative modeling through a parallel architecture, so that the network can more efficiently and accurately identify various types of land objects while maintaining global context perception. At the same time, the model adopts the cross entropy and Lovász-Softmax joint loss function to effectively alleviate the category imbalance problem that is prevalent in remote sensing images. The surface cover classification method provided by the present invention effectively improves the accuracy of land object classification and is suitable for surface cover classification in remote sensing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart of a Mamba-based remote sensing image surface cover classification method provided by the present invention;

[0060] Figure 2 This is a structural diagram of a Mamba-based remote sensing image land cover classification system provided by the present invention;

[0061] Figure 3 A structural diagram of a land cover classification model constructed in an embodiment of the present invention;

[0062] Figure 4 This is a structural diagram of the Mamba-KAN module constructed in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] like Figure 1 As shown, an embodiment of one aspect of the present invention provides a remote sensing image surface cover classification method based on Mamba, comprising the following steps:

[0065] S1. Obtain original remote sensing images. It should be noted that if the original remote sensing images are real-time remote sensing image data, they are input into the trained Mamba remote sensing image classification network and the classification results are output. If the original remote sensing images are from the past, they can be used to create a sample library.

[0066] Further preferably, the process of establishing the sample set includes the following steps:

[0067] Obtain original remote sensing image data and preprocess it;

[0068] Use ArcGIS software to outline the surface features, make the original image data and label data conform to the preset size, and obtain the pre-processed remote sensing image data;

[0069] The pre-processed images and corresponding labels are divided into training, test, and validation sets according to a preset ratio. This involves constructing a sample set for land cover classification using the original remote sensing images, manually labeling the samples, cropping the images after labeling, and then dividing the samples into training, validation, and test sets in a ratio of 7:2:1 to ensure the effectiveness and generalization of the model training.

[0070] S2. Input the original remote sensing image into the trained Mamba remote sensing image classification network. The Mamba remote sensing image classification network uses the Mamba-KAN module as the core module to form a symmetric encoder-decoder network. The core module connects the visual Mamba module and the KAG attention module in parallel. The Mamba remote sensing image classification network is trained using the established sample set. The specific structure of the Mamba remote sensing image classification network is as follows: Figure 3 shown.

[0071] Furthermore, the Mamba remote sensing image classification network includes:

[0072] Set the encoder to divide the slices by patch partition and project the slices into a high-dimensional space. Use the Mamba-KAN module to perform three downsampling to extract feature information.

[0073] At each horizontal stage, skip connections are used to fuse high-level semantic features with shallow features. Finally, the feature map is linearly interpolated and amplified by 4 times, and then the image pixels are segmented into multiple types of ground objects through softmax and argmax functions.

[0074] like Figure 4 As shown, the visual Mamba module is connected in parallel with the KAG attention module, including:

[0075] S210, extracting features from the input remote sensing image data to obtain input features F;

[0076] S211, inputting the input feature F into the visual Mamba module, performing dual-branch parallel processing, and obtaining a first feature element and a second feature element respectively;

[0077] In S211, the input feature F is input to the visual Mamba module, and dual-branch parallel processing is performed, including:

[0078] Normalize the input feature F to generate feature F';

[0079] The first branch generates a feature map through a linear layer, and the calculation formula (2) is as follows:

[0080] Formula (2)

[0081] is the first characteristic element;

[0082] After the second branch adjusts the channel dimension through the linear layer, it is processed by 3×3 depthwise separable convolution and SiLU activation function, and input into the SS2D module to achieve global modeling across channels and spatial dimensions. The calculation formula (3) is as follows:

[0083] Formula (3)

[0084] is the second characteristic element, Indicates the execution of separable convolution operation; LN indicates layer normalization operation.

[0085] S212: Input the input feature F into the KAG attention module, divide it into multiple sub-features, and normalize each sub-feature and then fuse them to obtain the third feature element;

[0086] In S212, the input feature F is input to the KAG attention module, divided into multiple sub-features, and each sub-feature is normalized and then fused to obtain the third feature element, including:

[0087] Input features Evenly divided into i sub-features along the channel dimension ;

[0088] The importance of each sub-feature channel is evaluated through the efficient channel attention ECA module;

[0089] , Evaluate the importance of each sub-feature;

[0090] Perform Tanh normalization on each sub-feature to generate a constrained feature in the range [-1, 1];

[0091] Perform polynomial fusion on normalized features based on Gram polynomial to generate enhanced features ;

[0092] ;

[0093] ;

[0094] ;

[0095] in, and are all learnable parameters, It's about A polynomial of degree ; bias function use function; Represents the result after Gram polynomial fusion.

[0096] S213, multiply the first feature element, the second feature element and the third feature element, perform feature mapping through a linear layer, and add them to the original input feature Get the final output.

[0097] S2131. Calculate attention weight based on spatial global average features ;

[0098] Further preferably, the attention weight is calculated based on the spatial global average feature The following formula (4) is used to calculate:

[0099] Formula (4)

[0100] in, H is the overall height of the feature map, h represents the height of each feature, W is the overall width of the feature map, and w is the width of each feature;

[0101] S2132, concatenate the original features and multiply them To form the final third characteristic element. Concatenate the original features and multiply them by The final output features are formed using the following calculation formula:

[0102] ;

[0103] ;

[0104] Multiply the corresponding elements of the features, perform feature mapping through a linear layer, and then add to the original features To generate the final output of the Mamba-KAN module, the calculation formula is as follows:

[0105] .

[0106] Furthermore, the training process of the Mamba remote sensing image classification network includes the following steps:

[0107] S201, input the preprocessed remote sensing image data into the surface cover classification network. During training, select the SGDoptimizer function as the parameter optimizer, use the cross entropy loss and Lovász-Softmax loss as the loss function, the initial learning rate is 0.01, the number of training iterations is 100, and the step size is 4;

[0108] S202, extracting features through the Mamba-KAN module of the encoder in the land cover classification network, and fusing the feature map generated by upsampling with the feature map on the left side by the decoder. During this process, each Mamba-KAN module obtains global and local features of the image;

[0109] S203, using a 1×1 convolution to perform channel adjustment, adjusting the number of channels in the final feature layer to the number of categories;

[0110] S204, after another four-fold upsampling, the feature map Adjust to original image resolution;

[0111] S205, through the softmax function and argmax function, the upsampled feature map Processing is performed to generate the final surface cover result map, which is the same size as the original Figure 1 The calculation formula (1) is as follows:

[0112] Formula (1)

[0113] Among them, F represents the feature map after upsampling, Indicates the proportion, It means that the number of channels is first expanded using the Linear layer, and then the height and width are enlarged by the rearrange function with a magnification ratio of 4 to achieve upsampling.

[0114] S3. Use the Mamba remote sensing image classification network to output the classified remote sensing image surface cover classification results.

[0115] like Figure 2 ,The present invention also provides a Mamba-based remote sensing image surface cover classification system for executing the steps of the above-mentioned classification method, including: a data acquisition module, a sample library and a Mamba remote sensing image classification network;

[0116] The data acquisition module is used to acquire original remote sensing images;

[0117] The sample library is used to train the constructed Mamba remote sensing image classification network;

[0118] The Mamba remote sensing image classification network uses the Mamba-KAN module as a core module to form a symmetric encoder-decoder network; the core module connects the visual Mamba module and the KAG attention module in parallel; and uses the trained network parameters to output a classified remote sensing image surface cover classification result based on the input original remote sensing image.

[0119] The experiments in this paper were conducted in an Ubuntu 22.04 operating system environment. All network models were implemented using the PyTorch framework and accelerated training using an NVIDIA RTX 4090 24GB GPU. During the model training phase, stochastic gradient descent (SGD) was used as the optimizer, and a joint cross-entropy and Lovász-Softmax loss function was used to guide model parameter updates. This joint loss function generates an error signal by quantifying the difference between the predicted value and the true label. The gradient is calculated using the backpropagation algorithm, driving the iterative optimization of the model weights and biases. The specific calculation method is as follows:

[0120] ,

[0121] in, represents the number of categories, is the true label distribution, is the predicted probability output by the model, is the Lovász extension of IoU, representing a piecewise linear function with a global minimum, yes The error vector of the class, , is the number of pixels considered.

[0122] The final loss function is expressed as:

[0123]

[0124] The method further includes inputting the image to be detected into the obtained Mamba remote sensing image classification network for classification. To test the performance of the surface cover classification method provided by the present invention, in one embodiment, the following formula is used to quantify the prediction results:

[0125]

[0126]

[0127]

[0128]

[0129] Where,

[0130] Precision: This indicates the reliability of the model's prediction results and is calculated as the ratio of the number of correctly predicted positive samples to the total number of positive samples predicted by the model.

[0131] Recall: Recall rate reflects the model's coverage of positive samples and is calculated as the ratio of the number of correctly predicted positive samples to the total number of positive samples actually existing in the dataset;

[0132] F1-score: A balanced indicator for comprehensively evaluating model performance, integrating precision and recall through the harmonic mean;

[0133] IoU: measures the spatial match between the predicted area and the true annotated area, calculated as the ratio of the intersection area to the union area of the two;

[0134] TP: positive samples correctly predicted as positive; FP: negative samples wrongly predicted as positive; FN: positive samples wrongly predicted as negative.

[0135] It is important to note that the method steps involved in this technical solution are subject to adjustment, including but not limited to reorganizing the step sequence, adding or deleting steps, or changing the execution method. Specifically, the technical steps described in this invention may be executed in parallel, sequentially, or in any other logical order, as long as the intended objectives of the technical solution are achieved. The specific implementation is not limited to the order described herein.

[0136] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A remote sensing image surface cover classification method based on Mamba, characterized in that: The following steps are involved: S1. Obtain original remote sensing images; S2. Inputting the original remote sensing image into the trained Mamba remote sensing image classification network, wherein the Mamba remote sensing image classification network uses the Mamba-KAN module as a core module to form a symmetric encoder-decoder network; the core module connects the visual Mamba module and the KAG attention module in parallel; the Mamba remote sensing image classification network is trained using the established sample set; the connecting the visual Mamba module and the KAG attention module in parallel includes: S210, extracting features from the input remote sensing image data to obtain input features F; S211, inputting the input feature F into the visual Mamba module, performing dual-branch parallel processing, and obtaining a first feature element and a second feature element respectively; S212: Input the input feature F into the KAG attention module, divide it into multiple sub-features, and normalize each sub-feature and then fuse them to obtain the third feature element; S213, multiply the first feature element, the second feature element and the third feature element, perform feature mapping through a linear layer, and add them to the original input feature Get the final output; S3. Use the Mamba remote sensing image classification network to output the classified remote sensing image surface cover classification results.

2. The Mamba-based remote sensing image land cover classification method according to claim 1, characterized in that: The sample set is established by the following steps: Obtain original remote sensing image data and preprocess it; Use ArcGIS software to outline the surface features, make the original image data and label data conform to the preset size, and obtain the pre-processed remote sensing image data; The preprocessed images and corresponding labels are divided into training sets, test sets, and validation sets according to preset ratios.

3. The Mamba-based remote sensing image land cover classification method according to claim 1, characterized in that: The Mamba remote sensing image classification network includes: Set the encoder to divide the slices by patch partition and project the slices into a high-dimensional space. Use the Mamba-KAN module to perform three downsampling to extract feature information. At each level, skip connections are used to fuse high-level semantic features with shallow features. The feature map is linearly interpolated and magnified 4 times, and then the image pixels are segmented into multiple types of ground objects through the softmax and argmax functions.

4. The Mamba-based remote sensing image land cover classification method according to claim 1, characterized in that: The training process of the Mamba remote sensing image classification network includes the following steps: S201, input the preprocessed remote sensing image data into the surface cover classification network. During training, the SGD-optimizer function is selected as the parameter optimizer, the cross entropy loss and Lovász-Softmax loss are used as the loss function, the initial learning rate is 0.01, the number of training iterations is 100, and the step size is 4; S202, extracting features through the Mamba-KAN module of the encoder in the land cover classification network, and fusing the feature map generated by upsampling with the feature map on the left side by the decoder. During this process, each Mamba-KAN module obtains global and local features of the image; S203, using a 1×1 convolution to perform channel adjustment, adjusting the number of channels in the final feature layer to the number of categories; S204, after another four-fold upsampling, the feature map Adjust to original image resolution; S205, through the softmax function and argmax function, the upsampled feature map After processing, the final surface cover result map is generated, whose size is consistent with the original map. The calculation formula (1) is as follows: Formula (1) Among them, F represents the feature map after upsampling, Indicates the proportion, It means that the number of channels is first expanded using the Linear layer, and then the height and width are enlarged by the rearrange function with a magnification ratio of 4 to achieve upsampling.

5. The Mamba-based remote sensing image land cover classification method according to claim 4, characterized in that: In S211, the input feature F is input to the visual Mamba module, and dual-branch parallel processing is performed, including: Normalize the input feature F to generate feature F'; The first branch generates a feature map through a linear layer, and the calculation formula (2) is as follows: Formula (2) is the first characteristic element; After the second branch adjusts the channel dimension through the linear layer, it is processed by 3×3 depthwise separable convolution and SiLU activation function, and input into the SS2D module to achieve global modeling across channels and spatial dimensions. The calculation formula (3) is as follows: Formula (3) is the second characteristic element, Indicates the execution of separable convolution operation; LN indicates layer normalization operation.

6. The Mamba-based remote sensing image land cover classification method according to claim 4, characterized in that: In S212, the input feature F is input to the KAG attention module, divided into multiple sub-features, and each sub-feature is normalized and then fused to obtain the third feature element, including: Input features Evenly divided into i sub-features along the channel dimension ; The importance of each sub-feature channel is evaluated through the efficient channel attention ECA module; , Evaluate the importance of each sub-feature; Perform Tanh normalization on each sub-feature to generate a constrained feature in the range [-1, 1]; Perform polynomial fusion on normalized features based on Gram polynomial to generate enhanced features ; ； ； ； in, and are all learnable parameters, It's about A polynomial of degree ; bias function use function; Represents the result after Gram polynomial fusion; Calculate attention weights based on spatial global average features ; Concatenate the original features and multiply them Forming the final third characteristic element.

7. The Mamba-based remote sensing image land cover classification method according to claim 6, characterized in that: The attention weight is calculated based on the spatial global average feature The following formula (4) is used to calculate: Formula (4) in, H is the overall height of the feature map, h represents the height of each feature, W is the overall width of the feature map, and w is the width of each feature.

8. A remote sensing image land cover classification system based on Mamba, characterized by , used to implement the Mamba-based remote sensing image surface cover classification method described in any one of claims 1 to 7, comprising: a data acquisition module, a sample library, and a Mamba remote sensing image classification network; The data acquisition module is used to acquire original remote sensing images; The sample library is used to train the constructed Mamba remote sensing image classification network; The Mamba remote sensing image classification network uses the Mamba-KAN module as a core module to form a symmetric encoder-decoder network; the core module connects the visual Mamba module and the KAG attention module in parallel; and uses the trained network parameters to output a classified remote sensing image surface cover classification result based on the input original remote sensing image.

Images