A mamba prompt learning semantic segmentation method and system for remote sensing images

Abstract

This invention discloses a Mamba cue-learning semantic segmentation method and system for remote sensing images, belonging to the field of remote sensing image processing technology. The invention first inputs RGB remote sensing images and elevation data into a teacher network, capturing multi-directional long-range dependencies and extracting bimodal teacher features by embedding a Mamba fusion module. Then, using a cue-learning module, the bimodal teacher features are transferred to a student network that only inputs RGB remote sensing images, followed by joint optimization using cross-entropy loss and temperature distillation loss. Finally, the student network performs semantic segmentation on the RGB remote sensing images. This invention significantly reduces the number of model parameters and computational complexity while improving the segmentation accuracy of slender structures and complex boundaries such as traditional village buildings, roads, farmland, vegetation, and drainage systems.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing technology, specifically relating to a method and system for semantic segmentation of remote sensing images using Mamba cue learning. Background Technology

[0002] With the widespread adoption of high-resolution remote sensing imagery and big data, semantic segmentation for automatic identification of features such as buildings, roads, farmland, and vegetation has become a crucial technology for land spatial management, digital rural construction, and the protection of traditional villages. Existing methods, mostly based on convolutional neural networks and Transformer structures, have achieved some success in typical urban scenarios. However, in scenarios like traditional villages where targets are small-scale, elongated, have complex boundaries, and mixed categories, problems such as missed detection of small targets, category confusion, and "flattened" boundaries often arise. Some works improve accuracy through multimodal fusion or complex decoding structures (such as RGB and depth information), but this introduces a large number of parameters and computational overhead, hindering efficient deployment on cloud platforms and edge terminals. Meanwhile, cue learning is gaining traction in the vision field, but systematic applications for remote sensing semantic segmentation, especially for remote sensing imagery of traditional villages, are still lacking. How to maintain model lightweightness while simultaneously ensuring global context modeling capabilities and accuracy in depicting elongated structures and complex boundaries remains a key unresolved issue and a research hotspot in current technologies. Summary of the Invention

[0003] This invention proposes a Mamba cue-learning semantic segmentation method and system for remote sensing images. By constructing a dedicated remote sensing image dataset for traditional villages, a Mamba fusion module with two-dimensional selective scanning capability is introduced. Combined with a cross-modal cue-learning module and a joint loss function, the segmentation accuracy and generalization ability of remote sensing images of traditional villages are improved without significantly increasing the number of parameters and computational overhead.

[0004] The present invention is achieved through the following technical solutions.

[0005] A semantic segmentation method for Mamba cue learning of remote sensing images includes the following steps:

[0006] Step 1: Acquire RGB remote sensing images, preprocess the RGB remote sensing images, construct a remote sensing image dataset, and divide it into a training set, a validation set, and a test set;

[0007] Step 2: Construct a teacher-student network consisting of a teacher network and a student network. The teacher network uses RGB remote sensing imagery and elevation data as bimodal inputs to generate RGB image features and elevation information features at different levels. A Mamba fusion module is introduced into each level of the teacher network, and the RGB image features and elevation information features are processed by the Mamba fusion module to obtain bimodal teacher features. The student network uses RGB remote sensing imagery as input to obtain student features at each level. The backbone of the student network corresponds one-to-one with the levels of the teacher network.

[0008] Step 3: Construct a prompting learning module. The student features of each layer are concatenated with the bimodal teacher features through the Mamba module to obtain student enhancement features. The student enhancement features output from each layer serve as the input to the next layer of the student network.

[0009] Step 4: Multiply and concatenate the predicted image obtained by the decoder processing of the bimodal teacher features output from the last layer of the teacher network and the predicted image obtained by the decoder processing of the student enhancement features output from the last layer of the student network; construct a joint loss function between the concatenated predicted image and the true value to calculate the loss, and perform end-to-end training on the student network and the prompting learning module.

[0010] Step 5: Call the already trained student network to perform semantic segmentation on the input RGB remote sensing image and generate the segmentation results.

[0011] Furthermore, the remote sensing image dataset consists of image blocks and label maps. The image blocks are obtained by sequentially performing geometric correction, radiometric correction, registration processing, region cropping, and scene filtering on RGB remote sensing images. The label maps are obtained by pixel-level annotation of image blocks based on preset land cover categories in RGB remote sensing images.

[0012] Furthermore, the Mamba fusion module includes a Mamba module and a residual layer. The Mamba module divides the input features into two paths. One path is processed sequentially through a linear layer, a C3S module, a two-dimensional selective scan module, a linear mapping module, and a layer normalization module to obtain fusion context features. The other path is processed sequentially through a linear layer and a nonlinear activation function, and then multiplied element-wise with the fusion context features to obtain the Mamba context gating operator. The RGB image features and elevation information features are processed by the Mamba module to obtain output features. The residual layer is used to receive the output features and perform weighted superposition and residual fusion on the output features to obtain bimodal teacher features.

[0013] Furthermore, the joint loss function is obtained by weighting the cross-entropy loss and the temperature distillation loss according to preset weighting coefficients, thereby minimizing the joint loss.

[0014] Furthermore, during the training phase, the parameters of the teacher network are frozen, and both the teacher network and the student network propagate forward simultaneously. The bimodal teacher features output by the teacher network provide structured guidance for the student network. During the inference phase, the teacher network and elevation data are completely removed, and the student network is allowed to independently complete the semantic segmentation task.

[0015] Furthermore, the two-dimensional selective scanning sequentially scans the features along four directions—forward, reverse, upward, and downward—to obtain the corresponding directional context states. These are then processed through channel concatenation, linear projection, and layer normalization to obtain fused context features. The C3S module performs 3×3 convolutions and activation functions on the features. Calculation.

[0016] Furthermore, the process of state knowledge transfer implemented by the prompting learning module is represented by the following formula:

[0017] ;

[0018] ;

[0019] in, To enhance students' characteristics, Let r represent the bimodal teacher features of the teacher network at layer i. i Let i represent the student characteristics of the i-th layer of the student network. () is the stopping gradient operator. For Mamba context gate operators, For layer normalization, It is a linear transformation matrix. It is a 1×1 linear projection matrix. For the Sigmoid function, ⊙ represents element-wise multiplication. The output of the i-th layer of the teacher network after decoding, where T is the temperature coefficient. This is the decoded output of the i-th layer of the student network.

[0020] This invention also provides a Mamba cue learning semantic segmentation system for remote sensing images, comprising:

[0021] Data acquisition module: Acquires high-resolution RGB remote sensing images and corresponding elevation data of traditional village areas, performs geometric correction, radiometric correction and registration processing, and outputs standardized RGB remote sensing images and elevation data;

[0022] Feature extraction module: includes a bimodal teacher feature extraction submodule and a unimodal student feature extraction submodule; the bimodal teacher feature extraction submodule performs multi-scale encoding on RGB remote sensing images and elevation data, and outputs bimodal teacher features; the student feature extraction submodule encodes RGB remote sensing images and outputs student features corresponding to the bimodal teacher feature levels;

[0023] Mamba fusion module: Connects to the bimodal teacher feature extraction submodule, performs four-directional two-dimensional selective scanning, gated feature transformation and residual fusion on the bimodal teacher features, and outputs fused bimodal teacher features;

[0024] Hint learning module: Connects the Mamba fusion module and the student feature extraction submodule, freezes the teacher network parameters, generates hint vectors and performs affine modulation on student features to complete cross-modal knowledge transfer;

[0025] Training and optimization module: Connects to the prompting learning module, constructs a joint loss function, and performs end-to-end optimization of the parameters of the student network and the prompting learning module;

[0026] Inference module: Connects to the student feature extraction submodule, loads the trained student network model, performs forward inference using only RGB remote sensing images as input features, and outputs the land cover category segmentation result;

[0027] Display module: Connected to the inference module, it overlays the segmentation results with geographic coordinates to generate a color raster map, and supports layer switching, transparency adjustment and result export.

[0028] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor implements the above-described Mamba cue learning semantic segmentation method for remote sensing images.

[0029] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described Mamba cue learning semantic segmentation method for remote sensing images.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] By introducing the Mamba fusion module into the network, multi-directional context modeling of features along the horizontal and vertical directions is performed, which enhances the ability to identify small targets and complex boundaries in traditional village scenes.

[0032] By using the prompting learning module, the semantic knowledge of the teacher network is injected into the student network in the form of feature prompts, which improves the segmentation performance and result stability while keeping the network lightweight and using only RGB remote sensing images as input.

[0033] Experimental results on a self-built remote sensing image dataset show that the method of the present invention can achieve an average accuracy of approximately 90.10% mAcc and an average intersection-union ratio of approximately 82.57% mIoU, indicating that it has high segmentation accuracy in the traditional semantic segmentation task of village remote sensing images. Attached Figure Description

[0034] Figure 1 This is a general structural diagram of this embodiment;

[0035] Figure 2 This is a schematic diagram of the Mamba Fusion Module (MFM) structure.

[0036] Figure 3 This is a comparison chart of semantic segmentation results. Detailed Implementation

[0037] The invention will now be explained in further detail with reference to the accompanying drawings.

[0038] like Figure 1 As shown in the figure, this embodiment of the invention provides a Mamba cue learning semantic segmentation method for remote sensing images, and the specific steps are as follows:

[0039] Step 1: Acquire RGB remote sensing images, preprocess the RGB remote sensing images, construct a remote sensing image dataset, and divide it into a training set, a validation set, and a test set;

[0040] Step 2: Construct a teacher-student network consisting of a teacher network and a student network. The teacher network uses RGB remote sensing imagery and elevation data as bimodal inputs to generate RGB image features and elevation information features at different levels. A Mamba fusion module is introduced into each level of the teacher network, and the RGB image features and elevation information features are processed by the Mamba fusion module to obtain bimodal teacher features. The student network uses RGB remote sensing imagery as input to obtain student features at each level. The backbone of the student network corresponds one-to-one with the levels of the teacher network.

[0041] Step 3: Construct a prompting learning module. The student features of each layer are concatenated with the bimodal teacher features through the Mamba module to obtain student enhancement features. The student enhancement features output from each layer serve as the input to the next layer of the student network.

[0042] Step 4: Multiply and concatenate the predicted image obtained by the decoder processing of the bimodal teacher features output from the last layer of the teacher network and the predicted image obtained by the decoder processing of the student enhancement features output from the last layer of the student network; construct a joint loss function between the concatenated predicted image and the true value to calculate the loss, and perform end-to-end training on the student network and the prompting learning module.

[0043] Step 5: Call the already trained student network to perform semantic segmentation on the input RGB remote sensing image and generate the segmentation results.

[0044] In this embodiment, a remote sensing image dataset (TV-RSI) is constructed by acquiring high-resolution RGB remote sensing images covering traditional village areas. The RGB remote sensing images are satellite remote sensing images. Geometric correction, radiometric correction, and registration processing are performed on the remote sensing images according to the national unified coordinate reference system to ensure consistency in spatial location and grayscale range between images from different sources and at different times. The processed images are then cropped and scene-filtered to extract image blocks containing the built-up areas of traditional villages and their surrounding environment. Each image block is denoted as:

[0045] ;

[0046] Where k represents any sample, x k The k-th sample is a traditional village RGB remote sensing image block. R represents the set of real numbers, H and W are the number of pixel rows and columns, respectively, and 3 represents 3 channels, which correspond to the RGB bands of the RGB remote sensing image.

[0047] Based on the spatial structure characteristics of traditional villages, the ground features in the RGB remote sensing imagery are divided into six preset categories: buildings, roads, farmland, vegetation, drainage, and background, forming a ground feature category set C={1,…,6}. On this basis, a combination of manual labeling and auxiliary tools is used to assign a category label to each pixel location p in the image block. This yields a label map that corresponds one-to-one with each image patch:

[0048] ;

[0049] Forming sample pairs y k The label image corresponding to the image patch of the k-th sample. The total number of samples is then used to obtain the TV-RSI remote sensing image dataset, in the following format:

[0050] ;

[0051] Where D represents the remote sensing image dataset, and N represents the number of samples. To enhance the network's adaptability to traditional village scenes of different scales and forms, the image patches can be normalized and sized based on the above process. Data augmentation operations such as sliding cropping, random flipping, rotation, and color perturbation can then be applied to make the distribution of pixels of different categories more balanced across the samples. Finally, the remote sensing image dataset D is divided into training, validation, and test sets according to a predetermined ratio for subsequent training, parameter tuning, and performance evaluation of the teacher-student network, Mamba fusion module, and cue learning module, providing a unified data foundation for the implementation of this embodiment.

[0052] like Figure 2 As shown, this embodiment introduces a Mamba fusion module in each layer of the teacher network to perform multi-directional contextual modeling and residual fusion of RGB image features and elevation information features. Specifically, the input feature X is divided into two paths. One path passes through a linear layer and undergoes 3×3 convolution and activation function operation (C3S module) to obtain intermediate feature U. Using two-dimensional selective scanning SS2D, the intermediate feature U is sequentially scanned in the forward and reverse directions in the horizontal direction and the downward and upward directions in the vertical direction to obtain contextual features S in four directions. → S ← S ↓ S ↑ The context features from the four directions are concatenated along the channel dimension and transformed sequentially by linear projection and layer normalization to obtain the fused context feature Y. Another path, after passing through a linear layer and activation function, is element-wise multiplied with the fused context feature Y to construct a Mamba context gating operator. ; Utilizing the Mamba context gating operator The RGB image features and elevation information features of each level of the teacher network are transformed to obtain the gated output feature F. r,i and F d,i Then through the feature transformation function For F r,i and F d,i Perform channel and scale matching, and input the residual layer. We perform weighted superposition and residual fusion to output the bimodal teacher features corresponding to level i. Its core calculation process is shown in the following formula:

[0053] ;

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] ;

[0059] ;

[0060] Where: X is the input feature. This represents a 3×3 convolution operation; To perform selective scanning of the intermediate feature U in the horizontal (→, ←) and vertical (↓, ↑) directions, we obtain the context features S→, S←, S↓, S↑ in four directions; This indicates splicing on the channel dimension. For linear projection, For layer normalization; For learnable weight matrix, Here, ⊙ represents the activation function, and ⊙ represents element-wise multiplication. For the characteristic transformation function, It is a residual layer.

[0061] This embodiment uses a prompting learning module that includes a multi-level prompting mapping module and an affine modulation module to form an interpretable cross-modal knowledge injection path within the network, enabling the student network to obtain more robust and discriminative semantic expressions in complex scenarios of traditional villages.

[0062] The affine modulation module is used to align bimodal teacher and student features at the same layer and scale, and to inject teacher-side bimodal semantics into the student network in the form of low-dimensional cues. Let the... The characteristics of a bimodal teacher are:

[0063] ;

[0064] in This indicates the characteristics of a bimodal teacher. This refers to the spatial height dimension of the feature of this layer. This refers to the spatial width dimension of the feature in this layer. For the number of teacher channels; the corresponding number of channels. The characteristics of students at each level are:

[0065] ;

[0066] in For student characteristics, The number of student channels indicates the mapping module's requirements. Spatial compression and channel mapping are performed to obtain a low-dimensional cue vector:

[0067] ;

[0068] in, Represents a low-dimensional cue vector. This represents a cue mapping operator consisting of a series of convolutions and nonlinear transformations, in order to... As input, scaling and offset factors are generated through linear mapping:

[0069] ;

[0070] in, This is the scaling factor. This is the offset coefficient. To scale learnable weights, For offset learnable weights;

[0071] And Broadcasting in the spatial dimension, affine modulation is applied to student characteristics to obtain the modulated student's [number]. Layer output

[0072] ;

[0073] in, To implement affine modulation of student characteristics, For learnable weights, This indicates element-wise multiplication. As a student network The updated features of the layer are then fed into subsequent student encoding and decoding layers to generate pixel-level predictions and participate in cross-entropy supervision and distillation optimization. Specifically, the corresponding layer... Student characteristics From At the level The vector representation of , whose corresponding quantity after the above affine modulation is . Both are equivalent expressions of the same process from the perspectives of "hierarchy" and "space".

[0074] The prompting learning module achieves cross-modal knowledge transfer through a multi-level prompting mapping module. The calculation process is as follows:

[0075] ;

[0076] ;

[0077] in, To enhance students' characteristics, Let r represent the bimodal teacher features of the teacher network at layer i. i Let i represent the student characteristics of the i-th layer of the student network. () is the stopping gradient operator. For Mamba context gate operators, For layer normalization, It is a linear transformation matrix. It is a 1×1 linear projection matrix. For the Sigmoid function, ⊙ represents element-wise multiplication. The output of the i-th layer of the teacher network after decoding, where T is the temperature coefficient. This is the decoded output of the i-th layer of the student network.

[0078] With the help of the above-mentioned learning module, the ability of the teacher network to encode geometric information such as terrain undulation, building height and spatial layout of ground features can be inherited while maintaining the student network's ability to accept only RGB remote sensing image input and its lightweight structure.

[0079] To simultaneously constrain the student network's ability to fit the true values ​​and its ability to inherit the semantic distribution of the teacher network, this invention employs cross-entropy loss and temperature distillation loss to jointly optimize the learnable parameters of the student network. The joint loss is constructed as shown in the formula:

[0080] ;

[0081] Where λ is the balance coefficient, used to adjust the relative weights of cross-entropy loss and temperature distillation loss during the training process. For cross-entropy loss, This represents the temperature distillation loss. Let Ω be the effective pixel set. For student networks in pixels The predicted probability distribution at the location, This corresponds to the actual value of the one-hot encoding. and Let the soft label distributions of the teacher network and student network be respectively under the influence of temperature coefficient T. Then, the cross-entropy loss and temperature distillation loss can be written as:

[0082] ;

[0083] ;

[0084] in, It is a pixel-level cross-entropy function. For the Kullback–Leibler divergence, This is used to compensate for the gradient magnitude caused by temperature scaling. By minimizing the joint loss mentioned above, and under the conditions of frozen teacher network parameters and the participation of the cue learning module in modulation, end-to-end training is achieved on the student network and its cue learning module. This allows the student network to accurately fit pixel-level real labels while maintaining a lightweight structure and receiving only RGB remote sensing image input. At the output distribution level, it also inherits the semantic knowledge formed by the bimodal teacher features and the Mamba fusion module, thereby reducing the difference between the real labels and the predicted probability distribution and improving the segmentation accuracy and stability in complex scenes of traditional villages.

[0085] During the reasoning phase, elevation information and the teacher network are no longer used; only the student network that has completed the training is retained. For input traditional village RGB remote sensing images Perform forward reasoning and output the semantic segmentation result:

[0086] ;

[0087] Where y represents the pixel-level classification results of preset categories such as buildings, roads, farmland, vegetation, and drainage. Here are the parameters for the student network, and C is the feature category index.

[0088] In this way, the present invention completely eliminates the additional computation and storage overhead brought about by bimodal branching and teacher network during the deployment phase, and can achieve automatic identification and segmentation of land cover categories such as buildings, roads, farmland, vegetation, and drainage by relying solely on student network.

[0089] Tables 1, 2, and 3 respectively show the data comparison of FCN-8s (fully convolutional network-8s), ACNet (attention-aware convolutional network), TSNet (spatiotemporal convolutional network), ESANet (efficient spatial attention network), DCSwin (dual-scale Swin network), DGPRNet (dual-graph pyramid representation network), and Ours (the method in this embodiment) on three datasets for remote sensing image segmentation: TV-RSI (remote sensing image dataset), Vaihingen (Weinheim dataset), and Potsdam (Potsdam dataset).

[0090] This embodiment demonstrates significant advantages in traditional village remote sensing image segmentation tasks. As shown in Table 1, the accuracy (Acc) and intersection-over-union ratio (IoU) of the method in this embodiment for various land features (farmland, buildings, drainage systems, roads, vegetation, etc.) are significantly better than those of the comparative models such as FCN-8s and ACNet. The average accuracy (mAcc) reaches 90.10%, and the average intersection-over-union ratio (mIoU) reaches 82.57%. It is comprehensively leading in the segmentation of slender land features (such as drainage systems and roads) and the comprehensive accuracy of multiple categories.

[0091] Table 1. Experimental results on the TV-RSI dataset

[0092]

[0093] In remote sensing image segmentation tasks involving various land features such as impervious surfaces, buildings, low vegetation, trees, and vehicles, the method in this embodiment (Ours) significantly outperforms comparable models such as FCN-8s, ACNet, TSNet, ESANet, DCSwin, and DGPRNet. As shown in Table 2, the average accuracy (mAcc) of this embodiment reaches 91.23%, and the average intersection-over-union ratio (mIoU) reaches 83.31%, both the highest among all models. The segmentation accuracy for impervious surfaces (Acc 96.56%, IoU 93.12%), buildings (Acc 97.94%, IoU 98.79%), and trees (IoU 91.32%) is significantly higher than other models, and the segmentation accuracy for low vegetation (Acc 87.55%, IoU 76.87%) is also among the top.

[0094] Table 2. Quantitative comparison results of the Vaihingen dataset

[0095]

[0096] In remote sensing segmentation tasks involving various targets such as impervious surfaces, buildings, low vegetation, trees, vehicles, and clutter, the method in this embodiment (Ours) comprehensively outperforms comparable models such as FCN-8s, ACNet, TSNet, ESANet, DCSwin, and DGPRNet. As shown in Table 3, the overall average accuracy (mAcc) of this embodiment reaches 87.53%, and the average intersection-over-union ratio (mIoU) reaches 79.02%, both the highest among all models. Specifically, the vehicle segmentation accuracy is the most superior. Significantly higher (Acc 97.16%, IoU 95.25%), far exceeding other models; the segmentation accuracy of impermeable surfaces (Acc 94.26%, IoU 86.15%), buildings (Acc 94.43%, IoU 92.34%), trees (Acc 88.76%, IoU 76.72%), and low vegetation (Acc 90.67%) are also at the leading level; even for clutter, which is more difficult to segment, its Acc 60.37% and IoU 48.19% are still the best among all models.

[0097] Table 3. Quantitative comparison results in the Potsdam dataset

[0098]

[0099] Figure 3This is a comparison chart of the semantic segmentation results in this embodiment. As can be seen from the visualization, the method in this embodiment (Ours) achieves more continuous target contours and clearer boundaries for categories such as buildings, vegetation, roads, backgrounds, farmland, and drainage systems. It exhibits better connectivity for slender structures (such as roads and drainage ditches), with fewer omissions and clutter. Compared to comparative models such as FCN-8s, ACNet, TSNet, ESANet, DCSwin, and DGPRNet, its segmentation results show a higher degree of agreement with the ground truth (GT) in complex edge regions. This phenomenon is consistent with the quantitative results: on the TV-RSI dataset, the mAcc of this embodiment reaches 90.10%, and the mIoU reaches 82.57%, with IoU for roads and drainage systems reaching 78.82% and 86.12% respectively, both optimal levels among the comparative models. This demonstrates that the present invention has a stable advantage in the fine segmentation of slender and complex terrain boundaries.

[0100] Another embodiment of the present invention provides a Mamba cue learning semantic segmentation system for traditional village remote sensing imagery, comprising:

[0101] Data acquisition module: Acquires high-resolution RGB remote sensing images and corresponding elevation data of traditional village areas, performs geometric correction, radiometric correction and registration processing, and outputs standardized RGB remote sensing images and elevation data;

[0102] Feature extraction module: includes a bimodal teacher feature extraction submodule and a unimodal student feature extraction submodule; the bimodal teacher feature extraction submodule performs multi-scale encoding on RGB remote sensing images and elevation data, and outputs bimodal teacher features; the student feature extraction submodule encodes RGB remote sensing images and outputs student features corresponding to the bimodal teacher feature levels;

[0103] Mamba fusion module: Connects to the bimodal teacher feature extraction submodule, performs four-directional two-dimensional selective scanning, gated feature transformation and residual fusion on the bimodal teacher features, and outputs fused bimodal teacher features;

[0104] Hint learning module: Connects the Mamba fusion module and the student feature extraction submodule, freezes the teacher network parameters, generates hint vectors and performs affine modulation on student features to complete cross-modal knowledge transfer;

[0105] Training and optimization module: Connects to the prompting learning module, constructs a joint loss function, and performs end-to-end optimization of the parameters of the student network and the prompting learning module;

[0106] Inference module: Connects to the student feature extraction submodule, loads the trained student network model, performs forward inference using only RGB remote sensing images as input features, and outputs the land cover category segmentation result;

[0107] Display module: Connected to the inference module, it overlays the segmentation results with geographic coordinates to generate a color raster map, and supports layer switching, transparency adjustment and result export.

[0108] Another embodiment of the present invention provides an electronic device including a memory and a processor. The memory stores computer-readable instructions, which, when executed by the processor, cause the processor to implement the Mamba cue learning semantic segmentation method for remote sensing images described above.

[0109] Another embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described Mamba cue learning semantic segmentation method for remote sensing images.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A Mamba cue learning semantic segmentation method for remote sensing images, characterized in that, Includes the following steps: Step 1: Acquire RGB remote sensing images, preprocess the RGB remote sensing images, construct a remote sensing image dataset, and divide it into a training set, a validation set, and a test set; Step 2: Construct a teacher-student network consisting of a teacher network and a student network. The teacher network uses RGB remote sensing imagery and elevation data as bimodal inputs to generate RGB image features and elevation information features at different levels. A Mamba fusion module is introduced into each level of the teacher network, and the RGB image features and elevation information features are processed by the Mamba fusion module to obtain bimodal teacher features. The student network uses RGB remote sensing imagery as input to obtain student features at each level. The backbone of the student network corresponds one-to-one with the levels of the teacher network. Step 3: Construct the prompting learning module. The student features of each layer are concatenated with the bimodal teacher features through the Mamba module to obtain the student augmentation features. The student augmentation features output from each layer serve as the input to the next layer of the student network. The process of state knowledge transfer implemented by the prompting learning module is represented by the following formula: ； ； in, To enhance students' characteristics, Let r represent the bimodal teacher features of the teacher network at layer i. i Let i represent the student characteristics of the i-th layer of the student network. To stop the gradient operator, For Mamba context gate operators, For layer normalization, It is a linear transformation matrix. It is a 1×1 linear projection matrix. For the Sigmoid function, ⊙ represents element-wise multiplication. The output of the i-th layer of the teacher network after decoding, where T is the temperature coefficient. This is the decoded output of the i-th layer of the student network; Step 4: Multiply and concatenate the predicted image obtained by the decoder processing of the bimodal teacher features output from the last layer of the teacher network and the predicted image obtained by the decoder processing of the student enhancement features output from the last layer of the student network; construct a joint loss function between the concatenated predicted image and the true value to calculate the loss, and perform end-to-end training on the student network and the prompting learning module. Step 5: Call the already trained student network to perform semantic segmentation on the input RGB remote sensing image and generate the segmentation results.

2. The Mamba cue learning semantic segmentation method for remote sensing images according to claim 1, characterized in that, The remote sensing image dataset consists of image blocks and label maps. The image blocks are obtained by sequentially performing geometric correction, radiometric correction, registration processing, region cropping, and scene filtering on RGB remote sensing images. The label maps are obtained by pixel-level annotation of image blocks based on preset land cover categories in RGB remote sensing images.

3. The Mamba cue learning semantic segmentation method for remote sensing images according to claim 1, characterized in that, The Mamba fusion module includes a Mamba module and a residual layer. The Mamba module divides the input features into two paths. One path is processed sequentially through a linear layer, a C3S module, a two-dimensional selective scan module, a linear mapping module, and a layer normalization module to obtain fusion context features. The other path is processed sequentially through a linear layer and a nonlinear activation function, and then multiplied element-wise with the fusion context features to obtain the Mamba context gating operator. The RGB image features and elevation information features are processed by the Mamba module to obtain output features. The residual layer is used to receive the output features and perform weighted superposition and residual fusion on the output features to obtain bimodal teacher features.

4. The Mamba cue learning semantic segmentation method for remote sensing images according to claim 1, characterized in that, The joint loss function is obtained by weighting the cross-entropy loss and the temperature distillation loss according to preset weighting coefficients, thereby minimizing the joint loss.

5. The Mamba cue learning semantic segmentation method for remote sensing images according to claim 1, characterized in that, During the training phase, the parameters of the teacher network are frozen, and the teacher network and student network propagate forward simultaneously. The bimodal teacher features output by the teacher network provide structured guidance for the student network. During the inference phase, the teacher network and elevation data are completely removed, and the student network is retained to independently complete the semantic segmentation task.

6. The Mamba cue learning semantic segmentation method for remote sensing images according to claim 3, characterized in that, The two-dimensional selective scanning performs serialized scanning of features along four directions: horizontal forward, reverse, vertical upward, and downward, and obtains the corresponding directional context state. Then, through channel cascading, linear projection, and layer normalization, the fused context features are obtained. The C3S module performs 3×3 convolution and activation functions on the features. Calculation.

7. A Mamba cue-learning semantic segmentation system for remote sensing images, characterized in that, include: Data acquisition module: Acquires high-resolution RGB remote sensing images and corresponding elevation data of traditional village areas, performs geometric correction, radiometric correction and registration processing, and outputs standardized RGB remote sensing images and elevation data; Feature extraction module: includes a bimodal teacher feature extraction submodule and a unimodal student feature extraction submodule; the bimodal teacher feature extraction submodule performs multi-scale encoding on RGB remote sensing images and elevation data, and outputs bimodal teacher features; the student feature extraction submodule encodes RGB remote sensing images and outputs student features corresponding to the bimodal teacher feature levels; Mamba fusion module: Connects to the bimodal teacher feature extraction submodule, performs four-directional two-dimensional selective scanning, gated feature transformation and residual fusion on the bimodal teacher features, and outputs fused bimodal teacher features; The prompting learning module connects the Mamba fusion module and the student feature extraction submodule. After freezing the teacher network parameters, it generates a prompting vector and performs affine modulation on the student features to complete cross-modal knowledge transfer. The process of state knowledge transfer implemented by the prompting learning module is represented by the following formula: ； ； in, To enhance students' characteristics, Let r represent the bimodal teacher features of the teacher network at layer i. i Let i represent the student characteristics of the i-th layer of the student network. To stop the gradient operator, For Mamba context gate operators, For layer normalization, It is a linear transformation matrix. It is a 1×1 linear projection matrix. For the Sigmoid function, ⊙ represents element-wise multiplication. The output of the i-th layer of the teacher network after decoding, where T is the temperature coefficient. This is the decoded output of the i-th layer of the student network; Training and optimization module: Connects to the prompting learning module, constructs a joint loss function, and performs end-to-end optimization of the parameters of the student network and the prompting learning module; Inference module: Connects to the student feature extraction submodule, loads the trained student network model, performs forward inference using only RGB remote sensing images as input features, and outputs the land cover category segmentation result; Display module: Connected to the inference module, it overlays the segmentation results with geographic coordinates to generate a color raster map, and supports layer switching, transparency adjustment and result export.

8. An electronic device comprising a memory and a processor, wherein the memory stores computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, the processor implements the Mamba cue learning semantic segmentation method for remote sensing images as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the Mamba cue learning semantic segmentation method for remote sensing images as described in any one of claims 1-6.

Images