Remote sensing image semantic segmentation method and system fusing convolutional neural network and visual state space model

Abstract

The invention provides a remote sensing image semantic segmentation method and system fusing a convolutional neural network and a visual state space model, and the method comprises the steps: firstly extracting the multi-scale semantic features of a remote sensing image based on a lightweight ResNet18 encoder; secondly, a decoder based on a visual state space module is used for modeling a long-distance dependency relationship in an image and recovering spatial resolution; the local feature compensation module is used for enhancing the perception capability of fine-grained semantic information and improving the segmentation precision of a small target area; and finally, the multi-scale attention enhancement module is used for fusing deep and shallow layer features and realizing collaborative optimization of spatial details and semantic information. According to the method, the global semantic features and the local detail information of the remote sensing image can be extracted at the same time, and the method has high segmentation precision and a good remote sensing image semantic segmentation effect.

Description

Technical Field

[0001] This invention proposes a method and system for semantic segmentation of remote sensing images that integrates convolutional neural networks and visual state space models, relating to the fields of remote sensing image processing and computer vision. Background Technology

[0002] Semantic segmentation of remote sensing images is one of the fundamental tasks in intelligent analysis of remote sensing images. Its main goal is to semantically classify each pixel in a remote sensing image, providing refined support for applications such as land feature identification, land use, urban planning, and environmental monitoring. With the continuous improvement of remote sensing imaging technology and spatial resolution, high-resolution remote sensing images exhibit characteristics such as mixed foreground and background, large variations in the scale of land features, and blurred boundaries, posing greater challenges to semantic segmentation tasks.

[0003] Currently, mainstream semantic segmentation methods for remote sensing images are mainly based on deep learning techniques, especially convolutional neural networks (CNNs) and visual Transformer structures. CNN-based segmentation models extract local spatial features through multi-layer convolutional operations and use an encoder-decoder structure to achieve feature reconstruction and segmentation output. Representative methods include UNet and the DeepLab series. These methods are effective in preserving local spatial structure, but they have certain limitations in modeling global dependencies.

[0004] Due to its powerful global modeling capabilities, the Transformer architecture has been introduced into remote sensing image semantic segmentation tasks. Typical models such as Segmenter, SegFormer, and Swin-Unet capture long-distance dependencies through self-attention mechanisms, effectively enhancing global semantic perception capabilities. However, the standard Transformer architecture suffers from problems such as a large number of parameters and high computational complexity, limiting its application efficiency in high-resolution remote sensing scenarios.

[0005] In recent years, State Space Models (SSMs) have been proposed as an alternative to self-attention mechanisms, possessing the ability to model long sequences with linear complexity. The Mamba structure, as a representation based on a selective state space model, has demonstrated good performance in natural language processing and vision tasks. Introducing it into remote sensing image semantic segmentation tasks holds promise for simultaneously achieving global modeling capabilities and computational efficiency, making it an important alternative to the Transformer.

[0006] However, most existing Mamba-based segmentation models focus only on global feature modeling, rarely considering the rich local details in high-resolution images. This results in shortcomings in boundary delineation and small object recognition. How to enhance the model's ability to perceive local details while maintaining the advantages of global modeling, and effectively integrate feature information from different levels, is a pressing problem in the field of semantic segmentation of remote sensing images. Summary of the Invention

[0007] In view of this, in order to fill the gaps and deficiencies of the existing technology, this invention proposes a remote sensing image semantic segmentation method and system that integrates convolutional neural networks and visual state space models. This method can overcome the problems of insufficient local detail representation ability and low global modeling efficiency in existing remote sensing image semantic segmentation methods, and effectively improve the model's ability to segment multi-scale targets in complex scenes.

[0008] This invention proposes a method and system for semantic segmentation of remote sensing images that integrates convolutional neural networks and a visual state-space model, including the following:

[0009] This invention proposes a semantic segmentation method for remote sensing images that integrates convolutional neural networks and a visual state-space model, comprising the following steps:

[0010] Step S1: Divide the remote sensing image dataset into a training set and a validation set, and preprocess the images.

[0011] Step S2: Input the encoder built on lightweight ResNet18 to extract multi-scale semantic features of the image;

[0012] Step S3: Input the feature map into the decoder built based on the visual state space module, and use the visual state space module to realize the modeling of long-distance dependency information and the restoration of spatial resolution;

[0013] Step S4: Input the feature map generated during the decoding process into the local feature compensation module and supervise it through the auxiliary loss function;

[0014] Step S5: After fusing the deep semantic features output by the decoder based on the visual state space module with the shallow spatial detail features output by the encoder, the fusion is input into the multi-scale attention enhancement module, and the feature map is refined.

[0015] Step S6: Input the feature map into the segmentation head, and generate a semantic segmentation map with the same size as the original image through 1×1 convolution and upsampling operations; during the training phase, use the loss function to supervise the prediction results and use the AdamW optimizer to update the model parameters; during the testing phase, input the remote sensing image to be segmented into the trained model to obtain a pixel-level semantic prediction map.

[0016] Further, step S1 includes the following:

[0017] Step S11: Establish a semantic segmentation dataset for remote sensing images, which includes remote sensing images with pixel-level semantic annotations; further, divide the semantic segmentation dataset for remote sensing images into a training set and a test set according to a set ratio.

[0018] Step S12: Preprocess the training set and the test set. The preprocessing includes data augmentation operations such as image cropping, scaling, flipping, and rotation. The images of the training set and the test set are converted into tensor form and then normalized.

[0019] Further, step S2 includes the following:

[0020] Step S21: Input the remote sensing image to the ResNet18 encoder, wherein the ResNet18 encoder contains multiple convolutional layers and residual connection structures to extract semantic features of the image at different scales;

[0021] Step S22: The ResNet18 encoder outputs four feature maps with different resolutions, representing spatial detail information and semantic abstraction information from low to high levels, respectively. All four feature maps with different resolutions are used for subsequent layered decoding and fusion processing.

[0022] Further, step S3 includes the following:

[0023] Step S31: Input the remote sensing image to the decoder constructed based on the visual state space module. Except for the first and last layers, each layer of the decoder constructed based on the visual state space module includes one image patch expansion module and two visual state space modules. The decoder constructed based on the visual state space module uses the image patch expansion module to upsample the feature map. Furthermore, the decoder constructed based on the visual state space module utilizes the two-dimensional selective scanning mechanism in the visual state space module to efficiently model global information with linear complexity. The calculation formula includes the following:

[0024] X1=LN(SS2D(SiLU(DWConv(Linear(X in )))))

[0025] X2=SiLU(Linear(X in ))

[0026] X out =Linear(X1⊙X2)

[0027] Among them, SS2D is a two-dimensional selective scanning mechanism, DWConv is a depthwise separable convolution, Linear is a linear layer, LN is a layer normalization process, SiLU is the SiLU activation function, ⊙ represents the Hadamard product, and X1 and X2 represent the output features of the two branches of the visual state space module, respectively.

[0028] Step S32: Upsample and stitch together the multi-layer decoding feature maps step by step to form the final decoding result.

[0029] Further, step S4 includes the following:

[0030] Step S41: The first branch of the local feature compensation module uses a convolutional structure with a smaller receptive field to extract fine-grained local features;

[0031] Step S42: The second branch of the local feature compensation module adopts a convolutional structure with a large receptive field to capture contextual information within the local region;

[0032] Step S43: The outputs of the first and second branches of the local feature compensation module are processed by batch normalization and ReLU activation function respectively, and then weighted and fused. During fusion, a learnable weight parameter is used to adjust the features of the two branches.

[0033] Step S44: Fuse features as auxiliary output, connect to auxiliary branch and calculate auxiliary loss function. The auxiliary loss function is cross-entropy loss, which is used to guide the local feature compensation module to learn the edge and detail information of the target region during the training phase.

[0034] Step S45: The local feature compensation module and its auxiliary loss supervision are only enabled during the model training phase and are not used during the inference phase, thus not increasing the inference computation overhead.

[0035] Further, step S5 includes the following:

[0036] Step S51: After weighted fusion of deep semantic features and shallow spatial features, the result is used as the input feature map for this module;

[0037] Step S52: The first branch of the multi-scale attention enhancement module performs adaptive average pooling along the horizontal and vertical directions respectively. The pooling results in the two directions are concatenated in the spatial dimension and then fused by 1×1 convolution to generate a spatial guided feature map.

[0038] Step S53: The second branch of the multi-scale attention enhancement module performs a 3×3 convolution operation on the input feature map to extract local spatial structure information and generate a spatial structure feature map;

[0039] Step S54: The outputs of the first and second branches of the multi-scale attention enhancement module are processed by global average pooling and Softmax respectively, and then element-wise dot product is performed to obtain the attention map;

[0040] Step S55: After the attention map is normalized by Sigmoid, it is fused with the input feature map by element-wise multiplication to obtain the output feature map, thereby achieving the collaborative optimization of spatial details and semantic information.

[0041] Further, step S6 includes the following:

[0042] Step S61: After the remote sensing images in the semantic segmentation dataset are processed by the ResNet18 encoder and the decoder built based on the visual state space module, the final feature map is obtained. The feature map is then input into a 1×1 convolutional layer for channel compression. After compression, the number of channels is equal to the number of classes C. This operation converts the channel vector of each pixel in the feature map into a class probability vector through a pixel-wise linear mapping, and generates a semantic segmentation map with the same size as the input image space.

[0043] Step S62: During model training, a combination of the cross-entropy loss function and the Dice loss function is used as the main loss function, and the cross-entropy loss function is used as the auxiliary loss function. The expressions for the two loss functions are as follows:

[0044]

[0045] Among them, L ce and L dice These represent the cross-entropy loss and Dice loss, respectively, where N is the total number of samples and K is the total number of classes. This represents the k-th element in the one-hot encoding of the true label of the n-th sample. The model predicts the confidence level that the nth sample belongs to the kth class.

[0046] Furthermore, step S6 also includes the following:

[0047] Step S63: During the optimization process, the AdamW optimizer is used to update the model parameters; the AdamW optimizer introduces an independent weight decay mechanism, which helps to improve the stability and generalization ability of the model during training; during training, the model gradually adjusts the parameters according to the feedback of the loss function to optimize feature extraction and prediction capabilities.

[0048] Step S64: In the inference stage, the remote sensing image to be tested is input into the trained model. The model outputs a semantic segmentation map through forward propagation. Each pixel in the map corresponds to a predicted category, thereby realizing the fine recognition and localization of various land features in the remote sensing image.

[0049] According to a second aspect of the present invention, a remote sensing image semantic segmentation system integrating a convolutional neural network and a visual state space model is provided, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a remote sensing image semantic segmentation method integrating a convolutional neural network and a visual state space model as described in any one of the present invention.

[0050] According to a third aspect of the present invention, a remote sensing image semantic segmentation system integrating a convolutional neural network and a visual state space model is provided, comprising a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements a remote sensing image semantic segmentation method integrating a convolutional neural network and a visual state space model as described in any one of the present invention.

[0051] The present invention has the following advantages:

[0052] (1) This invention combines the local modeling capability of convolutional neural networks with the global modeling capability of state space models. By introducing a visual state space module, it effectively models the long-distance dependency relationship between pixels in remote sensing images, reduces the computational burden brought by global modeling in the traditional Transformer structure, and improves the overall inference efficiency.

[0053] (2) By setting up a local feature compensation module, without increasing the computational load in the inference stage, the auxiliary supervision mechanism guides the model to learn fine-grained semantic information, which enhances the model's ability to express small targets and edge regions and improves the detail depiction effect.

[0054] (3) The multi-scale attention enhancement module effectively integrates the features between deep and shallow layers, which alleviates the semantic bias problem caused by direct splicing or addition and improves the boundary recognition accuracy of the target region.

[0055] (4) The remote sensing image semantic segmentation method proposed in this invention has a clear structure and independent modules, and has good adaptability and scalability. It is suitable for semantic segmentation tasks in a variety of complex remote sensing scenarios.

[0056] (5) Compared with existing mainstream methods, this invention reduces model complexity while maintaining model accuracy, and has high computational efficiency and cross-scene generalization ability. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating the steps of a remote sensing image semantic segmentation method and system that integrates convolutional neural networks and a visual state space model, according to the present invention.

[0058] Figure 2 This is a diagram illustrating the overall structure of a remote sensing image semantic segmentation method and system that integrates convolutional neural networks and a visual state space model, according to the present invention.

[0059] Figure 3 This is a structural diagram of the visual state space module of the present invention.

[0060] Figure 4 This is a structural diagram of the local feature compensation module of the present invention.

[0061] Figure 5 This is a structural diagram of the multi-scale attention enhancement module of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0063] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0064] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the present invention; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0065] like Figure 1 and Figure 2 As shown, this invention proposes a method and system for semantic segmentation of remote sensing images that integrates convolutional neural networks and a visual state-space model, including the following:

[0066] This invention proposes a semantic segmentation method for remote sensing images that integrates convolutional neural networks and a visual state-space model, characterized by the following steps:

[0067] Step S1: Divide the remote sensing image dataset into a training set and a validation set, and preprocess the images.

[0068] Step S2: Input the encoder built on lightweight ResNet18 to extract multi-scale semantic features of the image;

[0069] Step S3: Input the feature map into the decoder built based on the visual state space module, and use the visual state space module to realize the modeling of long-distance dependency information and the restoration of spatial resolution;

[0070] Step S4: Input the feature map generated during the decoding process into the local feature compensation module and supervise it through the auxiliary loss function;

[0071] Step S5: After fusing the deep semantic features output by the decoder based on the visual state space module with the shallow spatial detail features output by the encoder, the fusion is input into the multi-scale attention enhancement module, and the feature map is refined.

[0072] Step S6: Input the feature map into the segmentation head, and generate a semantic segmentation map with the same size as the original image through 1×1 convolution and upsampling operations; during the training phase, use the loss function to supervise the prediction results and use the AdamW optimizer to update the model parameters; during the testing phase, input the remote sensing image to be segmented into the trained model to obtain a pixel-level semantic prediction map.

[0073] In one embodiment of the present invention, step S1 includes the following:

[0074] Step S11: Establish a semantic segmentation dataset for remote sensing images, which includes remote sensing images with pixel-level semantic annotations; further, divide the semantic segmentation dataset for remote sensing images into a training set and a test set according to a set ratio.

[0075] Step S12: Preprocess the training set and the test set. The preprocessing includes data augmentation operations such as image cropping, scaling, flipping, and rotation. The images of the training set and the test set are converted into tensor form and then normalized.

[0076] In one embodiment of the present invention, step S2 includes the following:

[0077] Step S21: Input the remote sensing image to the ResNet18 encoder, wherein the ResNet18 encoder contains multiple convolutional layers and residual connection structures to extract semantic features of the image at different scales;

[0078] Step S22: The ResNet18 encoder outputs four feature maps with different resolutions, representing spatial detail information and semantic abstraction information from low to high levels, respectively. All four feature maps with different resolutions are used for subsequent layered decoding and fusion processing.

[0079] In one embodiment of the present invention, step S3 includes the following:

[0080] Step S31: Input the remote sensing image to the decoder constructed based on the visual state space module. Except for the first and last layers, each layer of the decoder constructed based on the visual state space module includes one image patch expansion module and two visual state space modules. The decoder constructed based on the visual state space module uses the image patch expansion module to upsample the feature map. Furthermore, the decoder constructed based on the visual state space module utilizes the two-dimensional selective scanning mechanism in the visual state space module to efficiently model global information with linear complexity. The calculation formula includes the following:

[0081] X1=LN(SS2D(SiLU(DWConv(Linear(X in )))))

[0082] X2=SiLU(Linear(X in ))

[0083] X out =Linear(X1⊙X2)

[0084] Among them, SS2D is a two-dimensional selective scanning mechanism, DWConv is a depthwise separable convolution, Linear is a linear layer, LN is a layer normalization process, SiLU is the SiLU activation function, ⊙ represents the Hadamard product, and X1 and X2 represent the output features of the two branches of the visual state space module, respectively.

[0085] Step S32: Upsample and stitch together the multi-layer decoding feature maps step by step to form the final decoding result.

[0086] In one embodiment of the present invention, step S4 includes the following:

[0087] Step S41: The first branch of the local feature compensation module uses a convolutional structure with a smaller receptive field to extract fine-grained local features;

[0088] Step S42: The second branch of the local feature compensation module adopts a convolutional structure with a large receptive field to capture contextual information within the local region;

[0089] Step S43: The outputs of the first and second branches of the local feature compensation module are processed by batch normalization and ReLU activation function respectively, and then weighted and fused. During fusion, a learnable weight parameter is used to adjust the features of the two branches.

[0090] Step S44: Fuse features as auxiliary output, connect to auxiliary branch and calculate auxiliary loss function. The auxiliary loss function is cross-entropy loss, which is used to guide the local feature compensation module to learn the edge and detail information of the target region during the training phase.

[0091] Step S45: The local feature compensation module and its auxiliary loss supervision are only enabled during the model training phase and are not used during the inference phase, thus not increasing the inference computation overhead.

[0092] In one embodiment of the present invention, step S5 includes the following:

[0093] Step S51: After weighted fusion of deep semantic features and shallow spatial features, the result is used as the input feature map for this module;

[0094] Step S52: The first branch of the multi-scale attention enhancement module performs adaptive average pooling along the horizontal and vertical directions respectively. The pooling results in the two directions are concatenated in the spatial dimension and then fused by 1×1 convolution to generate a spatial guided feature map.

[0095] Step S53: The second branch of the multi-scale attention enhancement module performs a 3×3 convolution operation on the input feature map to extract local spatial structure information and generate a spatial structure feature map;

[0096] Step S54: The outputs of the first and second branches of the multi-scale attention enhancement module are processed by global average pooling and Softmax respectively, and then element-wise dot product is performed to obtain the attention map;

[0097] Step S55: After the attention map is normalized by Sigmoid, it is fused with the input feature map by element-wise multiplication to obtain the output feature map, thereby achieving the collaborative optimization of spatial details and semantic information.

[0098] In one embodiment of the present invention, step S6 includes the following:

[0099] Step S61: After the remote sensing images in the semantic segmentation dataset are processed by the ResNet18 encoder and the decoder built based on the visual state space module, the final feature map is obtained. The feature map is then input into a 1×1 convolutional layer for channel compression. After compression, the number of channels is equal to the number of classes C. This operation converts the channel vector of each pixel in the feature map into a class probability vector through a pixel-wise linear mapping, and generates a semantic segmentation map with the same size as the input image space.

[0100] Step S62: During model training, a combination of the cross-entropy loss function and the Dice loss function is used as the main loss function, and the cross-entropy loss function is used as the auxiliary loss function. The expressions for the two loss functions are as follows:

[0101]

[0102] Among them, L ce and L diceThese represent the cross-entropy loss and Dice loss, respectively, where N is the total number of samples and K is the total number of classes. This represents the k-th element in the one-hot encoding of the true label of the n-th sample. The model predicts the confidence level that the nth sample belongs to the kth class.

[0103] In one embodiment of the present invention, step S6 further includes the following:

[0104] Step S63: During the optimization process, the AdamW optimizer is used to update the model parameters; the AdamW optimizer introduces an independent weight decay mechanism, which helps to improve the stability and generalization ability of the model during training; during training, the model gradually adjusts the parameters according to the feedback of the loss function to optimize feature extraction and prediction capabilities.

[0105] Step S64: In the inference stage, the remote sensing image to be tested is input into the trained model. The model outputs a semantic segmentation map through forward propagation. Each pixel in the map corresponds to a predicted category, thereby realizing the fine recognition and localization of various land features in the remote sensing image.

[0106] In one embodiment of the present invention, a remote sensing image semantic segmentation system integrating convolutional neural networks and a visual state space model is proposed, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements a remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state space model as described in any one of the present invention.

[0107] In one embodiment of the present invention, a remote sensing image semantic segmentation system integrating convolutional neural networks and a visual state space model is proposed, comprising a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements a remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state space model as described in any one of the present invention.

[0108] In addition to the above, the present invention also includes the following embodiments:

[0109] In one embodiment of the present invention, step S1 further includes the following:

[0110] Step S11: The remote sensing image dataset used is the Potsdam dataset, which consists of 38 ultra-high resolution orthophotos, each with a size of 6000×6000 pixels. Each image contains five foreground categories: impermeable surfaces, buildings, low vegetation, trees, and vehicles. The experiment selected images with IDs 2_13, 2_14, 3_13, 3_14, 4_13, 4_14, 4_15, 5_13, 5_14, 5_15, 6_13, 6_14, 6_15, and 7_13 as the validation set (14 images in total), and the remaining 24 images as the training set.

[0111] Step S12: The input image is randomly cropped into 512×512 pixel patches, and data augmentation strategies such as multi-scale random scaling (scaling factors of 0.5, 0.75, 1.0, 1.25, 1.5), vertical / horizontal flipping, and rotation are applied. The image is then converted into a tensor and finally normalized.

[0112] In one embodiment of the present invention, a Visual State Space Block (VSSBlock) module is introduced in step S3 to enhance the global modeling capability of input image features. Compared with the Transformer, the Visual State Space Block (VSSBlock) has better computational efficiency and memory utilization while modeling global dependencies, making it more suitable for deployment in high-resolution vision tasks. The Visual State Space Block includes the following components:

[0113] The first layer normalization (LayerNorm) normalizes the input feature map, improving the stability of the feature distribution and providing a better numerical basis for the training of subsequent sub-modules.

[0114] Dual-branch structure: The visual state space module divides the normalized feature map into two parallel branches for processing:

[0115] The main branches include the following:

[0116] Linear layer: Expands or maps the channel dimension;

[0117] Depthwise separable convolution (DWConv): used to extract local spatial features while maintaining computational efficiency;

[0118] SiLU activation function: enhances nonlinear expressive power;

[0119] SS2D module (2D Selective Scan module): used to simulate the state-space modeling process, introduce long-distance dependencies, and improve the model's ability to perceive global context information;

[0120] Second-layer normalization (LayerNorm): Further stabilizes the feature distribution and improves the fusion effect.

[0121] The auxiliary branches include the following:

[0122] Linear mapping + SiLU activation function: provides global channel information after lightweight feature transformation as an enhancement path.

[0123] Fusion Module: The output features of the two branches are fused using Hadamard, effectively enhancing key response regions in the main features. The fused features are then mapped through a linear layer to restore the number of channels to the input dimension.

[0124] Residual connections: The module output is connected through residuals that are directly added to the input features, achieving gradient stabilization and feature compensation, which helps with deep network training and information retention.

[0125] In one embodiment of the present invention, step S3, which involves progressively upsampling and concatenating the multi-layer decoding feature maps, includes the following:

[0126] Step S3, which involves progressively upsampling and concatenating multi-layer decoded feature maps, employs a strategy of layer-by-layer upsampling and feature fusion. Specifically, the model progressively restores the spatial dimensions of the deep feature maps extracted by the encoder through upsampling operations. Simultaneously, the model concatenates the upsampled feature maps of each layer with the corresponding shallow feature maps from the encoder along the channel dimension. The core purpose of this is to combine low-level spatial detail information with high-level semantic abstraction information, thereby enhancing the model's expressive ability in details such as segmentation boundaries or small targets.

[0127] In this invention, the multi-layer decoded feature map is obtained by upsampling layer by layer and fusing the corresponding layer feature maps of the encoder during the decoder stage of the model. Each time it is upsampled, it is concatenated with the corresponding layer feature map of the encoder and then convolved to obtain a new decoded feature map. The decoder of this invention generates multiple decoded feature maps step by step through such a multi-level structure, and finally completes image restoration and segmentation prediction.

[0128] In one embodiment of the present invention, such as Figure 3 As shown, the visual state space module structure diagram of the present invention includes the following:

[0129] Input feature map After layer normalization, the inputs proceed in parallel to two branches. The first branch expands the channels to λC (λ is a preset channel expansion factor) through a linear layer, then extracts features via depthwise separable convolution, SiLU activation function, and 2D selective scan (SS2D) module, followed by layer normalization for standardization. The second branch performs feature transformation through a linear layer and SiLU activation function. The features output from both branches are fused using Hadamard product, and then linear projection is used to restore the number of channels to C, ultimately generating an output with the same dimension as the input.

[0130] In one embodiment of the present invention, such as Figure 4 As shown, the structure diagram of the Local Feature Compensation Module (LFCM) of the present invention includes the input feature map first entering two branches in parallel, including the following:

[0131] The first branch of the multi-scale attention enhancement module first transforms the channel dimension through a 1×1 convolution, and then sequentially passes through batch normalization and ReLU6 activation function to perform lightweight channel compression and non-linear enhancement on the input features.

[0132] The second branch of the multi-scale attention enhancement module uses a 3×3 convolutional kernel to extract local spatial information, and also incorporates batch normalization and ReLU6 activation function to enhance feature representation.

[0133] Subsequently, the output features of the two branches of the multi-scale attention enhancement module are fused in a weighted manner to improve the collaborative expressive ability of receptive field features at different scales. The fused features are then processed by the Dropout module to suppress overfitting and further compressed by 1×1 convolution to compress the channel dimension. Finally, the spatial resolution is improved by upsampling to generate the final output feature map for subsequent segmentation or decoding operations.

[0134] In one embodiment of the present invention, such as Figure 5 As shown, the multi-scale attention enhancement module (MAEM) of the present invention includes weighted fusion of deep features output by the decoder and shallow features output by the encoder to integrate spatial details and semantic information, and uses the fused result (Fused Feature) as input to MAEM.

[0135] Furthermore, for the input feature map, MAEM employs two parallel branches for processing, focusing on channel dependence and spatial context information respectively. One branch performs adaptive average pooling operations along both the horizontal and vertical directions, effectively capturing channel response features in different spatial dimensions. The pooling results from both directions are then concatenated spatially and fused using a 1×1 convolution to obtain spatial guidance features used to guide channel attention distribution. The other branch uses a 3×3 convolution to locally model the original features, enhancing spatial structure perception.

[0136] In one embodiment of the present invention, step S62 during model training includes the following:

[0137] A joint supervision mechanism combining a main loss function and an auxiliary loss function is employed. The main loss function, used to optimize the final output, is the primary objective function for model training. In this scheme, the main loss function consists of cross-entropy loss and Dice loss to improve the model's robustness to issues such as class boundaries and class imbalance. In contrast, the auxiliary loss function operates on intermediate layers or assists the output, guiding the model to converge more stably in the early or middle stages of training, improving the separability of deep features, and preventing gradient vanishing or information weakening.

[0138] Furthermore, the auxiliary loss function is introduced only during the model training phase to assist in supervising intermediate features and enhance gradient propagation and training stability. During the validation or testing phase, the model retains only the main branch for forward inference; the auxiliary branch does not participate in computation and does not affect the final output. This design helps improve training performance while ensuring the efficiency and simplicity of the inference process.

[0139] In one embodiment of the present invention, the optimization process of step S63 includes the following:

[0140] The optimization process refers to the process of continuously adjusting the learnable parameters (such as weights and biases) in the network during model training by using the error signal provided by the loss function and the backpropagation algorithm, in order to minimize the loss and improve performance.

[0141] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for semantic segmentation of remote sensing images that integrates convolutional neural networks and a visual state-space model, characterized in that, Includes the following steps: Step S1: Divide the remote sensing image dataset into a training set and a validation set, and preprocess the images. Step S2: Input the encoder built on lightweight ResNet18 to extract multi-scale semantic features of the image; Step S3: Input the feature map into the decoder built based on the visual state space module, and use the visual state space module to realize the modeling of long-distance dependency information and the restoration of spatial resolution; Step S4: Input the feature map generated during the decoding process into the local feature compensation module and supervise it through the auxiliary loss function; Step S5: After fusing the deep semantic features output by the decoder based on the visual state space module with the shallow spatial detail features output by the encoder, the fusion is input into the multi-scale attention enhancement module, and the feature map is refined. Step S6: Input the feature map into the segmentation head, and generate a semantic segmentation map with the same size as the original image through 1×1 convolution and upsampling operations; during the training phase, use the loss function to supervise the prediction results and use the AdamW optimizer to update the model parameters; during the testing phase, input the remote sensing image to be segmented into the trained model to obtain a pixel-level semantic prediction map.

2. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 1, characterized in that, Step S1 includes the following: Step S11: Establish a semantic segmentation dataset for remote sensing images, which includes remote sensing images with pixel-level semantic annotations; further, divide the semantic segmentation dataset for remote sensing images into a training set and a test set according to a set ratio. Step S12: Preprocess the training set and the test set. The preprocessing includes data augmentation operations such as image cropping, scaling, flipping, and rotation. The images of the training set and the test set are converted into tensor form and then normalized.

3. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 1, characterized in that, Step S2 includes the following: Step S21: Input the remote sensing image to the ResNet18 encoder, wherein the ResNet18 encoder contains multiple convolutional layers and residual connection structures to extract semantic features of the image at different scales; Step S22: The ResNet18 encoder outputs four feature maps with different resolutions, representing spatial detail information and semantic abstraction information from low to high levels, respectively. All four feature maps with different resolutions are used for subsequent layered decoding and fusion processing.

4. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 1, characterized in that, Step S3 includes the following: Step S31: Input the remote sensing image to the decoder constructed based on the visual state space module. Except for the first and last layers, each layer of the decoder constructed based on the visual state space module includes one image patch expansion module and two visual state space modules. The decoder constructed based on the visual state space module uses the image patch expansion module to upsample the feature map. Furthermore, the decoder constructed based on the visual state space module utilizes the two-dimensional selective scanning mechanism in the visual state space module to efficiently model global information with linear complexity. The calculation formula includes the following: X1=LN(SS2D(SiLU(DWConv(Linear(X in ))))) X2=SiLU(Linear(X in )) X out =Linear(X1⊙X2) Among them, SS2D is a two-dimensional selective scanning mechanism, DWConv is a depthwise separable convolution, Linear is a linear layer, LN is a layer normalization process, SiLU is the SiLU activation function, ⊙ represents the Hadamard product, and X1 and X2 represent the output features of the two branches of the visual state space module, respectively. Step S32: Upsample and stitch together the multi-layer decoding feature maps step by step to form the final decoding result.

5. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 1, characterized in that, Step S4 includes the following: Step S41: The first branch of the local feature compensation module uses a convolutional structure with a smaller receptive field to extract fine-grained local features; Step S42: The second branch of the local feature compensation module adopts a convolutional structure with a large receptive field to capture contextual information within the local region; Step S43: The outputs of the first and second branches of the local feature compensation module are processed by batch normalization and ReLU activation function respectively, and then weighted and fused. During fusion, a learnable weight parameter is used to adjust the features of the two branches. Step S44: Fuse features as auxiliary output, connect to auxiliary branch and calculate auxiliary loss function. The auxiliary loss function is cross-entropy loss, which is used to guide the local feature compensation module to learn the edge and detail information of the target region during the training phase. Step S45: The local feature compensation module and its auxiliary loss supervision are only enabled during the model training phase and are not used during the inference phase, thus not increasing the inference computation overhead.

6. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 1, characterized in that, Step S5 includes the following: Step S51: After weighted fusion of deep semantic features and shallow spatial features, the result is used as the input feature map for this module; Step S52: The first branch of the multi-scale attention enhancement module performs adaptive average pooling along the horizontal and vertical directions respectively. The pooling results in the two directions are concatenated in the spatial dimension and then fused by 1×1 convolution to generate a spatial guided feature map. Step S53: The second branch of the multi-scale attention enhancement module performs a 3×3 convolution operation on the input feature map to extract local spatial structure information and generate a spatial structure feature map; Step S54: The outputs of the first and second branches of the multi-scale attention enhancement module are processed by global average pooling and Softmax respectively, and then element-wise dot product is performed to obtain the attention map; Step S55: After the attention map is normalized by Sigmoid, it is fused with the input feature map by element-wise multiplication to obtain the output feature map, thereby achieving the collaborative optimization of spatial details and semantic information.

7. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 1, characterized in that, Step S6 includes the following: Step S61: After the remote sensing images in the semantic segmentation dataset are processed by the ResNet18 encoder and the decoder built based on the visual state space module, the final feature map is obtained. The feature map is then input into a 1×1 convolutional layer for channel compression. After compression, the number of channels is equal to the number of classes C. This operation converts the channel vector of each pixel in the feature map into a class probability vector through a pixel-wise linear mapping, and generates a semantic segmentation map with the same size as the input image space. Step S62: During model training, a combination of cross-entropy loss function and Dice loss function is used as the main loss function, and cross-entropy loss function is used as the auxiliary loss function; the expressions for the two loss functions are as follows: Among them, L ce and L dice These represent the cross-entropy loss and Dice loss, respectively, where N is the total number of samples and K is the total number of classes. This represents the k-th element in the one-hot encoding of the true label of the n-th sample. The model predicts the confidence level that the nth sample belongs to the kth class.

8. The remote sensing image semantic segmentation method integrating convolutional neural networks and a visual state-space model according to claim 7, characterized in that, Step S6 also includes the following: Step S63: During the optimization process, the AdamW optimizer is used to update the model parameters; the AdamW optimizer introduces an independent weight decay mechanism, which helps to improve the stability and generalization ability of the model during training. During training, the model gradually adjusts its parameters based on feedback from the loss function to optimize its feature extraction and prediction capabilities. Step S64: In the inference stage, the remote sensing image to be tested is input into the trained model. The model outputs a semantic segmentation map through forward propagation. Each pixel in the map corresponds to a predicted category, thereby realizing the fine recognition and localization of various land features in the remote sensing image.

9. A remote sensing image semantic segmentation system integrating convolutional neural networks and a visual state-space model, comprising an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a remote sensing image semantic segmentation method that integrates convolutional neural networks and a visual state space model as described in any one of claims 1 to 8.

10. A remote sensing image semantic segmentation system integrating convolutional neural networks and a visual state-space model, comprising a computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a remote sensing image semantic segmentation method that integrates a convolutional neural network and a visual state space model as described in any one of claims 1 to 8.

Images