Semantic segmentation model and segmentation method for high-resolution remote sensing image

Abstract

The invention discloses a high-resolution remote sensing image semantic segmentation model and segmentation method, and relates to the technical field of remote sensing image processing, and the model comprises a feature representation and preliminary fusion coding module, a deep feature processing and fusion coding module, and a decoding and feature reconstruction module. A self-adaptive weighted fusion module is introduced into the feature representation and preliminary fusion coding module, and effective fusion of multi-modal data is realized by dynamically adjusting weights of different modal features; the deep feature processing and fusion coding module realizes gradual enhancement of features and deep fusion of cross-modal information through a multi-level sparse attention mechanism and a gating fusion mechanism combined with attention; and the decoding and feature reconstruction module recovers the low-resolution feature map generated by the encoder to the high resolution of the input image through step-by-step up-sampling and multi-scale feature fusion. The method is superior to the prior art in the aspects of calculation efficiency, segmentation precision and robustness for processing complex scenes.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and in particular to a high-resolution remote sensing image semantic segmentation model and a segmentation method. Background Art

[0002] In the field of remote sensing image processing, semantic segmentation technology of high-resolution remote sensing images has broad application prospects, especially in urban planning, land use analysis and natural resource monitoring. However, existing remote sensing image semantic segmentation methods are mainly based on single-modality data, usually including optical images or radar data. These methods are prone to insufficient segmentation accuracy when faced with complex environments, different weather conditions or terrain features.

[0003] Multimodal fusion technology can provide richer environmental information and improve the accuracy of remote sensing image segmentation by combining data from different sensors, such as optical images, radar images, and lidar data. Existing multimodal fusion algorithms, such as feature fusion methods based on deep learning, have shown great advantages.

[0004] However, existing semantic segmentation methods for high-resolution remote sensing images have deficiencies in multimodal fusion, noise suppression, and long-distance dependency modeling, making it difficult to cope with complex scenes, especially when processing high-resolution images, with high computational cost and low segmentation accuracy. The main problems include: Insufficient feature fusion: Multimodal fusion methods cannot balance the features of different modalities, resulting in insufficient expression of key information and affecting the segmentation effect. Weak noise suppression: In the process of multimodal data fusion, existing methods are sensitive to data noise and cannot fully suppress irrelevant information, resulting in unsatisfactory model segmentation effects. Difficulty in modeling long-distance dependencies: Existing methods cannot effectively capture long-distance dependencies, especially in terms of balancing global features with local details, resulting in decreased segmentation accuracy for small-scale targets. High computational complexity: Although traditional models based on self-attention mechanisms can capture global information, they consume significant computing resources when processing high-resolution remote sensing images, making it difficult to achieve efficient computing while ensuring segmentation accuracy. Summary of the invention

[0005] In order to overcome the above problems existing in the prior art, the present invention proposes a high-resolution remote sensing image semantic segmentation model and a segmentation method.

[0006] The technical solution adopted by the present invention to solve its technical problems is: a high-resolution remote sensing image semantic segmentation model, including a feature representation and preliminary fusion coding module, a deep feature processing and fusion coding module, and a decoding and feature reconstruction module, wherein the feature representation and preliminary fusion coding module is used to extract preliminary features from multimodal input data to achieve preliminary fusion; the deep feature processing and fusion coding module is based on a multi-layer encoder of a Transformer architecture, and realizes gradual enhancement of features and deep fusion of cross-modal information through multiple levels of sparse attention mechanisms and a gated fusion mechanism combined with attention; the decoding and feature reconstruction module restores the low-resolution feature map generated by the encoder to the high resolution of the input image through gradual upsampling and multi-scale feature fusion.

[0007] The above-mentioned high-resolution remote sensing image semantic segmentation model, the feature representation and preliminary fusion coding module includes a VIS encoder and a DSM encoder, which are used to process VIS images and DSM data respectively, and feature extraction is performed through a ResNet structure. Each branch uses a standard two-dimensional convolutional layer for convolution operations, and uses multiple residual blocks to achieve feature extraction and enhancement. Each residual block contains a standard convolutional layer, a ReLU activation layer and a group normalization layer. In the feature extraction process, both the VIS encoder and the DSM encoder introduce an adaptive weighted fusion module.

[0008] In the above-mentioned high-resolution remote sensing image semantic segmentation model, the adaptive weighted fusion module consists of two branches for processing features from DSM data and VIS data. In each branch, the input features are globally averaged and pooled to capture global context information; the features are compressed through a 1×1 convolution layer and a ReLU activation function; the feature representation is further adjusted through a depth-separable 1×1 convolution layer; the output is normalized by a Sigmoid function to generate adaptive weights; after obtaining the feature weights of two different modalities, the two feature weights are multiplied by their respective features to generate a weighted feature representation, and the features of the two modalities are weighted and fused through an adaptive weight mechanism. The fused output is: ; in, and Represent the adaptive weights of depth features and RGB features respectively.

[0009] In the above-mentioned high-resolution remote sensing image semantic segmentation model, the multi-layer encoder of the Transformer architecture consists of 12 layers, including a gated fusion layer and a sparse attention layer combined with attention, wherein the first three layers are sparse attention layers for selectively activating key feature points; the next three layers are gated fusion layers, which combine the attention mechanism for deep fusion of features; the next three layers are sparse attention layers, which further optimize the capture of global features; the last three layers are gated fusion layers, which further enhance the fusion and selectivity of features.

[0010] In the above-mentioned high-resolution remote sensing image semantic segmentation model, the sparse attention module in the sparse attention layer is specifically: Input feature vector The query matrix Q, key matrix K and value matrix V are generated through three linear layers respectively. The specific calculation formula is: ; in, , , are the weight matrices for query, key, and value, respectively; Calculate the attention score based on the query matrix Q and the key matrix K , the specific calculation formula is: ; in, is the scaling factor, The dimension of the key.

[0011] Select the largest values ​​of the first k% in the attention matrix and set other smaller values ​​to negative infinity to achieve sparseness. The sparse attention matrix can be expressed as: ; The threshold Threshold is determined according to the maximum values ​​of the first k%.

[0012] The sparse attention matrix is ​​soft-maximized to obtain the weight distribution, which is applied to the value matrix V, and the output features are obtained by weighted summation. The weighted summed features are projected through a linear layer to obtain the final output.

[0013] In the above-mentioned high-resolution remote sensing image semantic segmentation model, the gated fusion layer combined with attention is specifically: For input features , Perform self-attention calculations respectively to generate enhanced feature representations attentionsy and attentionsx. The formula is as follows: attentionsx and attentionsy generate gating weights through their respective gating networks The gating network consists of a linear layer and a Sigmoid activation function, which is used to control the fusion ratio of the two features. The generation process of the gating weight is as follows: gate_x1=σ( *attentionsx) gate_x2=σ( *attentionsx) gate_y1=σ( *attentionsy) gate_y2=σ( *attentionsy) in, is the weight of the linear layer σ represents the Sigmoid function, which ensures that the generated weight value is in the range of [0,1].

[0014] According to the gate weight , perform weighted fusion of the two features attentionsx and attentionsy, and the fusion calculation process is as follows: fused_x=gate_x1*attentionsx+gate_x2*attentionsy fused_y=gate_y1*attentionsy+gate_y2*attentionsx Among them, fused_x represents the fused feature x, and fused_y represents the fused feature y.

[0015] A high-resolution remote sensing image semantic segmentation method, the above-mentioned high-resolution remote sensing image semantic segmentation model specifically includes the following steps: Step 1, obtaining a remote sensing image to be segmented; Step 2, inputting the remote sensing image to be segmented into a pre-established and trained high-resolution remote sensing image semantic segmentation model, extracting preliminary features through a feature representation and preliminary fusion coding module, and achieving preliminary fusion; Step 3: The image obtained in step 2 is processed by deep feature processing and fusion coding module, and the gradual enhancement of features and the deep fusion of cross-modal information are achieved through multiple levels of sparse attention mechanism and gated fusion mechanism combined with attention; Step 4: The image obtained in step 3 is processed by the decoding and feature reconstruction module. Through step-by-step upsampling and multi-scale feature fusion, the low-resolution feature map generated by the encoder is restored to the high resolution of the input image, and finally a high-precision semantic segmentation image is obtained.

[0016] The beneficial effect of the present invention is that the model of the present invention introduces an adaptive weighted fusion (AWF) module in the shallow feature extraction stage, and realizes the effective fusion of multimodal data by dynamically adjusting the weights of different modal features. After completing the shallow extraction and fusion, the deep feature extraction stage is entered. The model uses sparse attention instead of the traditional self-attention mechanism, focusing on key feature points and reducing unnecessary calculations. Subsequently, the model introduces a gating mechanism combined with attention to perform fine fusion of deep features. During the decoding process, global semantic information and detail features are effectively retained through jump connections and layer-by-layer upsampling, and fine recovery of spatial and semantic information is achieved. Overall, the present invention is superior to the prior art in terms of computational efficiency, segmentation accuracy, and robustness in processing complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the overall framework diagram of MSGT-Net of the present invention; Figure 2 This is a module diagram of the adaptive weighted fusion (AWF) of the present invention; Figure 3 This is a framework diagram of the sparse attention layer of the present invention; Figure 4 This is a schematic diagram of the sparse attention module of the present invention; Figure 5 This is a framework diagram of the gated fusion layer of the present invention; Figure 6 This is a schematic diagram of the gated fusion module combined with attention in the present invention; Figure 7 A segmentation comparison diagram of the model constructed by the present invention and the prior art model on the Vaihingen dataset; Figure 8 This is a comparison chart of ablation experiments between the model constructed in the present invention and the prior art model. DETAILED DESCRIPTION

[0018] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0019] This embodiment discloses a high-resolution remote sensing image semantic segmentation model, namely MSGT-Net. Figure 1As shown, the overall framework of MSGT-Net proposed in this embodiment adopts a hierarchical feature extraction and fusion strategy, which gradually extracts and fuses the features of multimodal input data (such as VIS images and DSM data) through multiple modules, and finally generates high-precision semantic segmentation results. It is mainly composed of a feature representation and preliminary fusion coding module, a deep feature processing and fusion coding module, and a decoding and feature reconstruction module.

[0020] 1. Feature representation and preliminary fusion encoding module: The core task of this module is to extract preliminary features from multimodal input data (such as VIS images and DSM data) and achieve preliminary fusion, providing a solid foundation for subsequent deep feature extraction and decoding processes.

[0021] This module adopts a dual-branch structure to process VIS images and DSM data respectively, and extracts features through an improved ResNet structure. Each branch performs convolution operations using a standard two-dimensional convolution layer and uses multiple residual blocks to extract and enhance features. Each residual block contains a standard convolution layer, a ReLU activation layer, and a group normalization layer, and is optimized for cross-scale information extraction. In the feature extraction process, both the VIS and DSM encoders introduce an adaptive weighted fusion (AWF) module to achieve efficient fusion of dual-modal features. The adaptive weighted fusion module is as follows: Figure 2 As shown in the figure, it consists of two main branches, which process features from DSM data and VIS data respectively. First, in each branch, the input features are globally average pooled to capture global context information. Next, the features are compressed through a 1×1 convolution layer and a ReLU activation function. Then, the feature representation is further adjusted through a depth-wise separable 1×1 convolution layer to improve computational efficiency. Finally, the output is normalized by the Sigmoid function to generate adaptive weights, which are used to dynamically adjust the importance of features from different modalities during the fusion process. After obtaining the feature weights of the two different modalities, these weights are multiplied by their respective features to generate a weighted feature representation. The features of the two modalities are weighted fused through the adaptive weight mechanism, and the fused output is: ; Among them, ω y and ω x Represent the adaptive weights of deep features and VIS features respectively.

[0022] 2. Deep feature processing and fusion coding module: After completing feature representation and preliminary fusion coding, the deep feature processing and fusion coding module proposed in this embodiment further performs deep processing and fusion on the extracted multimodal features.

[0023] The core of this module is a multi-layer encoder based on the Transformer architecture, which realizes the gradual enhancement of features and the deep fusion of cross-modal information through multiple levels of sparse attention mechanisms and gated fusion mechanisms combined with attention. First, the preliminary features from VIS images and DSM data are mapped to a unified feature space through an embedding layer. The embedded features are passed as input to the multi-layer Transformer encoder.

[0024] The Transformer encoder consists of 12 layers, including gated fusion layers and sparse attention layers combined with attention. The first three layers are sparse attention layers, which are responsible for selectively activating key feature points; the next three layers are gated fusion layers, which combine the attention mechanism for deep fusion of features; the next three layers are sparse attention layers, which further optimize the capture of global features; the last three layers are gated fusion layers, which further enhance the fusion and selectivity of features.

[0025] Sparse attention mechanisms, such as Figure 3 As shown in the figure, the sparse attention layer selectively activates important feature points, reduces computational redundancy, and improves the efficiency and feature extraction capability of the model. The sparse attention layer is an improvement on the self-attention layer in the traditional transform. The principle of the sparse attention module in the sparse attention layer is as follows: Figure 4 As shown in Figure 1, by concentrating calculations in specific areas, the model can capture global and local key features more quickly and effectively. Specifically: (1) First, the input feature vector The query matrix Q, key matrix K, and value matrix V are generated through three linear layers respectively. These matrices are generated by taking the input feature vector This is achieved by projecting it into multiple subspaces. The calculation formula is: ; in, , , are the weight matrices for query, key, and value, respectively.

[0026] (2) To calculate the attention score, multiply the query matrix Q by the transpose of the key matrix K and then divide by the scaling factor in is the dimension of the key) to prevent the inner product value from being too large, causing the gradient to disappear or explode. The formula for calculating the attention score is: .

[0027] (3) After calculating the attention matrix, in order to improve computational efficiency and remove unimportant information, the larger weight scores are retained through sparse processing, and the smaller scores are set to zero. The specific method is to select the largest values ​​in the first k% of the attention matrix and set the other smaller values ​​to negative infinity −∞ to achieve sparseness. The attention matrix after sparse processing can be expressed as: ; The threshold Threshold is determined based on the first k% maximum values.

[0028] (4) The sparse attention matrix is ​​soft-maximized to obtain the weight distribution. These weight distributions are then applied to the value matrix V and the output features are obtained by weighted summation. Finally, the weighted summed features are passed through a linear layer ( ) to project and get the final output: ; ; Output represents the output feature matrix after weighted summation. is the sparse attention matrix.

[0029] The gated fusion mechanism ensures that key information is transmitted first by regulating the information flow between the features of each modality, while suppressing the interference of irrelevant features. The gated fusion module combined with attention further improves the accuracy and robustness of fusion through self-attention and cross-attention mechanisms.

[0030] The structure of the gated fusion layer combined with attention is similar to that of the sparse attention layer. The specific framework diagram is as follows Figure 5 As shown, the principle of the gated fusion module combined with attention is as follows Figure 6 As shown, the details are as follows: (1) Input features ,、 Perform self-attention calculations respectively to generate enhanced feature representations attentionsy and attentionsx. The formula is as follows: .

[0031] (2) attentionsx and attentionsy generate gating weights through their respective gating networks The gating network consists of a linear layer and a Sigmoid activation function, which is used to control the fusion ratio of the two features. The generation process of the gating weight is as follows: gate_x1=σ( *attentionsx) gate_x2=σ( *attentionsx) gate_y1=σ( *attentionsy) gate_y2=σ( *attentionsy) in, is the weight of the linear layer σ represents the Sigmoid function, which ensures that the generated weight value is in the range of [0,1].

[0032] (3) According to the gate weight , perform weighted fusion of the two features attentionsx and attentionsy, and the fusion calculation process is as follows: fused_x=gate_x1*attentionsx+gate_x2*attentionsy fused_y=gate_y1*attentionsy+gate_y2*attentionsx Among them, fused_x represents the fused feature x, and fused_y represents the fused feature y.

[0033] 3. Decoding and feature reconstruction module: After completing the feature extraction and encoding of multimodal data, the encoder generates low-resolution deep feature maps. These feature maps contain rich fused semantic and spatial information. However, since the encoder downsamples the input data layer by layer, the resolution of the feature map is low and cannot be directly used for accurate semantic segmentation. Therefore, in the next step, we introduce a decoding and feature reconstruction module to restore the low-resolution feature map output by the encoder to the same resolution as the original input through step-by-step upsampling and feature reconstruction.

[0034] Specifically, this module restores the low-resolution feature map generated by the encoder to the high resolution of the input image through gradual upsampling and multi-scale feature fusion. Each decoding block uses deconvolution for upsampling, and the multi-scale features in the encoder are spliced ​​with the features of the decoder through jump connections to ensure that more context and detail information is retained. The spliced ​​feature map undergoes two convolutions to adjust the number of channels and further extract features, and finally generates a semantic segmentation result with the same resolution as the input image through the segmentation head. This upsampling strategy ensures the complete recovery of detailed features, and combines the multi-scale features of the encoder to achieve high-precision semantic segmentation output.

[0035] Based on the MSGT-Net model framework constructed in this embodiment, semantic segmentation of high-resolution remote sensing images is performed, and the specific method is as follows: step 1, obtaining a remote sensing image to be segmented; Step 2, inputting the remote sensing image to be segmented into a pre-established and trained high-resolution remote sensing image semantic segmentation model, extracting preliminary features through a feature representation and preliminary fusion coding module, and achieving preliminary fusion; Step 3: The image obtained in step 2 is processed by deep feature processing and fusion coding module, and the gradual enhancement of features and the deep fusion of cross-modal information are achieved through multiple levels of sparse attention mechanism and gated fusion mechanism combined with attention; Step 4: The image obtained in step 3 is processed by the decoding and feature reconstruction module. Through step-by-step upsampling and multi-scale feature fusion, the low-resolution feature map generated by the encoder is restored to the high resolution of the input image, and finally a high-precision semantic segmentation image is obtained.

[0036] In this embodiment, the Vaihingen and Potsdam datasets provided by ISPRS are selected for model training and experimental testing. The Vaihingen dataset consists of 16 high-resolution orthophotos with an average size of 2500×2000 pixels. Each orthophoto has three channels: near infrared, red, and green (NIRRG), and provides a normalized digital surface model (DSM) with a ground sampling distance (GSD) of 9 cm. The dataset includes 5 foreground categories, namely buildings, trees, low vegetation, cars, impermeable surfaces, and 1 background category. Further, the 16 orthophotos are divided into a training set containing 12 tiles and a test set containing 4 tiles. The training set contains orthophotos with indexes 1, 3, 23, 26, 7, 11, 13, 28, 17, 32, 34, and 37, while the test set is 5, 21, 15, and 30.

[0037] Potsdam dataset: This dataset consists of 24 very high-resolution orthophotos, each with a size of 6000×6000 pixels. The dataset provides four multispectral channels: infrared, red, green, and blue (IRRGB), as well as a normalized digital surface model (DSM) with a ground sampling distance (GSD) of 5 cm. The foreground categories of the Potsdam dataset are the same as those of Vaihingen, but the distribution of the foreground categories is different due to different geographical locations. This dataset uses RGB combined images. The 24 orthophotos are divided into 18 tiles for training and 6 tiles for testing. Specifically, the training set contains orthoimages with indices 6_10, 7_10, 2_12, 3_11, 2_10, 7_8, 5_10, 3_12, 5_12, 7_11, 7_9, 6_9, 7_7, 4_12, 6_8, 6_12, 6_7, 4_11, while the test set contains 2_11, 3_10, 4_10, 5_11, 6_11, 7_12.

[0038] To ensure that the model can be trained effectively, the dataset was preprocessed in detail. First, the image pixel values ​​were normalized to the range of [0, 1] to reduce the impact of illumination differences and accelerate model convergence. Then, the large image was cut into small blocks of 256x256 pixels using a sliding window technique with a sliding step of 32 pixels. In addition, the ground truth labels were converted from RGB color encoding to numerical labels, and the boundary areas were refined. To improve the generalization ability of the model, data augmentation techniques were also applied, including random horizontal flipping, vertical flipping, and rotation. For the DSM data, the same normalization was performed and the sliding window technique was used for slicing to ensure consistency with the image block size.

[0039] Overall Accuracy (OA), mean F1 score (mF1) and mIoU mean (Intersection over Union) are used to objectively quantify the segmentation performance of the network. The formulas of the above indicators can be expressed by the confusion matrix as follows: ; ; ; ; ; TP, TN, FP and FN represent true positive, true negative, false positive and false negative, respectively. It represents the proportion of samples that are actually positive (TP) among all samples predicted to be positive (TP + FP). It represents the proportion of samples that are correctly predicted as positive (TP) among all samples that are actually positive (TP + FN). According to the above definition, after obtaining the F1 and IoU of all classes, the average values, namely mF1 and mIoU, are derived.

[0040] During the model training process, the batch size was set to 10 to balance computing resources and model convergence speed. Each training batch contained remote sensing data of different modalities (such as VIS data and DSM data). These data were preprocessed and input into the model for feature extraction and fusion. The training used the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.01. The MultiStepLR scheduler was used to adjust the learning rate during the training process. Due to the problem of class imbalance, the experiment used a weighted cross entropy loss function. By setting weights for different categories, the model's ability to handle imbalanced data was improved, especially the recognition effect of minority classes.

[0041] This experiment used the PyTorch deep learning framework, combined with the NVIDIA RTX 3090 GPU (24GB) for accelerated computing, and ran in the CUDA 11.8 environment.

[0042] The performance of the proposed model is compared with SA-GATE, TransUNet, MFTransNet, CMFNet and MSGT-Net on the Vaihingen and Potsdam datasets.

[0043] Table 1 shows the comparison results on the Vaihingen dataset. MSGT-Net performs well in multiple categories, especially in the classification of buildings (97.76%), trees (91.73%), and cars (90.33%), with an average F1 score of 90.88% and mIoU of 83.65%, which are significantly better than other methods.

[0044] Table 1: Comparison results of different models on the Vaihingen dataset

[0045] As can be seen from Table 2, MSGT-Net also performs well on the Potsdam dataset, especially in the classification of buildings (98.24%) and low vegetation (89.20%), with an average F1 score of 92.30% and an mIoU of 86.10%.

[0046] Table 2: Comparison results of different models on the Potsdam dataset

[0047] Figure 7The segmentation diagrams of the model constructed by the present invention and the prior art model on two data sets are obtained from Figure 7 As can be seen from the first and second rows of , SA-GATE and TransUNet as single-modal models perform slightly poorly in building boundary segmentation, especially when the edges are occluded or the building and ground features are similar. In contrast, MFTransNet and CMFNet as multi-modal models, combined with the height information of nDSM, can better segment the complete building boundaries. However, MSGT-Net performs best in this type of scenario, not only the segmentation boundaries are more accurate, but also the integrity of the building is better preserved.

[0048] In the results of the third and fourth rows, although the color and texture of low vegetation are similar to other categories (such as trees and cluttered objects), MSGT-Net can more accurately identify low vegetation by fusing the height information of nDSM, and shows significant advantages in the classification of this area. In particular, when distinguishing low vegetation from trees, MSGT-Net further improves the classification accuracy by effectively fusing height information, demonstrating its excellent parsing ability in distinguishing complex categories. This shows that MSGT-Net can distinguish objects with similar appearance in detail under multimodal data fusion, significantly improving the classification effect.

[0049] Table 3 shows the impact of different module combinations on the model performance, and evaluates the performance of the adaptive weighted fusion (AWF) module and the self-attention gating mechanism (SGF) module in terms of overall accuracy (OA), mean F1 score (mF1), and mean intersection over union (mIoU).

[0050] Table 3: Model performance of different module combinations

[0051] In the base model without the fusion module of AWF and SGF, the model achieves OA of 90.96%, Mean F1 of 87.34%, and mIoU of 78.26%, providing a reference point for baseline performance.

[0052] After the introduction of the AWF module, the performance was significantly improved, with OA reaching 92.10%, Mean F1 increasing to 90.54%, and mIoU reaching 83.13%. This shows that the AWF module improves the fusion effect of multimodal data through adaptive weight adjustment, thereby improving the semantic segmentation performance.

[0053] When only the SGF module is introduced, the performance of the model is also enhanced, with OA of 91.57%, Mean F1 of 90.08%, and mIoU of 82.35%. The self-attention gating mechanism improves global semantic understanding and segmentation accuracy by dynamically adjusting the information flow between different modal features.

[0054] When the model introduces both AWF and SGF modules, the model performance reaches the best, with OA reaching 92.30%, Mean F1 reaching 90.65%, and mIoU reaching 83.29%. This shows that the synergy of the two modules can maximize the fusion and feature extraction of multimodal data, ensuring that the model achieves optimal performance in multiple evaluation indicators.

[0055] Figure 8 The ablation experiment comparison results of different models in the remote sensing image semantic segmentation task are shown, including the baseline model, the introduction of the AWF module, the introduction of the SGF module, and the complete MSGT-Net model. Through the comparison, it can be clearly seen that the introduction of different modules has a significant impact on segmentation accuracy and detail processing.

[0056] Without introducing any modules, the segmentation results of the baseline model show obvious errors, especially in boundary processing and small object recognition. For example, the segmentation of trees and low vegetation is rough, and the boundaries of buildings are blurred.

[0057] After adding the AWF module, the segmentation results have been significantly improved, especially after multimodal data fusion, the model has a more refined segmentation of ground object categories. It can be seen that the boundaries of buildings are clearer than the baseline model, and the classification of trees and low vegetation is more accurate.

[0058] The performance of the model is further improved when the self-attention gated fusion (SGF) module is introduced. The gated fusion mechanism can not only dynamically adjust the transmission of features, but also enhance the capture of key information through the self-attention mechanism. Compared with the baseline model and the model using only AWF, the SGF module significantly improves the segmentation performance of small-scale objects (such as trees and low vegetation), and the boundaries of buildings are also clearer and more accurate.

[0059] When the AWF module and the SGF module were introduced at the same time, the model achieved the best performance. The segmentation results show that the model can better capture the detailed information of the ground objects, the boundaries of buildings and vegetation are clear, and the classification accuracy is higher. Especially in complex scenes, MSGT-Net can more accurately distinguish objects of different categories. The framed part in the figure further demonstrates the significant advantages of MSGT-Net in detail processing.

[0060] The above embodiments are only exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present invention, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. A high-resolution remote sensing image semantic segmentation model, characterized in that: It includes a feature representation and preliminary fusion coding module, a deep feature processing and fusion coding module, and a decoding and feature reconstruction module. The feature representation and preliminary fusion coding module is used to extract preliminary features from multimodal input data to achieve preliminary fusion; the deep feature processing and fusion coding module is based on a multi-layer encoder of the Transformer architecture, and realizes the gradual enhancement of features and the deep fusion of cross-modal information through multiple levels of sparse attention mechanisms and a gated fusion mechanism combined with attention; the decoding and feature reconstruction module restores the low-resolution feature map generated by the encoder to the high resolution of the input image through gradual upsampling and multi-scale feature fusion.

2. A high-resolution remote sensing image semantic segmentation model according to claim 1, characterized in that: The feature representation and preliminary fusion coding module includes a VIS encoder and a DSM encoder, which are used to process VIS images and DSM data respectively, and feature extraction is performed through a ResNet structure. Each branch uses a standard two-dimensional convolutional layer for convolution operations, and uses multiple residual blocks to achieve feature extraction and enhancement. Each residual block contains a standard convolutional layer, a ReLU activation layer and a group normalization layer. In the feature extraction process, both the VIS encoder and the DSM encoder introduce an adaptive weighted fusion module.

3. A high-resolution remote sensing image semantic segmentation model according to claim 2, characterized in that: The adaptive weighted fusion module consists of two branches for processing features from DSM data and VIS data. In each branch, global average pooling is performed on the input features to capture global context information. After a 1×1 convolution layer and ReLU activation function, the features are compressed; the feature representation is further adjusted through a depthwise separable 1×1 convolution layer; The output is normalized by the Sigmoid function to generate adaptive weights. After obtaining the feature weights of the two different modalities, the two feature weights are multiplied by their respective features to generate a weighted feature representation. The features of the two modalities are weighted and fused through the adaptive weight mechanism. The fused output is: ； in, and Represent the adaptive weights of deep features and VIS features respectively.

4. The high-resolution remote sensing image semantic segmentation model according to claim 1, characterized in that: The multi-layer encoder of the Transformer architecture consists of 12 layers, including gated fusion layers and sparse attention layers combined with attention, of which the first three layers are sparse attention layers, which are used to selectively activate key feature points; the next three layers are gated fusion layers, which combine the attention mechanism for deep fusion of features; the next three layers are sparse attention layers, which further optimize the capture of global features; the last three layers are gated fusion layers, which further enhance the fusion and selectivity of features.

5. A high-resolution remote sensing image semantic segmentation model according to claim 4, characterized in that: The sparse attention module in the sparse attention layer is specifically: Input feature vector The query matrix Q, key matrix K and value matrix V are generated through three linear layers respectively. The specific calculation formula is: ； in, , , are the weight matrices for query, key, and value, respectively; Calculate the attention score based on the query matrix Q and the key matrix K , the specific calculation formula is: ； in, is the scaling factor, is the dimension of the key; Select the largest values ​​of the first k% in the attention matrix and set other smaller values ​​to negative infinity to achieve sparseness. The sparse attention matrix can be expressed as: ； Among them, the threshold Threshold is determined according to the maximum value of the first k%; The sparse attention matrix is ​​soft-maximized to obtain the weight distribution, which is applied to the value matrix V, and the output features are obtained by weighted summation. The weighted summed features are projected through a linear layer to obtain the final output.

6. A high-resolution remote sensing image semantic segmentation model according to claim 4, characterized in that: The gated fusion layer combined with attention is specifically: For input features , Perform self-attention calculations respectively to generate enhanced feature representations attentionsy and attentionsx. The formula is as follows: attentionsx and attentionsy generate gating weights through their respective gating networks ; The gating network consists of a linear layer and a Sigmoid activation function, which is used to control the fusion ratio of the two features. The generation process of the gating weight is as follows: gate_x1=σ( *attentionsx) gate_x2=σ( *attentionsx) gate_y1=σ( *attentionsy) gate_y2=σ( *attentionsy) in, is the weight of the linear layer, σ represents the Sigmoid function, and the Sigmoid function ensures that the generated weight value is in the range of [0,1]; According to the gate weight , perform weighted fusion of the two features attentionsx and attentionsy, and the fusion calculation process is as follows: fused_x=gate_x1*attentionsx+gate_x2*attentionsy fused_y=gate_y1*attentionsy+gate_y2*attentionsx Among them, fused_x represents the fused feature x, and fused_y represents the fused feature y.

7. A high-resolution remote sensing image semantic segmentation method, characterized in that: A high-resolution remote sensing image semantic segmentation model based on any one of claims 1 to 6 specifically comprises the following steps: Step 1, obtaining a remote sensing image to be segmented; Step 2, inputting the remote sensing image to be segmented into a pre-established and trained high-resolution remote sensing image semantic segmentation model, extracting preliminary features through a feature representation and preliminary fusion coding module, and achieving preliminary fusion; Step 3: The image obtained in step 2 is processed by deep feature processing and fusion coding module, and the gradual enhancement of features and the deep fusion of cross-modal information are achieved through multiple levels of sparse attention mechanism and gated fusion mechanism combined with attention; Step 4: The image obtained in step 3 is processed by the decoding and feature reconstruction module. Through step-by-step upsampling and multi-scale feature fusion, the low-resolution feature map generated by the encoder is restored to the high resolution of the input image, and finally a high-precision semantic segmentation image is obtained.

Images