Multi-source remote sensing image segmentation method and device based on global semantic gating mechanism

Abstract

The present application relates to the technical field of remote sensing intelligent interpretation, in particular to a multi-source remote sensing image segmentation method and device based on a global semantic gating mechanism. The method comprises: using a double-branch to perform multi-scale feature extraction on optical images and DSM images respectively, fusing and enhancing each shallow feature extracted to obtain the highest semantic optical feature and the highest semantic DSM feature; performing cross-region and cross-modal context modeling on the highest semantic optical feature and the highest semantic DSM feature to obtain deep optical features and deep DSM features; calculating the deep optical features and the deep DSM features through self-attention and cross-attention, and combining a semantic gating mechanism to perform adaptive fusion to obtain deep fusion features; and using a layer-by-layer upsampling and convolution manner to process the deep fusion features and the i-th level jump features output by the multi-layer adaptive gating jump to obtain multi-class ground object segmentation results. The present application can effectively improve the segmentation performance.

Description

Technical Field

[0001] This invention relates to the field of remote sensing intelligent interpretation technology, and in particular to a multi-source remote sensing image segmentation method and apparatus based on a global semantic gating mechanism. Background Technology

[0002] Semantic segmentation of remote sensing images is a fundamental and challenging task in the field of intelligent remote sensing interpretation. Its goal is to accurately classify each pixel in a remote sensing image, thereby achieving a refined spatial representation of ground features. Unlike traditional image classification tasks, remote sensing semantic segmentation not only needs to identify the categories of targets in the image but also accurately locate target boundaries and preserve spatial structure information. Therefore, it plays a crucial role in fields such as urban planning, land cover mapping, natural resource monitoring, and disaster assessment.

[0003] However, due to the high complexity and diversity of remote sensing scenes, remote sensing semantic segmentation tasks still face many challenges. For example, the widespread intra-class differences and inter-class similarities in high-resolution remote sensing images make it more difficult to distinguish between different ground features; at the same time, factors such as shadow occlusion, changes in imaging conditions, differences in target scale, and complex background interference can also lead to blurred boundaries and semantic misjudgments.

[0004] With the development of multimodal remote sensing technology, data from different sensors can provide complementary information. Optical images can provide rich spectral and texture information, while Digital Surface Model (DSM) images can effectively supplement information on ground feature height and spatial structure, helping to improve the ability to distinguish targets in remote sensing scenes. However, there are significant differences between different modalities, and shallow details are easily lost, while deep cross-modal interactions are insufficient. Existing methods still face problems such as blurred boundaries, category confusion, and low fusion efficiency in complex urban scenes.

[0005] Therefore, overcoming the shortcomings of the existing technology is an urgent problem to be solved in this technical field. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention proposes a multi-source remote sensing image segmentation method and apparatus based on a global semantic gating mechanism, which can effectively improve segmentation performance.

[0007] The embodiments of the present invention adopt the following technical solutions: In a first aspect, the present invention provides a multi-source remote sensing image segmentation method based on a global semantic gating mechanism, specifically: segmenting optical images... and DSM images The input is a multi-layer encoder; the multi-layer encoder uses two branches to extract features at multiple scales, and then fuses and enhances each shallow feature extracted by the two branches through multi-layer adaptive gating jump to obtain the highest semantic optical features. and highest semantic DSM features ; The highest semantic optical features and highest semantic DSM features Cross-regional and cross-modal context modeling is performed to obtain deep optical features. and deep DSM features ; Deep optical features and deep DSM features Deep fusion features are obtained by calculating self-attention and cross-attention, and by combining semantic gating mechanism for adaptive fusion. The deep fusion features are combined with the i-th level jump feature of the multi-layer adaptive gated jump output. The data are input into a multi-layer decoder; the multi-layer decoder uses layer-by-layer upsampling and convolution to restore spatial resolution and obtain the segmentation result.

[0008] Preferably, the method further includes: The dual-branch optical image and DSM images Feature extraction is performed layer by layer to obtain the first... Level shallow optical features and the Level shallow DSM features ; The first Level shallow optical features and the Level shallow DSM features By fusing multiple layers of adaptive gated jumps, the 1st... Level fusion features ; Using the first Level fusion features For the Level shallow optical features After enhancement, feature extraction is performed to obtain the first... +1 level shallow optical features ; Individually for the first Level shallow DSM features Feature extraction is performed to obtain the first... +1 level shallow DSM features ; The above feature extraction and enhancement processes are repeated cyclically to obtain the highest semantic optical features. and highest semantic DSM features .

[0009] Preferably, the method further includes: For the first Level shallow optical features and the Level shallow DSM features The concatenated features are then stitched together, and a 1×1 convolution is used to compress and reassemble the channels of the stitched features. The output after convolution is then scaled using a channel attention mechanism to obtain the nth channel. Level fusion features .

[0010] Preferably, the method further includes: Obtaining higher semantic features in multi-layer decoders ; For higher semantic features Perform global average pooling to extract global semantic statistics for each channel and obtain channel feature vectors; use 1×1 convolution to perform channel dimensionality reduction on the channel feature vectors and perform non-linear mapping through a non-linear activation function; use 1×1 convolution to perform channel dimension mapping on the non-linearly mapped feature vectors to obtain gate vectors, and feed the gate vectors into the Sigmoid function to obtain the gate coefficients G. Using the gating constant coefficient G to the first Level fusion features Pixel-level weighting is performed to obtain the i-th level jump feature. .

[0011] Preferably, the method further includes: The highest semantic optical features and highest semantic DSM features Each feature is mapped to a unified token space and its position is encoded to obtain optical structural features. and DSM structural features ; Optical structural features and DSM structural features Modulate them separately, and then combine the modulated bimodal token with the global token. By splicing, a unified sequence is obtained. ; Unified sequence Multi-layer Transformer encoding is performed to obtain a globally context-enhanced sequence representation. ; Sequence representation Deep optical features are obtained by splitting. and deep DSM features .

[0012] Preferably, the method further includes: Input sequence features As the first The input of the first layer Transformer, the third Layer Transformer for input sequence features Perform multi-head self-attention computation, and then combine the results of the multi-head self-attention computation with the features of the input sequence. After addition, layer normalization is performed to obtain the output sequence features. ,in, Values Unified sequence For the first The input to the Transformer layer is a sequence representation. This is the output of the (L-1)th layer Transformer.

[0013] Preferably, the method further includes: Deep optical features and deep DSM features Project and flatten to obtain the optical sequence and DSM sequence ; For optical sequences and DSM sequence Single-modal self-attention and global average pooling are performed respectively to obtain optical vectors. and DSM vector ; For optical vectors and DSM vector Local similarity calculations are performed to obtain modal similarity vectors. ; Modality similarity vectors were analyzed using MLP and the sigmoid function. Perform mapping to generate semantic gates ; semantic gate We modulate the cross-attention results using weights to obtain deep fusion features.

[0014] Preferably, the method further includes: For optical sequences and DSM sequence Perform cross-attention calculation, and then combine the cross-attention calculation result with the semantic gate. Pixel-weighted injection of optical sequences The representation z is obtained; The representation z is sequentially subjected to nonlinear encoding, layer normalization, residual connection and reconstruction to obtain deep fusion features.

[0015] In a second aspect, the present invention provides a multi-source remote sensing image segmentation apparatus based on a global semantic gating mechanism, the apparatus comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to perform the multi-source remote sensing image segmentation method based on a global semantic gating mechanism in the first aspect.

[0016] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions, which are executed by one or more processors to perform the multi-source remote sensing image segmentation method based on a global semantic gating mechanism described in the first aspect.

[0017] Compared with existing technologies, the advantages of this invention are as follows: This invention uses a dual-branch multi-layer encoder skeleton, introducing a multi-layer adaptive gating skip strategy in the shallow stage to enhance the preservation of boundary and fine-grained structural information; in the deep stage, it models long-distance dependencies and cross-modal global semantic relationships through a global context-guided Transformer encoder; further, it combines sequential gating cross-attention to achieve semantic-driven on-demand cross-modal fusion, thereby improving deep feature discrimination capabilities while maintaining modal characteristics. The multi-layer adaptive gating skip strategy, global context-guided Transformer encoder modeling, and sequential gating cross-attention strategy form a complete enhancement chain from shallow detail preservation, deep global harmonization to on-demand cross-modal fusion. This complete enhancement chain effectively improves segmentation performance, exhibits stronger stability in complex region, small target, and similar object recognition tasks, while ensuring the inference efficiency of feature fusion. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is a framework diagram of GSGTNet, a multi-source remote sensing image segmentation based on a global semantic gating mechanism, provided in an embodiment of the present invention. Figure 2 This is a flowchart illustrating a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention; Figure 3 This is a flowchart of a method for fusing dual-branch feature extraction and MAGS module in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided by an embodiment of the present invention; Figure 4 This is a flowchart of the lightweight gate network GateNet in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention. Figure 5 This is a framework diagram of the MAGS module in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention; Figure 6 This is a flowchart of the GCGE module in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention. Figure 7 This is a framework diagram of the GCGE module in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention; Figure 8 This is a flowchart of the SGCA module in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention. Figure 9 This is a framework diagram of the SGCA module in a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in an embodiment of the present invention; Figure 10 This is a visualization of a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in this embodiment of the invention on the ISPRS Vaihingen dataset; Figure 11 This is a visualization of a multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in this embodiment of the invention on the ISPRS Potsdam dataset; Figure 12 This is a framework diagram of a multi-source remote sensing image segmentation device based on a global semantic gating mechanism, provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0021] Unless the context otherwise requires, throughout the specification and claims, the term "comprising" is interpreted as openly inclusive, meaning "including, but not limited to." In the description of the specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this disclosure. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics mentioned may be included in any suitable manner in any one or more embodiments or examples; that is, although they may be incorporated into embodiments or examples using the above terms for reasons such as order and position, it does not limit them to be incorporated in combination by a single embodiment or example.

[0022] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more. Furthermore, for example, the description may use the prefix "A" or "B" to describe the same type of nouns as two independent entities. In this case, the corresponding features defined with "A" and "B" are used only to distinguish between similar entities and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features.

[0023] In the description of this invention, the expression “A and / or B” (where A and B are used to formally represent specific features) will be used. The corresponding expression includes the following three combinations: only A, only B, and a combination of A and B.

[0024] As used in this invention, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from a particular value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the particular quantity (i.e., the limitations of the measurement system).

[0025] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0026] Before proposing a multi-source remote sensing image segmentation method based on a global semantic gating mechanism, the applicant first studied the existing technologies. The following is the applicant's research on the existing technologies: Remote sensing semantic segmentation aims to classify land cover elements pixel-by-pixel in multispectral, high-resolution remote sensing images to identify and distinguish various land cover features. Based on the different input data sources, existing methods can be broadly categorized into two types: unimodal semantic segmentation and multimodal semantic segmentation.

[0027] In unimodal semantic segmentation methods, Fully Convolutional Networks (FCNs) pioneered a new paradigm for end-to-end semantic segmentation of remote sensing images. Subsequently, DeepLab expanded the receptive field by introducing dilated convolutions and combined them with dilated spatial pyramid pooling to capture multi-scale contextual information. DeepLabv3+ further improved the accuracy of remote sensing image semantic segmentation by introducing an encoder-decoder structure to recover object boundary details and combining it with an improved dilated spatial pyramid pooling module to capture richer multi-scale contextual information.

[0028] Despite significant advancements achieved by methods based on FCN and its variants, as well as the DeepLab series, these methods still face several key challenges. The fundamental reason is that they are unimodal methods, relying solely on RGB (Red, Green, Blue) or grayscale images as a single input. Therefore, unimodal methods perceive very limited information and often struggle to handle complex scenes. For example, in remote sensing images, shadows, lighting variations, and the spectral similarity between different ground features (such as trees and low vegetation) can severely degrade segmentation accuracy.

[0029] To address these limitations, multimodal learning has emerged. By fusing complementary information from multiple modal data sources, multimodal learning significantly improves segmentation accuracy. These multiple modal data sources include optical images, DSM images, depth maps, and polarization images.

[0030] Early multimodal methods often employed data-level fusion. For example, FuseNet designed a dual-stream encoder that element-wise fused RGB and depth features after each convolutional layer. CMGFNet proposed a cross-modal gating fusion module that adaptively mines discriminative features and suppresses redundant information, achieving good results in building extraction tasks.

[0031] However, the aforementioned fusion methods based on Convolutional Neural Networks (CNNs) are limited by the receptive field of convolution, making it difficult to capture the interactions between modalities. This limitation has prompted researchers to introduce the Transformer architecture, which has greater global modeling capabilities. In recent years, the global receptive field has been realized through self-attention mechanisms, thus revolutionizing the field of computer vision. Among them, ViT (Vision Transformer) and its variants, such as SwinTransformer, have achieved state-of-the-art performance in image classification and segmentation tasks. However, pure Transformer encoders often lose low-level spatial details, making them ineffective in tasks requiring fine boundary delineation. As a result, existing multimodal methods relying solely on CNNs or Transformers still face problems such as blurred boundaries, category confusion, and low fusion efficiency in complex urban scenes.

[0032] To address this, this invention proposes a segmentation network based on a globally semantically gated Transformer, namely GSGTNet (Global Semantic Gated Transformer Network). The proposed GSGTNet follows a classic encoder-decoder architecture, such as... Figure 1 As shown, in the encoding structure, a dual-branch model extracts shallow, fine-grained feature maps from optical and DSM images, which are then fused by a multi-layer adaptive gated skip fusion (MAGS) module. The MAGS module consists of a channel attention mechanism and a gated network. This design preserves fine-grained structures such as boundaries and small targets at lower levels while effectively suppressing local noise from additional modalities, achieving adaptive handling of modal heterogeneity. Deeper in the network, a Global Context-Guided Transformer Encoder (GCGE) module enhances the long-distance dependencies between bimodal features. A Sequential Gated Cross-Attention (SGCA) module is then introduced. The SGCA module, composed of semantic gating and cross-attention mechanisms, forces information fusion between the two modalities only when it is beneficial and semantically consistent. This strategy avoids conflicts and noise propagation caused by blind parallel fusion, contributing to excellent performance in large-scale semantics and overall semantic consistency. In the decoding structure, spatial resolution is restored by layer-by-layer upsampling and convolution, while fusing detailed information from shallow layers to ensure boundary accuracy.

[0033] Based on the proposed GSGTNet network architecture, this invention provides a multi-source remote sensing image segmentation method based on a global semantic gating mechanism, such as... Figure 2 As shown, the specific steps include the following: Step 101: Obtain the optical image and DSM images The input is a multi-layer encoder; the multi-layer encoder uses two branches to extract features at multiple scales, and then uses multi-layer adaptive gating skipping to fuse and enhance each shallow feature extracted by the two branches to obtain the highest semantic optical feature. and highest semantic DSM features .

[0034] In one embodiment, the dual branches are two ResNet-50 branches. The multilayer encoder uses the two ResNet-50 branches to extract optical images separately. and DSM images The ResNet-50 branch consists of four CNN blocks, and the features extracted by each CNN block are called shallow features. Each CNN block extracts shallow features at different scales. After each CNN block, a MAGS module is introduced to fuse shallow features of the same scale from two ResNet-50 branches to obtain fused features. After four CNN blocks, the GCGE module is used to improve the global feature representation capability.

[0035] On one hand, the fused features are passed to the next CNN block as enhancements to the shallow features. On the other hand, the fused features are passed to the corresponding layers of the multi-layer decoder. However, to avoid the fused features being passed to the decoder indiscriminately, this invention introduces a lightweight gated network, GateNet, at the skip connections. The lightweight gated network GateNet performs pixel-level weighting on the fused features to obtain the i-th level skip features. The i-th level jump feature It is passed into the corresponding layer of the multi-layer decoder.

[0036] The ResNet-50 network utilizes pre-trained weights from ImageNet to accelerate convergence.

[0037] Step 102: Extract the highest semantic optical features and highest semantic DSM features Cross-regional and cross-modal context modeling is performed to obtain deep optical features. and deep DSM features .

[0038] In one embodiment, the highest semantic optical features are obtained through the GCGE module. and highest semantic DSM features Perform cross-regional and cross-modal context modeling to improve the ability to represent global features.

[0039] Step 103: Deep optical features and deep DSM features Deep fusion features are obtained by calculating self-attention and cross-attention, and by combining semantic gating mechanism for adaptive fusion.

[0040] In one embodiment, after the GCGE module completes its processing, an SGCA module is introduced, which is responsible for completing the deep optical features. and deep DSM features The deep fusion feature is obtained by fusing the features and then combining it with the i-th level jump feature from the multi-layer encoder. They are input into the multi-layer decoder together.

[0041] Step 104: Combine the deep fusion features with the i-th level jump feature of the multi-layer adaptive gated jump output. The data are input into a multi-layer decoder; the multi-layer decoder uses layer-by-layer upsampling and convolution to restore spatial resolution and obtain the segmentation result.

[0042] In one embodiment, the multilayer decoder receives deep fusion features and the i-th level jump feature. Spatial resolution is restored by layer-by-layer upsampling and convolution, while detailed information from shallow layers is fused to ensure boundary accuracy, resulting in segmentation results. In the field of remote sensing intelligent interpretation technology, the segmentation results are ground feature segmentation results, where ground features include natural features (such as vegetation, water bodies, and bare land) and artificial features (such as buildings, roads, and farmland).

[0043] In this embodiment, the present invention uses a dual-branch multi-layer encoder skeleton. In the shallow stage, a multi-layer adaptive gating skip strategy is introduced to enhance the preservation of boundary and fine-grained structural information. In the deep stage, a global context-guided Transformer encoder models long-distance dependencies and cross-modal global semantic relationships. Furthermore, sequential gating cross-attention is combined to achieve semantic-driven on-demand cross-modal fusion, thereby improving deep feature discrimination capabilities while maintaining modal characteristics. The multi-layer adaptive gating skip strategy, global context-guided Transformer encoder modeling, and sequential gating cross-attention strategy form a complete enhancement chain from shallow detail preservation, deep global harmonization to on-demand cross-modal fusion. This complete enhancement chain effectively improves segmentation performance, exhibits stronger stability in complex region, small target, and similar object recognition tasks, while ensuring the inference efficiency of feature fusion.

[0044] This invention extracts optical images through a dual-branch method. and DSM images Multi-scale features are fused and compressed through channel stitching using the MAGS module, combined with channel attention and pixel-level adaptive gating mechanisms to achieve effective fusion and enhancement of cross-modal features. For example... Figure 3 As shown, the highest semantic optical features are obtained. and highest semantic DSM features Specifically, the steps include the following: Step 201: The dual-branch optical image and DSM images Feature extraction is performed layer by layer to obtain the first... Level shallow optical features and the Level shallow DSM features .

[0045] In one embodiment, optical image and DSM images Where H and W are the height and width of the image, optical image. The image has three channels, while the DSM image... The channel is one.

[0046] In one embodiment, optical images After the first The features extracted from the nth CNN block are called the nth... Level shallow optical features ,in, .

[0047] In one embodiment, a DSM image After the first The features extracted from the nth CNN block are called the nth... Level shallow DSM features ,in, .

[0048] Step 202: Place the first Level shallow optical features and the Level shallow DSM features By fusing multiple layers of adaptive gated jumps, the 1st... Level fusion features .

[0049] In one embodiment, the first Level shallow optical features and the Level shallow DSM features Input the corresponding layer's MAGS module. Within the MAGS module, for the first... Level shallow optical features and the Level shallow DSM features The concatenated features are then stitched together, and 1×1 convolutions are used to compress and reorganize the channels of the stitched features, reducing redundancy and promoting cross-modal channel interaction. The output after convolution is then scaled for each channel using a channel attention mechanism to obtain the [previous] [feature / value]. Level fusion features .

[0050] Step 203: Utilize the first Level fusion features For the Level shallow optical features After enhancement, feature extraction is performed to obtain the first... +1 level shallow optical features .

[0051] In one embodiment, the first Level fusion features It is passed as the primary modality to the next CNN block, that is, the first... Level fusion features With the Level shallow optical features Combine, and then through the first +1 CNN blocks are used for feature extraction to obtain the first... +1 level shallow optical features .

[0052] Step 204: Individually analyze the first... Level shallow DSM features Feature extraction is performed to obtain the first... +1 level shallow DSM features .

[0053] In one embodiment, through the first +1 CNN block pair Level shallow DSM features Feature extraction is performed to obtain the first... +1 level shallow DSM features .

[0054] Step 205: Repeat the above feature extraction and enhancement process to obtain the highest semantic optical features. and highest semantic DSM features .

[0055] In one embodiment, the above feature extraction and enhancement are performed iteratively to obtain the highest semantic optical features. and highest semantic DSM features .

[0056] This invention will also include the first Level fusion features It is sent to the decoder of the corresponding layer, but to avoid the first layer... Level fusion features The data is passed indiscriminately to the decoder. A lightweight gated network, GateNet, is introduced at the skip connection. The GateNet performs pixel-level weighting on the fused features to obtain the i-th level skip feature. ,like Figure 4 As shown, the i-th level jump feature is obtained. Specifically, the steps include the following: Step 301: Obtain higher semantic features from the multi-layer decoder .

[0057] Among them, higher semantic features These are features output by the SGCA module. They are features extracted through an attention mechanism that possess both strong semantic discriminative power and retain clear spatial structure.

[0058] Step 302: For higher semantic features Perform global average pooling to extract global semantic statistics for each channel and obtain channel feature vectors; use 1×1 convolution to perform channel dimensionality reduction on the channel feature vectors and perform non-linear mapping through a non-linear activation function; use 1×1 convolution to perform channel dimension mapping on the non-linearly mapped feature vectors to obtain gate vectors, and feed the gate vectors into the Sigmoid function to obtain the gate coefficients G.

[0059] In one embodiment, the gating coefficient G is expressed as follows: ; in, The gating coefficient, This indicates a global average pooling operation. For higher semantic features, Represents a 1×1 convolution. This represents a non-linear activation function.

[0060] Step 303: Use the gating constant coefficient G to apply the first... Level fusion features Pixel-level weighting is performed to obtain the i-th level jump feature. .

[0061] In one embodiment, the i-th level jump feature The expression is as follows: ; in, Indicates the first Level jump characteristics, This represents element-wise multiplication. is the gating coefficient.

[0062] like Figure 5 The diagram shows the framework of the MAGS module. The MAGS module performs channel concatenation and compression fusion on shallow features of the same scale from both branches to obtain fused features. Before inputting the fused features into the next CNN block, the fused features are introduced into shallow optical features of the same scale. Furthermore, the fused features are processed by the lightweight gated network GateNet before being fed into the decoder of the corresponding layer.

[0063] It is important to note that in the fourth MAGS module, since there are no higher semantic features, the lightweight gated network GateNet only exists in the first three layers of feature extraction.

[0064] After feature extraction from four CNN blocks, the global feature representation capability is further enhanced through the GCGE module. GCGE is designed to enhance the global semantic representation capability of bimodal features. It focuses on global context modeling and cross-modal semantic association, establishing long-distance dependencies through a multi-head self-attention mechanism in a unified token space. It also leverages global tokens and modal embeddings to enhance cross-modal context interaction capabilities, thereby improving global feature representation capability while maintaining modal characteristics. Figure 6 As shown, deep optical features are obtained. and deep DSM features Specifically, the steps include the following: Step 401: Extract the highest semantic optical features and highest semantic DSM features Each feature is mapped to a unified token space and its position is encoded to obtain optical structural features. and DSM structural features .

[0065] In one embodiment, optical structural features The expression is as follows: ; in, As an optical structural feature, The highest semantic optical feature, This indicates the Patch Embedding operation. This represents a location encoding used to preserve the spatial location information of features.

[0066] In one embodiment, DSM structural features The expression is as follows: ; in, This is a characteristic of the DSM structure. The highest semantic DSM feature, This indicates the Patch Embedding operation. This represents a location encoding used to preserve the spatial location information of features.

[0067] Step 402: Analyze the optical structural features and DSM structural features Modulate them separately, and then combine the modulated bimodal token with the global token. By splicing, a unified sequence is obtained. .

[0068] In one embodiment, to enhance optical structural features and DSM structural features To enhance discriminability in a unified sequence space, learnable modal embeddings are introduced to improve optical structural features. and DSM structural features Modulation is performed to obtain a bimodal token. Then, the global token is... Concatenate with bimodal tokens to form a unified sequence .

[0069] In one embodiment, the corresponding expression is as follows: ; ; ; in, As an optical structural feature, This is a characteristic of the DSM structure. and Learnable modal embeddings are used to enhance the model's ability to perceive the sources of features from different modalities. and For the modulated bimodal token, It is a global token. For a unified sequence.

[0070] Among them, the global token is used to aggregate cross-regional semantic context information, and the multi-head self-attention mechanism can model long-distance dependencies in a unified token space, thereby breaking through the limitations of the local receptive field of convolution; modal embedding is used to enhance the discriminability of different modalities, so that the model can maintain modal feature differences while sharing global modeling, and ultimately improve the global semantic expression ability and cross-modal information interaction ability in complex remote sensing scenarios.

[0071] Step 403: Unify the sequence Multi-layer Transformer encoding is performed to obtain a globally context-enhanced sequence representation. .

[0072] Specifically, input sequence features As the first The input of the first layer Transformer, the third Layer Transformer for input sequence features Perform multi-head self-attention computation, and then combine the results of the multi-head self-attention computation with the features of the input sequence. After addition, layer normalization is performed to obtain the output sequence features. ,in, Values , Indicates the number of Transformer Blocks. Uniform sequence. For the first The input to the L-1th layer Transformer is used as the input to the next layer Transformer, and the calculation is performed layer by layer until the sequence representation of the output of the L-1th layer Transformer is obtained. .

[0073] In one embodiment, sequence representation The expression is as follows: ; in, Indicates the first The output sequence features of the Transformer layer Indicates the first Input sequence features of the Transformer layer MSA(·) represents multi-head self-attention computation, and LN(·) represents layer normalization processing.

[0074] In one embodiment, the expression for multi-head self-attention is: ; ; in, This represents the output projection matrix of the linear mapping performed after concatenating the features of each attention head in a multi-head attention process. It is used to fuse the features of different heads into a unified feature space. , , These represent the query, key, and value matrices, respectively. This represents the scaling factor.

[0075] Step 404: Sequence representation Decomposition yields deep optical features and deep DSM features .

[0076] In one embodiment, the sequence is represented Deconstructing back to a bimodal representation yields deep optical features. and deep DSM features .

[0077] like Figure 7 The diagram shown is a framework diagram of the GCGE module. The GCGE module achieves the highest semantic optical features by introducing modality embedding and global tokens. and highest semantic DSM features It uses a unified tokenized representation and leverages the Transformer encoder for cross-modal global dependency modeling, thereby improving feature representation and context modeling capabilities.

[0078] To achieve semantically driven, on-demand deep cross-modal information fusion, an SGCA module is introduced after the GCGE module. In the SGCA module, unimodal self-attention is first performed on each modality to solidify the internal representation of the modality, and then semantic gates are generated based on the similarity between the two modalities. According to semantic gates Sequential cross-attention injection is performed to obtain deep fusion features, such as... Figure 8 As shown, obtaining deep fusion features specifically includes the following steps: Step 501: Deep optical features and deep DSM features Project and flatten to obtain the optical sequence and DSM sequence .

[0079] Step 502: Optical sequence and DSM sequence Single-modal self-attention and global average pooling are performed respectively to obtain optical vectors. and DSM vector .

[0080] In one embodiment, for optical sequences Perform single-modal self-attention and global average pooling to obtain the optical vector. Optical vector The expression is as follows: ; in, For optical vectors, It is an optical sequence. This indicates single-modal self-attention computation. This indicates global average pooling.

[0081] In one embodiment, for DSM sequences Perform unimodal self-attention and global average pooling to obtain the DSM vector. DSM vector The expression is as follows: ; in, For DSM vectors, It is a DSM sequence. This indicates single-modal self-attention computation. This indicates global average pooling.

[0082] Step 503: Adjust the optical vector and DSM vector Local similarity calculations are performed to obtain modal similarity vectors. .

[0083] In one embodiment, modal similarity vectors The calculation formula is as follows: ; in, For modal similarity vectors, For optical vectors, For DSM vectors, This represents matrix multiplication.

[0084] Step 504: Analyze the modality similarity vector using MLP and the Sigmoid function. Perform mapping to generate semantic gates .

[0085] In one embodiment, semantic gates The expression is as follows: ; in, For semantic gates, is the modal similarity vector, and MLP is a multilayer perceptron.

[0086] Step 505: Using semantic gates We modulate the cross-attention results using weights to obtain deep fusion features.

[0087] Specifically, semantic gates After generation, the optical sequence and DSM sequence Perform cross-attention calculation, and then combine the cross-attention calculation result with the semantic gate. Pixel-weighted injection of optical sequences , thus obtaining z.

[0088] ; in, This represents the representation after the main modality is injected, and the operators are... This represents element-wise multiplication. Indicates an optical sequence. Semantic gates, This indicates cross-attention calculation. It is an optical sequence. It is a DSM sequence.

[0089] The representation z is sequentially subjected to nonlinear encoding, layer normalization, residual connection and reconstruction to obtain deep fusion features.

[0090] The deep fusion features are combined with the i-th level jump feature of the multi-layer adaptive gated jump output. The data are input into a multi-layer decoder; the multi-layer decoder uses layer-by-layer upsampling and convolution to restore spatial resolution and obtain the segmentation result.

[0091] like Figure 9 The diagram shown is a framework diagram of the SGCA module. The SGCA module achieves deep optical features through self-attention and cross-attention. and deep DSM features The features are interacted and combined with a semantic gating mechanism for adaptive fusion, ultimately generating enhanced deep fusion features.

[0092] In summary, this invention proposes a multi-source remote sensing image segmentation network, GSGTNet, based on a global semantic gating mechanism. In GSGTNet, optical images are extracted separately using two branches. and DSM images The shallow features extracted from the two branches are fused and enhanced through multi-layer adaptive gating jumps to improve the ability to retain fine-grained structural information.

[0093] A Global Context Guided Transformer Encoder (GCGE) is designed for the deep layer stage to model long-distance dependencies and enhance cross-modal global semantic interactions. Simultaneously, a Sequential Gated Cross-Attention Module (SGCA) is incorporated to achieve semantically driven selective cross-modal feature fusion, thereby improving the discriminativeness and robustness of deep features.

[0094] Experimental results show that the multi-source remote sensing image segmentation method based on global semantic gating proposed in this invention achieves superior segmentation performance compared to existing mainstream methods on multiple high-resolution remote sensing datasets, and exhibits stronger stability in the identification of complex regions, small targets, and similar ground objects. Furthermore, ablation experiments and visualization analysis further validate the effectiveness and complementarity of each module, demonstrating that GSGTNet can effectively improve the overall performance of multimodal remote sensing semantic segmentation.

[0095] The applicant conducted comparative experiments using the ISPRS Vaihingen and ISPRS Potsdam datasets. Based on three metrics—Overall Accuracy (OA), Mean Intersection over Union (mIoU), and Mean F1 Score (mF1)—the study comprehensively captured pixel-level classification accuracy, the balance between precision and recall, and segmentation quality in overlapping regions. Specifically, OA evaluates foreground and background classes, while mF1 and mIoU are calculated for the five foreground classes. The results show that the multimodal method significantly outperforms the single-modal method, highlighting the added value of the multimodal fusion approach.

[0096] The Vaihingen dataset contains 16 high-resolution ground orthophotos of varying sizes, with an average pixel count of 2500×2000 pixels per patch. Each orthophoto is represented by three spectral channels: infrared, red, and green bands, and a normalized DSM at a ground sampling distance of 9 cm. The dataset is divided into a training set of 12 orthophotos and a test set of 4 orthophotos. The training set includes patches with indices 1, 3, 23, 26, 7, 11, 13, 28, 17, 32, 34, and 37, while the test set includes patches with indices 5, 21, 15, and 30. The dataset includes five foreground categories: buildings (Bui.), trees (Tre.), low vegetation (Low veg.), cars (Car), and impermeable surfaces (Imp.).

[0097] The Potsdam dataset consists of 24 very high-resolution real orthophotos, each tile being 6000×6000 pixels in size. It provides four multispectral channels: near-infrared, red, green, and blue, as well as a normalized DSM at a 5 cm ground sampling distance. For experimental convenience, the orthophotos are divided into 18 tiles for training and 6 tiles for testing. The training set includes tiles with indices 6_10, 7_10, 2_12, 3_11, 2_10, 7_8, 5_10, 3_12, 5_12, 7_11, 7_9, 6_9, 7_7, 4_12, 6_8, 6_12, 6_7, and 4_11, while the test set includes tiles with indices 4_10, 5_11, 2_11, 3_10, 6_11, and 7_12.

[0098] Unlike the Vaihingen dataset, the Potsdam dataset contains six categories: buildings (Bui.), trees (Tre.), low vegetation (Low.), cars (Car), impermeable surfaces (Imp.), and clutter (Clutter). In the Potsdam dataset, we define Clutter as the background class, and the other five classes as foreground classes.

[0099] The experiments in this paper were conducted on a Linux system using the PyTorch framework and a single NVIDIA V100 GPU. During training, stochastic gradient descent was used to train all considered models, with an initial learning rate of 0.005 for the encoder, 0.01 for the decoder, and a weight decay parameter of 0.0005. After testing, [the following data was obtained]. The value was set to 8. During training, various data augmentation techniques were used, including randomly cropping 256×256 image patches and randomly flipping horizontally and vertically.

[0100] To validate the effectiveness of GSGTNet, it was benchmarked against 12 segmentation methods, including 4 unimodal methods and 8 multimodal methods. To demonstrate the applicability of GSGTNet in complex scene remote sensing images, comparative experiments were conducted using the ISPRS Vaihingen and ISPRS Potsdam datasets. Based on three evaluation metrics (F1 score, overall accuracy, and union-mean intersection), the multimodal methods significantly outperformed the unimodal methods, highlighting the added value of the multimodal fusion approach. Comparisons among the multimodal methods further help evaluate the effectiveness of our strategy.

[0101] On the ISPRS Vaihingen dataset, GSGTNet demonstrated excellent overall performance, ranking first in six out of eight evaluation metrics, achieving an overall OA of 92.20%, an mF1 score of 90.93%, and an mIoU of 83.78%. This indicates that our proposed method possesses stronger feature representation capabilities and superior class discrimination performance in multimodal remote sensing semantic segmentation tasks. Compared to single-modal methods, GSGTNet exhibits significant advantages. Single-modal methods, such as DeepLabv3+, primarily rely on optical images for feature extraction. While they can capture local texture information to some extent, they are easily affected by factors such as shadow interference, class similarity, and boundary blurring in complex scenes, leading to limited segmentation performance. In contrast, GSGTNet, by introducing DSM modal information, effectively supplements the target's height and spatial structure features, thereby significantly improving overall segmentation accuracy.

[0102] Looking at the results across different categories, GSGTNet performs particularly well in impermeable pavement, low vegetation, and vehicle categories, achieving 93.07%, 80.32%, and 92.75% respectively. The vehicle category is particularly challenging due to its small target size, fragmented boundaries, and susceptibility to background interference. MAGS, through channel compression and recombination, channel attention enhancement, and pixel-level gated skip connections, enables the decoder to obtain cleaner shallow features. GSGTNet achieved the best results in this category, demonstrating MAGS's strong feature preservation ability for small targets and boundary-sensitive categories.

[0103] At the same time, low vegetation categories often have strong similarities to tree categories in terms of texture and appearance. Simply relying on local convolution often makes it easy to confuse trees and low vegetation. GCGE enhances the separability between categories through global dependency modeling, thereby improving the discrimination ability in complex scenes.

[0104] Figure 10 This paper presents the semantic segmentation visualization results of different methods on the Vaihingen dataset. The red boxes highlight two representative challenging scenarios: one is a region where building edges and small objects are mixed, and the other is a complex region where vegetation and trees intertwine along a roadside. Overall, GSGTNet's predictions not only preserve the integrity of building outlines but also more accurately distinguish between low vegetation, trees, and small-scale objects. In the upper red box area, the object boundaries are relatively tight, and there is obvious spatial coupling between buildings, roads, and vegetation. GSGTNet's ability to accurately recover the local structure within the red box demonstrates that the MAGS module designed in this paper filters and enhances shallow details through gated skip connections, allowing high-resolution boundary information to be more fully preserved.

[0105] In the red-boxed area below, trees and low vegetation spatially overlap. As can be seen, some methods suffer from confusion and class breaks between low vegetation and trees in this region, especially noticeable at vegetation edges. GSGTNet's predictions in this region are more continuous and complete, effectively distinguishing between trees and low vegetation. This advantage primarily stems from the synergistic effect of the GCGE and SGCA modules. GCGE enhances the characterization of long-distance dependencies and cross-regional semantic relationships through global context modeling. SGCA selectively injects cross-modal information, enabling the effective incorporation of DSM structural information into key regions.

[0106] Compared to Vaihingen, the Potsdam scene features denser building areas, more complex vehicle target distribution, and stronger spatial coupling between different categories. Therefore, it places higher demands on the model's fine-grained structural modeling capabilities and cross-modal complementary utilization capabilities. Experiments show that the OA reaches 93.89%, mF1 reaches 92.70%, and mIoU reaches 86.71%. This result demonstrates that the proposed method maintains stable and excellent segmentation performance in more challenging high-resolution urban scenes, validating the generalization ability and robustness of the proposed model. From a category perspective, GSGTNet's advantages on the Potsdam dataset mainly lie in its ability to collaboratively model large-area regular features, fine-grained boundaries, and small targets. Firstly, in the impermeable pavement and building categories, the model achieves high accuracy of 93.89% and 98.41%, respectively. This demonstrates that the proposed MAGS module can effectively preserve edge contours and local texture information at the shallow layer stage. Since these two types of land features typically possess strong spatial continuity and clear geometric boundaries in urban scenes, the full preservation of shallow features is crucial for accurately depicting their contours. MAGS adaptively filters multi-scale shallow features through gated skip connections, suppressing redundant noise while enhancing key structural information, thus helping to improve the boundary consistency and regional integrity of regular land features.

[0107] For categories like low-lying vegetation and trees, which have similar textures and are easily confused, GSGTNet achieved accuracy rates of 88.02% and 88.60%, respectively, demonstrating the crucial role of the proposed GCGE module in global semantic modeling. GCGE leverages the global context modeling capabilities of the Transformer to map high-level semantics from optics and DSM to a unified token space for interaction. This helps enhance category discrimination by utilizing spatial context and height difference information, effectively compensating for the limited receptive field of convolution. In the vehicle category, despite the small target scale and frequent interference from roads, buildings, and shadowed backgrounds, GSGTNet still maintained high recognition accuracy, demonstrating strong small target perception capabilities.

[0108] This result is attributed to the sequential gating cross-attention mechanism of the SGCA module. SGCA first stabilizes the features within each modality through single-modal self-attention, and then selectively introduces cross-modal information based on semantic relevance, enabling the structural features of the DSM to be more effectively injected into optical features in key regions. For locally fragmented and boundary-sensitive targets such as vehicles, SGCA can avoid information aliasing caused by premature fusion, thereby enhancing the discriminative power and target preservation capabilities of deep features.

[0109] Figure 11 Visualization results on the Potsdam dataset are presented. The red-boxed areas mainly contain challenging scenes such as building boundaries, vehicles, and mixed backgrounds. It can be seen that some comparative methods still suffer from blurred boundaries, missed vehicle detections, and confusion with surrounding background categories in these areas. In contrast, GSGTNet's predictions are closer to the labeled images, effectively recovering building outlines while maintaining the integrity of vehicle targets. This further demonstrates that our proposed method not only improves overall performance but also achieves more stable segmentation results in complex local regions.

[0110] To verify the effectiveness of the proposed modules, ablation experiments were conducted on the Vaihingen and Potsdam datasets. The baseline model does not introduce any arbitrary modules, employs a dual-branch encoder structure, and inputs deep features into the encoder after simple fusion. Overall, the combined use of the three innovative modules achieves state-of-the-art performance on both datasets.

[0111] MAGS enhances local details and improves overall classification accuracy in the shallow feature fusion stage, but it is insufficient to fully address class confusion in complex regions. However, introducing the GCGE module on top of MAGS significantly improves model performance. This indicates that the global context modeling introduced by GCGE in the deeper stage plays a crucial role in improving class discrimination and overall segmentation quality. Especially in scenes like Vaihingen with dense architecture and complex boundaries, global semantic information can effectively alleviate misclassification caused by local texture ambiguity.

[0112] When MAGS and SGCA are introduced, although this configuration achieves a significant improvement over the baseline model, its overall performance is still lower than the setting that simultaneously introduces MAGS and GCGE. This indicates that relying solely on localized cross-fusion is insufficient to fully leverage the complementary advantages of multimodal information. This result further illustrates the crucial role of the Transformer utilized by GCGE within the framework presented in this paper.

[0113] Furthermore, SGCA's sequential fusion strategy further enhances the cross-modal interaction capabilities of deep features, avoiding semantic aliasing issues caused by direct fusion. When all three modules are introduced simultaneously, MAGS, GCGE, and SGCA form a complete enhancement chain from shallow detail preservation and deep global reconciliation to on-demand cross-modal fusion, jointly driving performance improvement. The synergistic effect of the three modules enables the model to exhibit stable and superior performance across different datasets and complex scenarios, fully validating the effectiveness and complementarity of the three innovations presented in this paper.

[0114] like Figure 12 As shown, based on the multi-source remote sensing image segmentation method based on a global semantic gating mechanism provided in the foregoing embodiments, the present invention also provides a multi-source remote sensing image segmentation device based on a global semantic gating mechanism that can be used to implement the above method, such as... Figure 12 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The multi-source remote sensing image segmentation device based on a global semantic gating mechanism in this embodiment includes one or more processors 21 and a memory 22. Figure 12 Take a processor 21 as an example.

[0115] Processor 21 and memory 22 can be connected via a bus or other means. Figure 12 Taking the example of a connection between China and Israel via a bus.

[0116] The memory 22 serves as a non-volatile computer-readable storage medium for a multi-source remote sensing image segmentation method based on a global semantic gating mechanism. It can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the multi-source remote sensing image segmentation method based on a global semantic gating mechanism in the aforementioned embodiments. The processor 21 executes various functional applications and data processing of the multi-source remote sensing image segmentation device based on a global semantic gating mechanism by running the non-volatile software programs, instructions, and modules stored in the memory 22, thereby realizing the multi-source remote sensing image segmentation method based on a global semantic gating mechanism as described in the aforementioned embodiments.

[0117] Memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 22 may include memory remotely located relative to processor 21, which can be connected to processor 21 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0118] The program instructions / modules are stored in memory 22. When executed by one or more processors 21, they execute the multi-source remote sensing image segmentation method based on the global semantic gating mechanism in the aforementioned embodiments.

[0119] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 12 One of the processors 21 can enable the above one or more processors to execute the multi-source remote sensing image segmentation method based on the global semantic gating mechanism in the foregoing embodiments.

[0120] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0121] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0122] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-source remote sensing image segmentation method based on a global semantic gating mechanism, characterized in that, include: Optical images and DSM images The input is a multi-layer encoder; the multi-layer encoder uses two branches to extract features at multiple scales, and then uses multi-layer adaptive gating skipping to fuse and enhance each shallow feature extracted by the two branches to obtain the highest semantic optical feature. and highest semantic DSM features ; The highest semantic optical features and highest semantic DSM features Cross-regional and cross-modal context modeling is performed to obtain deep optical features. and deep DSM features ; Deep optical features and deep DSM features Deep fusion features are obtained by calculating self-attention and cross-attention, and by combining semantic gating mechanism for adaptive fusion. The deep fusion features are combined with the i-th level jump feature of the multi-layer adaptive gated jump output. The data are input into a multi-layer decoder; the multi-layer decoder uses layer-by-layer upsampling and convolution to restore spatial resolution and obtain the segmentation result.

2. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 1, characterized in that, The method further includes: The dual-branch optical image and DSM images Feature extraction is performed layer by layer to obtain the first... Level shallow optical features and the Level shallow DSM features ; The first Level shallow optical features and the Level shallow DSM features By fusing multiple layers of adaptive gated jumps, the 1st... Level fusion features ; Using the first Level fusion features For the first Level shallow optical features After enhancement, feature extraction is performed to obtain the first... +1 level shallow optical features ; Individually for the first Level shallow DSM features Feature extraction is performed to obtain the first... +1 level shallow DSM features ; The above feature extraction and enhancement processes are repeated cyclically to obtain the highest semantic optical features. and highest semantic DSM features .

3. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 2, characterized in that, The method further includes: For the Level shallow optical features and the Level shallow DSM features The concatenated features are then stitched together, and a 1×1 convolution is used to compress and reassemble the channels of the stitched features. The output after convolution is then scaled using a channel attention mechanism to obtain the nth channel. Level fusion features .

4. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 1, characterized in that, The method further includes: Obtaining higher semantic features in multi-layer decoders ; For higher semantic features Perform global average pooling to extract global semantic statistics for each channel and obtain channel feature vectors; use 1×1 convolution to perform channel dimensionality reduction on the channel feature vectors and perform non-linear mapping through a non-linear activation function; use 1×1 convolution to perform channel dimension mapping on the non-linearly mapped feature vectors to obtain gate vectors, and feed the gate vectors into the Sigmoid function to obtain the gate coefficients G. Using the gating constant coefficient G to the first Level fusion features Pixel-level weighting is performed to obtain the i-th level jump feature. .

5. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 1, characterized in that, The method further includes: The highest semantic optical features and highest semantic DSM features Each feature is mapped to a unified token space and its position is encoded to obtain optical structural features. and DSM structural features ; Optical structural features and DSM structural features Modulate them separately, and then combine the modulated bimodal token with the global token. By splicing, a unified sequence is obtained. ; Unified sequence Multi-layer Transformer encoding is performed to obtain a globally context-enhanced sequence representation. ; Sequence representation Deep optical features are obtained by splitting. and deep DSM features .

6. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 5, characterized in that, The method further includes: Input sequence features As the first The input of the first layer Transformer, the third Layer Transformer for input sequence features Perform multi-head self-attention computation, and then combine the results of the multi-head self-attention computation with the features of the input sequence. After addition, layer normalization is performed to obtain the output sequence features. ,in, Values Unified sequence For the first The input to the Transformer layer is a sequence representation. This is the output of the (L-1)th layer Transformer.

7. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 1, characterized in that, The method further includes: Deep optical features and deep DSM features Project and flatten to obtain the optical sequence and DSM sequence ; For optical sequences and DSM sequence Single-modal self-attention and global average pooling are performed respectively to obtain optical vectors. and DSM vector ; For optical vectors and DSM vector Local similarity calculations are performed to obtain modal similarity vectors. ; Modality similarity vectors are analyzed using MLP and the sigmoid function. Perform mapping to generate semantic gates ; semantic gate We modulate the cross-attention results using weights to obtain deep fusion features.

8. The multi-source remote sensing image segmentation method based on global semantic gating mechanism according to claim 7, characterized in that, The method further includes: For optical sequences and DSM sequence Perform cross-attention calculation, and then combine the cross-attention calculation result with the semantic gate. Pixel-weighted injection of optical sequences The representation z is obtained; The representation z is sequentially subjected to nonlinear encoding, layer normalization, residual connection and reconstruction to obtain deep fusion features.

9. A multi-source remote sensing image segmentation device based on a global semantic gating mechanism, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor for performing the multi-source remote sensing image segmentation method based on a global semantic gating mechanism as described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the multi-source remote sensing image segmentation method based on a global semantic gating mechanism as described in any one of claims 1-8.

Images