Remote sensing image semantic segmentation method and device, and storage medium

Abstract

This invention relates to a method, apparatus, and storage medium for semantic segmentation of remote sensing images. The method includes: inputting the remote sensing image to be segmented into the visual encoder of a trained semantic segmentation model; extracting local enhanced visual features of the remote sensing image using multiple cascaded neighborhood-aware attention modules in the visual encoder; obtaining a text description set of the target segmentation category and the corresponding text semantic embedding vector; performing semantic alignment and feature fusion of the local enhanced visual features and the text semantic embedding vector to obtain a fused feature map; and inputting the fused feature map into a segmentation head to generate the semantic segmentation result of the remote sensing image. This method combines neighborhood-aware local spatial modeling with visual-linguistic semantic alignment, taking into account both the global semantic consistency of ground objects and the accuracy of local boundary details, thereby effectively improving the segmentation performance in complex ground object scenes in remote sensing images.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image analysis technology, and in particular to a method, apparatus and storage medium for semantic segmentation of remote sensing images. Background Technology

[0002] Semantic segmentation of remote sensing images is a key task for pixel-level semantic understanding of high-resolution satellite or aerial images, and it is widely used in land use classification, ecological monitoring, crop identification and urban planning.

[0003] In related technologies, semantic segmentation methods based on deep learning are commonly used, such as fully convolutional networks, U-Net, DeepLab, and Transformer-based SegFormer models. However, existing models generally lack explicit modeling of spatial correlations in the neighborhood of pixels. Land features in remote sensing images usually have continuous spatial distribution characteristics, and adjacent pixels often belong to the same semantic category, resulting in blurred boundaries and poor internal consistency of land features in the segmentation results. This is especially true for land feature categories with complex textures and irregular boundaries, such as vegetation and water bodies.

[0004] Therefore, there is an urgent need for a remote sensing image semantic segmentation method that can simultaneously model local spatial relationships and cross-modal semantic associations to improve boundary recognition accuracy, semantic consistency, and category generalization ability. Summary of the Invention

[0005] In view of this, it is necessary to provide a method, apparatus and storage medium for semantic segmentation of remote sensing images to solve the technical problems of low boundary recognition accuracy and semantic expression ability of existing semantic segmentation models for semantic segmentation of remote sensing images.

[0006] To address the aforementioned problems, in a first aspect, the present invention provides a method for semantic segmentation of remote sensing images, comprising:

[0007] The remote sensing image to be segmented is input into the visual encoder of the trained semantic segmentation model. The local enhanced visual features of the remote sensing image are extracted using multiple cascaded neighborhood-aware attention modules in the visual encoder. The visual encoder is an encoder obtained by replacing the standard self-attention module in the basic Vision Transformer with a neighborhood-aware attention module.

[0008] Obtain a text description set for the target segmentation category, wherein the text description set contains text descriptions of at least one segmentation category;

[0009] The text description set is input into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector;

[0010] The local enhanced visual features and the text semantic embedding vector are input into the feature fusion module of the trained semantic segmentation model to perform semantic alignment and feature fusion to obtain a fused feature map.

[0011] The fused feature map is input into the segmentation head of the trained semantic segmentation model to generate the semantic segmentation result of the remote sensing image.

[0012] In one possible implementation, the attention weights of the neighborhood-aware attention module are calculated using the following formula:

[0013] ;

[0014] Where Q, K, and V are the query, key, and value matrices of the input remote sensing image, respectively; G is the Gaussian weighted neighborhood matrix, with the corresponding element values ​​calculated based on the spatial distance between corresponding pixel locations; and d is the feature dimension.

[0015] In one possible implementation, the formula for calculating the Gaussian weighted neighborhood matrix G is: Where, p i With p j These represent the spatial coordinates of the i-th and j-th pixels in the remote sensing image, respectively, and σ is the Gaussian kernel standard deviation, used to control the degree of neighborhood weight decay.

[0016] In one possible implementation, the self-similarity KK of the key matrix K is... Replace QK in the standard self-attention module .

[0017] In one possible implementation, there are M neighborhood-aware attention modules, and the Mth neighborhood-aware attention module removes the multilayer perceptron from the standard Transformer structure in the Vision Transformer.

[0018] In one possible implementation, the step of inputting the locally enhanced visual features and the text semantic embedding vector into the feature fusion module of the trained semantic segmentation model for semantic alignment and feature fusion to obtain a fused feature map includes:

[0019] The feature fusion module is used to normalize the local enhanced visual features and the text semantic embedding vector respectively to obtain the corresponding normalized visual features and normalized semantic features.

[0020] Calculate the similarity between the normalized visual features and the normalized semantic features to obtain a similarity matrix;

[0021] The local enhanced visual features are weighted and fused based on the similarity matrix to obtain the fused feature map.

[0022] In one possible implementation, the text encoder is a CLIP-based text encoder.

[0023] In one possible implementation, the trained semantic segmentation model is trained using a multi-task joint loss function, wherein the multi-task joint loss function... L The expression is as follows:

[0024] ;

[0025] in, The pixel-level cross-entropy loss is used to constrain the pixel-level classification accuracy between the semantic segmentation results and the ground truth annotations of the remote sensing images. The visual-language contrastive loss is used to constrain the semantic alignment between the locally enhanced visual features and the text semantic embedding vector in the feature space. This is the preset balance coefficient.

[0026] Secondly, the present invention also provides a remote sensing image semantic segmentation device, comprising:

[0027] The feature extraction unit is used to input the remote sensing image to be segmented into the visual encoder of the trained semantic segmentation model, and to extract the local enhanced visual features of the remote sensing image using multiple cascaded neighborhood-aware attention modules in the visual encoder. The visual encoder is an encoder obtained by replacing the standard self-attention module in the basic VisionTransformer with a neighborhood-aware attention module.

[0028] An acquisition unit is used to acquire a text description set of a target segmentation category, wherein the text description set contains text descriptions of at least one segmentation category;

[0029] The semantic extraction unit is used to input the text description set into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector;

[0030] The feature fusion unit is used to input the local enhanced visual features and the text semantic embedding vector into the feature fusion module of the trained semantic segmentation model to perform semantic alignment and feature fusion to obtain a fused feature map.

[0031] The segmentation unit is used to input the fused feature map into the segmentation head of the trained semantic segmentation model to generate the semantic segmentation result of the remote sensing image.

[0032] Thirdly, the present invention also provides a computer-readable storage medium for storing a computer-readable program or instructions, which, when executed by a processor, can implement the steps in the remote sensing image semantic segmentation method described in any of the above implementations.

[0033] The beneficial effects of this invention are:

[0034] The remote sensing image semantic segmentation method provided by this invention involves inputting the remote sensing image to be segmented into the visual encoder of a trained semantic segmentation model. The visual encoder utilizes multiple cascaded neighborhood-aware attention modules to extract local enhanced visual features of the remote sensing image. Specifically, the visual encoder replaces the standard self-attention module in the basic VisionTransformer with a neighborhood-aware attention module, explicitly strengthening the relationship between pixels and their spatial neighbors at each layer of feature extraction. This enhances the modeling of dependencies between neighboring pixels and improves the local consistency of locally enhanced visual features. A text description set for the target segmentation category is obtained, containing text descriptions for at least one segmentation category, improving the practicality and generalization ability of the trained semantic segmentation model. The text description set is then input into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector. It fully leverages the guiding role of language semantics, providing semantic guidance for enhancing local visual features. The locally enhanced visual features and textual semantic embedding vectors are input into the feature fusion module of the trained semantic segmentation model for semantic alignment and feature fusion, resulting in a fused feature map. This achieves visual-linguistic multimodal feature fusion, enabling the trained semantic segmentation model to simultaneously model local spatial relationships and semantic contextual associations between pixels within the Transformer structure, thereby improving subsequent semantic segmentation accuracy and generalization ability. The fused feature map is input into the segmentation head of the trained semantic segmentation model to generate semantic segmentation results for remote sensing images, enhancing semantic generalization ability. This combines neighborhood-aware local spatial modeling with visual-linguistic semantic alignment, balancing global semantic consistency of ground objects with boundary local detail accuracy, thus effectively improving segmentation performance in complex ground object scenes in remote sensing images. Attached Figure Description

[0035] Figure 1 A schematic flowchart of an embodiment of the remote sensing image semantic segmentation method provided by the present invention;

[0036] Figure 2 Provided by the present invention Figure 1 A schematic diagram of an embodiment of S104;

[0037] Figure 3 This is a flowchart of semantic segmentation of remote sensing images provided by the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of the remote sensing image semantic segmentation device provided by the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] In the description of the embodiments of the present invention, unless otherwise stated, "a plurality of" means two or more.

[0041] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0042] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0043] This invention provides a method, apparatus, and storage medium for semantic segmentation of remote sensing images, which will be described below.

[0044] The execution subject of the remote sensing image semantic segmentation method in this application embodiment can be the remote sensing image semantic segmentation device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the remote sensing image semantic segmentation device. The remote sensing image semantic segmentation device can be implemented in hardware or software. The UE can be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0045] Figure 1 This is a schematic flowchart of an embodiment of the remote sensing image semantic segmentation method provided by the present invention, as shown below. Figure 1 As shown, the semantic segmentation methods for remote sensing images include:

[0046] S101. Input the remote sensing image to be segmented into the visual encoder of the trained semantic segmentation model, and use multiple cascaded neighborhood-aware attention modules in the visual encoder to extract local enhanced visual features of the remote sensing image. The visual encoder is an encoder obtained by replacing the standard self-attention module in the basic Vision Transformer with a neighborhood-aware attention module.

[0047] The remote sensing image to be segmented refers to the original remote sensing image with higher resolution obtained from the original remote sensing image that needs to be semantically segmented and then preprocessed. For example, the size of the remote sensing image with higher resolution is 1024×1024×3, that is, the length and width of the original remote sensing image are both 1024 pixels. Then, the original remote sensing image with higher resolution is preprocessed, including but not limited to merging and normalization, size adjustment and data augmentation.

[0048] A trained semantic segmentation model refers to a pre-trained neural network model used for semantic segmentation of remote sensing images. In this embodiment, it is a vision-language joint model, which is trained on a remote sensing dataset using a joint objective function that includes segmentation loss and alignment loss. It includes a visual encoder, a text encoder, a feature fusion module, and a segmentation head. The visual encoder and text encoder are connected in parallel to process inputs of different modalities, such as remote sensing images and target segmentation category labels, and output their respective feature representations. The output feature representations are then simultaneously input into the feature fusion module, where semantic alignment and feature fusion are performed. The output of the feature fusion module is connected to the segmentation head for upsampling and classification to generate the final semantic segmentation map.

[0049] The basic Vision Transformer (ViT) is an encoder that extracts visual features using standard self-attention modules in intermediate layers. In this embodiment, the visual encoder is an improved model of the intermediate layers in the basic Vision Transformer. The improvement involves replacing the standard self-attention modules in the intermediate layers with multiple cascaded neighborhood-aware attention (NA-Attention) modules. The basic Vision Transformer can be a ViT with different weight numbers, for example, a ViT with weight number ViT-B / 32. The basic Vision Transformer (ViT) is a Transformer architecture, including a first-layer image patch partitioning and embedding module, intermediate-layer standard self-attention modules, and a tail-layer multi-scale output module. The encoding process of the basic Vision Transformer for remote sensing images is as follows:

[0050] The first-layer image segmentation and embedding module divides the input remote sensing image into image patches and performs linear projection and positional encoding to obtain the input feature sequence. For example, the input image (size H×W×3) is divided into fixed-size non-overlapping patches (e.g., 16×16 pixels). Each patch is flattened into a vector, and then linear projection and positional encoding are performed. That is, each patch vector is mapped to the feature dimension D through a learnable linear layer to obtain patch embeddings. At the same time, a learnable positional encoding (e.g., 2D) is added to preserve spatial information. A learnable [class] token can also be added at the beginning of the sequence for classification.

[0051] The standard self-attention module in the intermediate layer consists of L identical Transformer Blocks stacked together. Each Block contains two Multi-Head Self-Attention (MSA) layers and a feedforward network layer (MLP). The MSA divides the input feature sequence into multiple heads, independently calculates Q, K, and V on each head, and performs attention operations. Finally, the output is concatenated, as shown in the formula:

[0052] ;

[0053] A feedforward network layer (MLP) is typically a two-layer feedforward neural network that performs a non-linear transformation on the features at each location.

[0054] The tail layer's multi-scale feature output module is used to extract and output multi-scale visual features from different depths of the standard self-attention module in the middle layer.

[0055] The neighborhood-aware attention module is an encoder that employs a neighborhood-aware attention mechanism. This mechanism embeds spatial locality priors into the Transformer architecture in a learnable and differentiable manner, enabling the model to prioritize and analyze local regions when understanding images. It then combines this with the global context to make a comprehensive judgment, ensuring that while focusing on content similarity, the model prioritizes pixels within their spatial neighborhood, thus enhancing its ability to model local structures (such as the internal uniformity of ground features and boundary contours).

[0056] The improvement of the visual encoder in this embodiment is that the standard self-attention module in the middle layer is replaced by a neighborhood-aware attention module, while the first-layer image patching network and the last-layer multi-scale output module remain unchanged.

[0057] The inventors discovered that current mainstream methods based on deep learning models (such as U-Net, DeepLab, Transformer, etc.) lack modeling of spatial correlations in pixel neighborhoods, leading to blurred feature boundary segmentation. For example, traditional attention mechanisms are based on the global self-attention Transformer model, which treats all pixels equally when calculating attention, ignoring spatial proximity priors. This results in blurred feature boundaries and poor internal consistency in the segmentation results, ignoring the spatial dependence between neighboring pixels and affecting boundary accuracy. Therefore, this application uses a neighborhood-aware attention module to replace the standard self-attention module, enabling the visual features extracted by the visual encoder to have better local spatial consistency, which is beneficial to improving the clarity of boundary recognition.

[0058] Specifically, through cascaded neighborhood perception modules, the visual encoder explicitly strengthens the relationship between pixels and their spatial neighborhoods at each layer of feature extraction, enhances the modeling of the dependency relationship of neighboring pixels, and improves the local consistency of the locally enhanced visual features extracted by the neighborhood perception attention module.

[0059] S102. Obtain a text description set of the target segmentation category, wherein the text description set contains text descriptions of at least one segmentation category.

[0060] The text description set of the target segmentation category refers to the label of the target segmentation category, which can be predefined according to the actual task requirements. Its text description may not be completely consistent with the category used during model training.

[0061] Specifically, a list of target segmentation categories for remote sensing images can be defined based on actual task requirements. For example, for land use classification tasks, the text description set can be defined as: [buildings, main roads, woodland, farmland, rivers]. This list can be dynamically modified and expanded according to the application scenario after model training, without retraining the model.

[0062] Understandably, in this embodiment, the semantic category of segmentation is defined as a set of readable text rather than fixed numerical labels. This transforms the segmentation task from a closed, static mode to an open, dynamic, and semantically definable mode. Users can flexibly specify the categories to be identified by modifying the text description set, thereby improving the practicality and generalization ability of the trained semantic segmentation model.

[0063] S103. Input the text description set into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector.

[0064] Among them, text encoders are semantic feature extractors based on Transformer and specifically pre-trained on image and text data in the remote sensing field, such as CLIP series text encoders, ALIGN text encoders, BERT series encoders, or FILIP / Florace text encoders.

[0065] The inventors discovered that existing models fail to effectively incorporate linguistic semantic information for guidance. Traditional segmentation models are typically trained on closed sets of categories, making it difficult to adapt to the recognition needs of open vocabulary or newly added segmentation categories.

[0066] Specifically, the text description set is input into the text encoder to obtain the text semantic embedding vector, which fully leverages the guiding role of language semantics and provides semantic guidance for local enhancement of visual features, facilitating the semantic alignment of subsequent visual features.

[0067] S104. Input the local enhanced visual features and the text semantic embedding vector into the feature fusion module of the trained semantic segmentation model to perform semantic alignment and feature fusion to obtain a fused feature map.

[0068] The feature fusion module is used to align and fuse local enhanced visual features with text semantic embedding vectors.

[0069] Specifically, the feature fusion module can be a cross-attention network, where text semantic embeddings serve as queries and locally enhanced visual features serve as keys and values; or it can be a feature modulation network, using text semantic embedding vectors to generate modulation parameters to perform affine transformations on the locally enhanced visual features. The feature fusion module can also normalize the locally enhanced visual features and text semantic embedding vectors and calculate their similarity matrix. This similarity matrix is ​​then used as pixel-level semantic guidance weights to perform weighted fusion of visual features to obtain a fused feature map. This achieves pixel-level visual-language semantic alignment, thereby realizing visual-language multimodal feature fusion. This allows the trained semantic segmentation model to simultaneously model the local spatial relationships and semantic contextual associations between pixels within the Transformer structure, thus improving subsequent semantic segmentation accuracy and generalization ability.

[0070] S105. Input the fused feature map into the segmentation head of the trained semantic segmentation model to generate the semantic segmentation result of the remote sensing image.

[0071] The segmentation head typically consists of a lightweight decoder containing several transposed convolutional or bilinear upsampling layers. For example, the segmentation head includes an upsampling structure and a classification network.

[0072] Specifically, the segmentation head can be used to multiply the fused feature map by contrast scaling parameters, spatial structure restoration, bilinear interpolation, and pixel prediction head processing to generate a pixel-level classification probability map, thereby obtaining the final semantic segmentation result. Understandably, this embodiment, by performing semantic segmentation on the fused feature map, enhances semantic generalization ability, achieving a combination of neighborhood-aware local spatial modeling and visual-linguistic semantic alignment. It balances the global semantic consistency of ground objects with the accuracy of local boundary details, thus effectively improving the segmentation performance in complex ground object scenes in high-resolution remote sensing imagery. It is suitable for automated semantic segmentation and ground object recognition tasks in multi-source remote sensing imagery.

[0073] In summary, the remote sensing image semantic segmentation method provided in this invention involves inputting the remote sensing image to be segmented into the visual encoder of a trained semantic segmentation model. The visual encoder utilizes multiple cascaded neighborhood-aware attention modules to extract local enhanced visual features of the remote sensing image. Specifically, the visual encoder replaces the standard self-attention module in the basic Vision Transformer with a neighborhood-aware attention module, explicitly strengthening the relationship between pixels and their spatial neighborhoods at each layer of feature extraction. This enhances the modeling of neighborhood pixel dependencies and improves the local consistency of the enhanced visual features. A text description set for the target segmentation category is obtained, containing text descriptions for at least one segmentation category, improving the practicality and generalization ability of the trained semantic segmentation model. The text description set is input into the text encoder of the trained semantic segmentation model to obtain corresponding text semantic embedding vectors, fully leveraging the guiding role of language semantics and providing semantic guidance for local enhanced visual features. Finally, the local enhanced visual features and text semantic embedding vectors are input into the training... A good semantic segmentation model's feature fusion module performs semantic alignment and feature fusion to obtain a fused feature map, realizing visual-linguistic multimodal feature fusion. This allows the trained semantic segmentation model to simultaneously model the local spatial relationships and semantic context associations between pixels within the Transformer structure, thereby improving subsequent semantic segmentation accuracy and generalization ability. The fused feature map is input into the segmentation head of the trained semantic segmentation model to generate semantic segmentation results for remote sensing images, enhancing semantic generalization ability. It combines neighborhood-aware local spatial modeling with visual-linguistic semantic alignment, taking into account both the global semantic consistency of ground objects and the accuracy of local boundary details, thus effectively improving the segmentation performance in complex ground object scenes in remote sensing images.

[0074] In some embodiments of the present invention, the attention weights of the neighborhood-aware attention module are calculated using the following formula:

[0075] ;

[0076] Where Q, K, and V are the query, key, and value matrices of the input remote sensing image, respectively; G is the Gaussian weighted neighborhood matrix, with the corresponding element values ​​calculated based on the spatial distance between corresponding pixel locations; and d is the feature dimension.

[0077] In this embodiment, the neighborhood-aware attention module adds a Gaussian-weighted neighborhood matrix G to the standard attention module. The element values ​​in G are calculated based on the spatial distance between corresponding pixel locations in the remote sensing image. Therefore, while retaining the original attention calculation logic (QK... Or KK Based on this, an additional matrix G based on spatial distance is introduced, which is used to modify and guide the final attention weight distribution through addition.

[0078] Specifically, in this embodiment, a neighborhood-aware attention mechanism is introduced during the encoding process of the visual encoder. The spatial relationship between pixels is modeled by a Gaussian weighted neighborhood matrix. The introduction of the Gaussian weighted neighborhood matrix ensures that geographically nearby pixels automatically receive higher association weights when calculating attention. The spatial distance information between pixels is injected as a priori into the calculation of attention weights, making the model pay more attention to the local neighborhood, enhancing the ability to perceive local features, and enhancing local spatial consistency. It effectively models the inherent spatial continuity prior of remotely sensed ground objects, thereby helping to improve the boundary segmentation effect.

[0079] In some embodiments of the present invention, the formula for calculating the Gaussian weighted neighborhood matrix G is as follows: Where, p i With p j These represent the spatial coordinates of the i-th and j-th pixels in the remote sensing image, respectively, and σ is the Gaussian kernel standard deviation, used to control the degree of neighborhood weight decay.

[0080] in, It can be obtained by calculating the square of the Euclidean distance between the coordinates of all position pairs.

[0081] Specifically, by introducing a Gaussian weighted neighborhood matrix G, the local weight decay between pixels is controlled, so that the attention mechanism retains neighborhood consistency when focusing on the target area, thereby enhancing the local consistency of the visual features of the remote sensing image.

[0082] In some embodiments of the present invention, the self-similarity KK of the key matrix K is... Replace QK in the standard self-attention module .

[0083] Specifically, the self-similarity KK of the key matrix K Replace QK in the standard self-attention module That is, let the query matrix Q be equal to the key matrix K, that is, the attention weights of the neighborhood-aware attention module are calculated using the following formula:

[0084] ;

[0085] Understandably, in this embodiment, self-bonded attention is used, which uses the similarity between the pixel's own features and the features of its neighbors to model local relationships. This helps to automatically aggregate semantically similar neighboring pixels (such as pixels belonging to the same farmland) together, thereby implicitly enhancing the local consistency of features to a certain extent and providing a more compact feature representation for subsequent segmentation.

[0086] In some embodiments of the present invention, there are M neighborhood-aware attention modules, and the Mth neighborhood-aware attention module removes the multilayer perceptron in the standard Transformer structure of the Vision Transformer.

[0087] Specifically, in the neighborhood-aware attention module, the feedforward network layer, i.e., the multilayer perceptron, in the last neighborhood-aware attention module is removed. This avoids the unnecessary global computation introduced by the multilayer perceptron structure in the standard self-attention module, reducing efficiency and local consistency, decreasing computational overhead, and preventing the destruction of local consistency by MLP, thereby better preserving the local spatial relationships established by neighborhood-aware attention. It also simplifies the structure of the neighborhood-aware attention module.

[0088] In one specific implementation, the 1-(M-1)th neighborhood-aware attention module is a key-value Gaussian attention encoder module, and the Mth neighborhood-aware attention module is a feedforward-free Gaussian attention module.

[0089] In some embodiments of the present invention, such as Figure 2 As shown, step S104 includes:

[0090] S201. The feature fusion module is used to normalize the local enhanced visual features and the text semantic embedding vector respectively to obtain the corresponding normalized visual features and normalized semantic features.

[0091] S202. Calculate the similarity between the normalized visual features and the normalized semantic features to obtain a similarity matrix;

[0092] S203. The local enhanced visual features are weighted and fused based on the similarity matrix to obtain the fused feature map.

[0093] The feature fusion module includes: a first linear transformation layer for transforming local enhanced visual features to dimension D; a second linear transformation layer for transforming text semantic embedding vectors to dimension D; a normalization layer for performing L2 normalization on the transformed visual features and text features respectively; a similarity calculation layer for calculating the matrix product of normalized visual features and normalized semantic features to obtain a similarity matrix; a weight generation layer for processing the similarity matrix through a Softmax function to obtain pixel-level semantic guidance weights; and a weighted fusion layer for semantically modulating the local enhanced visual features with pixel-level semantic guidance weights, thereby generating a fused feature map that simultaneously integrates spatial structure information and text semantic priors.

[0094] Specifically, firstly, the local enhanced visual features and text semantic embedding vectors are dimensionally aligned and normalized, and a similarity matrix is ​​calculated between them. This similarity matrix characterizes the degree of matching between the local enhanced visual features at each pixel location and the text semantic embedding vectors of each target segmentation category. Subsequently, the similarity matrix is ​​normalized along the dimension of the target segmentation category, converting it into pixel-level semantic guidance weights to ensure the comparability of the relative importance between different target segmentation categories. Based on the obtained pixel-level semantic guidance weights, the corresponding semantic features at each pixel location are weighted and aggregated to form a semantic guidance vector corresponding to that pixel location. This semantic guidance vector is then applied to the local enhanced visual features, such as through channel-wise modulation, to enhance the local enhanced visual features, thereby highlighting feature responses highly relevant to the target semantics and suppressing irrelevant or noisy information. This accurately maps the abstract semantics of the text to each pixel of the remote sensing image, providing interpretable semantic guidance, which is beneficial for improving semantic segmentation accuracy and the model's generalization ability.

[0095] In some embodiments of the present invention, the text encoder is a CLIP-based text encoder.

[0096] The text encoder can be a RemoteCLIP model pre-trained on a general CLIP model (Contrastive Language–Image Pre-training) for remote sensing image-text pairs. The text encoder, pre-trained on remote sensing image-text pairs, generates text embeddings that are not general semantics but rather enhance specialized semantic features related to the visual patterns of remote sensing land features. This improves the domain specificity and accuracy of the text semantic embedding vectors as alignment anchors, enabling subsequent similarity calculations in the feature fusion module to more accurately distinguish different land feature categories.

[0097] Specifically, the text encoder using the CLIP architecture improves the accuracy of text semantic embedding vectors as alignment anchors with enhanced visual features, enhances the model's semantic understanding and open vocabulary segmentation capabilities, and improves the robustness and generalization of semantic guidance, enabling the model to adapt to open vocabulary segmentation tasks.

[0098] In some embodiments of the present invention, the trained semantic segmentation model is trained using a multi-task joint loss function, wherein the multi-task joint loss function... L The expression is as follows:

[0099] ;

[0100] in, The pixel-level cross-entropy loss is used to constrain the pixel-level classification accuracy between the semantic segmentation results and the ground truth annotations of the remote sensing images. The visual-language contrastive loss is used to constrain the semantic alignment between the locally enhanced visual features and the text semantic embedding vector in the feature space. This is the preset balance coefficient.

[0101] Specifically, a multi-task joint loss function is used to train the semantic segmentation model to be trained, thereby obtaining a trained semantic segmentation model. The expression of the multi-task joint loss function is as follows:

[0102] ;

[0103] The loss is pixel-level cross-entropy, used to constrain segmentation accuracy. The visual-linguistic contrastive loss is used to maintain consistency between the visual and semantic spaces. It can be implemented using a similarity-based contrastive loss function (such as InfoNCE loss) to narrow the distance between matched visual-text feature pairs. The model parameters of the semantic segmentation model to be trained are optimized through the backpropagation algorithm, so that visual features and linguistic features are consistent in the semantic space. This makes the modeling have global semantic consistency and local spatial detail perception capabilities, improving boundary recognition accuracy and semantic expression capabilities.

[0104] In one specific implementation, such as Figure 3 The diagram shows a flowchart for semantic segmentation of remote sensing images. The specific process is as follows:

[0105] The preprocessed remote sensing image is input into the visual encoder module. The Vision Transformer (ViT) is used as the visual encoder to extract features from the input image: the input image is divided into fixed-size patches, and each patch is embedded into a high-dimensional feature vector through convolutional or linear layers; two-dimensional position encoding is added to preserve spatial location information; after passing through multiple Transformer encoding modules, the visual encoder outputs features at different levels, such as F1 to F4. F1 to F4 are fed into a neighborhood-aware attention module for local feature enhancement, resulting in corresponding local enhancement features f1-f4. f1-f4 employ cascaded upsampling and convolution operations to gradually restore spatial resolution, and feature concatenation achieves complementarity between low-level detail information and high-level semantic information. Finally, a high-resolution local enhancement visual feature F1 with the same size as the input image is obtained.

[0106] The text description set is feature-encoded by a text encoder to obtain the corresponding text semantic embedding vector T1. The text encoder is a RemoteCLIP structure.

[0107] The local enhanced visual features and the text semantic embedding vector T1 are matched and aligned in the feature fusion module: First, the dimensions of T1 and F1 are unified through linear transformation. For example, the local enhanced visual feature F1 is normalized by L2 and its similarity to the L2 normalized text semantic embedding vector is calculated in the same semantic space, resulting in the similarity matrix S=Norm(F). Norm(T) T Where F is the normalized visual feature and T is the normalized semantic feature; finally, the similarity matrix is ​​mapped as the pixel-level semantic guiding weight to obtain the fused feature map.

[0108] The fused feature map is input into the segmentation head. The similarity score is multiplied by the contrast scaling parameter (logit_scale), and spatial structure restoration is performed using the position transformation matrix provided by Torch. Bilinear interpolation and pixel prediction head processing are then performed to generate a pixel-level classification probability map. This map is then input into the softmax layer to generate a pixel-level classification probability map, thus obtaining the final semantic segmentation result.

[0109] In another specific implementation, semantic segmentation experiments were conducted using the GID dataset, where all images are 1024×1024 pixels in size. Table 1 shows the performance of the dataset on different models. The table compares the semantic segmentation results of the semantic segmentation model trained in this application with existing segmentation models. As can be seen from Table 1, the method of this invention achieves 11.1 percentage points higher mIou (mIoU, average intersection-union ratio) than SFA-Net. Table 1 also clearly shows that the segmentation result of the method of this invention is more complete, and the segmentation effect is closest to the labeled image.

[0110] Table 1. Comparison of semantic segmentation results between the trained semantic segmentation model and existing segmentation models.

[0111]

[0112] To better implement the remote sensing image semantic segmentation method in this embodiment of the invention, based on the remote sensing image semantic segmentation method, correspondingly, as follows: Figure 4 As shown, this embodiment of the invention also provides a remote sensing image semantic segmentation device, the remote sensing image semantic segmentation device 400 including:

[0113] The feature extraction unit 401 is used to input the remote sensing image to be segmented into the visual encoder of the trained semantic segmentation model, and to extract the local enhanced visual features of the remote sensing image using multiple cascaded neighborhood-aware attention modules in the visual encoder. The visual encoder is an encoder obtained by replacing the standard self-attention module in the basic VisionTransformer with a neighborhood-aware attention module.

[0114] The acquisition unit 402 is used to acquire a text description set of the target segmentation category, wherein the text description set contains a text description of at least one segmentation category;

[0115] The semantic extraction unit 403 is used to input the text description set into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector;

[0116] The feature fusion unit 404 is used to input the local enhanced visual features and the text semantic embedding vector into the feature fusion module of the trained semantic segmentation model to perform semantic alignment and feature fusion to obtain a fused feature map.

[0117] The segmentation unit 405 is used to input the fused feature map into the segmentation head of the trained semantic segmentation model to generate the semantic segmentation result of the remote sensing image.

[0118] The remote sensing image semantic segmentation device 400 provided in the above embodiments can realize the technical solutions described in the above remote sensing image semantic segmentation method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above remote sensing image semantic segmentation method embodiments, and will not be repeated here.

[0119] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the remote sensing image semantic segmentation methods provided in the above-described method embodiments.

[0120] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0121] The remote sensing image semantic segmentation method, apparatus, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for semantic segmentation of remote sensing images, characterized in that, include: The remote sensing image to be segmented is input into the visual encoder of the trained semantic segmentation model. The local enhanced visual features of the remote sensing image are extracted using multiple cascaded neighborhood-aware attention modules in the visual encoder. The visual encoder is an encoder obtained by replacing the standard self-attention module in the basic Vision Transformer with a neighborhood-aware attention module. This includes replacing the standard self-attention module in the intermediate layer with a neighborhood-aware attention module, while keeping the image block network in the first layer and the multi-scale output module in the last layer unchanged. The neighborhood-aware attention module modifies the attention weights by introducing a Gaussian-weighted neighborhood matrix G; the attention weights of the neighborhood-aware attention module are calculated using the following formula: Where K and V are the key and value matrices of the input remote sensing image, respectively; G is the Gaussian weighted neighborhood matrix, and the corresponding element values ​​are calculated based on the spatial distance between corresponding pixel positions; and d is the feature dimension. Obtain a text description set for the target segmentation category, wherein the text description set contains text descriptions of at least one segmentation category; The text description set is input into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector; The local enhanced visual features and the text semantic embedding vector are input into the feature fusion module of the trained semantic segmentation model for semantic alignment and feature fusion to obtain a fused feature map. This includes: normalizing the local enhanced visual features and the text semantic embedding vector using the feature fusion module to obtain corresponding normalized visual features and normalized semantic features; calculating the similarity between the normalized visual features and the normalized semantic features to obtain a similarity matrix; and performing weighted fusion of the local enhanced visual features based on the similarity matrix to obtain the fused feature map. The fused feature map is input into the segmentation head of the trained semantic segmentation model to generate the semantic segmentation result of the remote sensing image. The trained semantic segmentation model is trained using a multi-task joint loss function. L The expression is as follows: ； in, The pixel-level cross-entropy loss is used to constrain the pixel-level classification accuracy between the semantic segmentation results and the ground truth annotations of the remote sensing images. The visual-language contrastive loss is used to constrain the semantic alignment between the locally enhanced visual features and the text semantic embedding vector in the feature space. This is the preset balance coefficient.

2. The method of claim 1, wherein, The neighborhood awareness attention module has M units, and the Mth neighborhood awareness attention module removes the multilayer perceptron from the standard Transformer structure in the Vision Transformer.

3. The method of claim 1, wherein, The text encoder is a CLIP-based text encoder.

4. A device for semantic segmentation of remote sensing images, characterized in that it comprises: include: The feature extraction unit is used to input the remote sensing image to be segmented into the visual encoder of the trained semantic segmentation model, and to extract the local enhanced visual features of the remote sensing image using multiple cascaded neighborhood-aware attention modules in the visual encoder. The visual encoder is an encoder obtained by replacing the standard self-attention module in the basic Vision Transformer with a neighborhood-aware attention module, including: replacing the standard self-attention module in the intermediate layer with a neighborhood-aware attention module, while keeping the image block network in the first layer and the multi-scale output module in the last layer unchanged. The neighborhood-aware attention module modifies the attention weights by introducing a Gaussian-weighted neighborhood matrix G; the attention weights of the neighborhood-aware attention module are calculated using the following formula: Where K and V are the key and value matrices of the input remote sensing image, respectively; G is the Gaussian weighted neighborhood matrix, and the corresponding element values ​​are calculated based on the spatial distance between corresponding pixel positions; and d is the feature dimension. An acquisition unit is used to acquire a text description set of a target segmentation category, wherein the text description set contains text descriptions of at least one segmentation category; The semantic extraction unit is used to input the text description set into the text encoder of the trained semantic segmentation model to obtain the corresponding text semantic embedding vector; The feature fusion unit is used to input the local enhanced visual features and the text semantic embedding vector into the feature fusion module of the trained semantic segmentation model, perform semantic alignment and feature fusion, and obtain a fused feature map. The unit includes: normalizing the local enhanced visual features and the text semantic embedding vector using the feature fusion module to obtain corresponding normalized visual features and normalized semantic features; calculating the similarity between the normalized visual features and the normalized semantic features to obtain a similarity matrix; and performing weighted fusion of the local enhanced visual features based on the similarity matrix to obtain the fused feature map. The segmentation unit is used to input the fused feature map into the segmentation head of the trained semantic segmentation model to generate the semantic segmentation result of the remote sensing image. The trained semantic segmentation model is trained using a multi-task joint loss function. L The expression is as follows: ； wherein, is a pixel-level cross-entropy loss for constraining the pixel-level classification accuracy between the semantic segmentation result and the true label of the remote sensing image, is a vision-language contrastive loss for constraining the semantic alignment degree of the local enhanced visual feature and the text semantic embedding vector in the feature space, is a preset balance coefficient.

5. A computer readable storage medium, characterized in that, Used to store computer-readable programs or instructions, which, when executed by a processor, can implement the steps in the remote sensing image semantic segmentation method according to any one of claims 1 to 3.

Images