A remote sensing image semantic segmentation method based on visual structure information guidance

Abstract

The present invention discloses a remote sensing image semantic segmentation method based on visual structural information guidance, including the following operations: selecting a remote sensing data set and a tree mapping data set as an experimental data set; based on the experimental data set, selecting a visual basic model, and extracting a visual structural image from the remote sensing image of the experimental data set through the visual basic model; based on the visual structural image, constructing a feature co-embedding module to extract a visual feature image in the visual structural image; based on the visual feature image, constructing a structural heuristic feature fusion module to perform feature fusion and structural prediction on the visual feature image to obtain a processed experimental data set; constructing a target loss function to predict the processed experimental data set. The method of the present invention extracts a structural image through a visual model, constructs a module to extract and fuse features, and predicts the processed data set, thereby improving the accuracy and robustness of remote sensing image segmentation.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image semantic segmentation, and in particular to a remote sensing image semantic segmentation method based on guidance of visual structure information. Background Art

[0002] In recent years, with the rapid development of earth observation sensors, remote sensing (RS) data fusion in multimodal semantic segmentation has become a research focus. Land cover is an important attribute of the earth's surface, as high spatial resolution land cover maps are essential for effectively monitoring human activities and assessing surface conditions. Compared with satellite methods, very high resolution (VHR) remote sensing (RS) land cover maps can be drawn using aerial images, so that complex terrain details can be captured from the images. When using remote sensing (RS) to draw land cover maps, it is necessary to assign the most likely class label to each image pixel from a limited set, which is similar to semantic segmentation.

[0003] In the existing technology, multimodal data fusion is facing severe challenges due to the differences in image acquisition mechanisms between different sensors, which leads to a series of alignment problems. With the continuous advancement of deep convolutional neural networks (CNNs), various studies have applied a series of CNN-based semantic segmentation architectures to land cover mapping, such as fully convolutional networks (FCNs), U-Net, SegNet, and DeepLab. These methods improve features by extending and deepening the original network structure, integrating multi-scale information, and incorporating attention mechanisms, and can be customized for unique obstacles found in remote sensing (RS) scenes. Although CNNs are good at capturing local contexts and can maintain translation invariance through convolution, they are insufficient in directly representing global semantic interactions and contextual information. In recent years, transformer models have achieved outstanding results in the field of natural language processing (NLP), which has led to their widespread development in computer vision applications such as image classification and semantic segmentation. Compared with CNN-based architectures, the Visual Transformer (ViT) has been pre-trained on large-scale datasets and can show higher accuracy. Therefore, the researchers introduced a Transformer based model for RS semantic segmentation, which can highlight its ability to capture long-range dependencies in VHR RGB images.

[0004] In view of this, the multimodal fusion framework VSGNet proposed in the present invention can perform RS semantic segmentation, and the visual structure guidance derived from the visual basic model can perform accurate segmentation without the use of auxiliary sensors. The cross-modal collaborative network can be used for feature embedding, which integrates convolutional neural networks and visual converters, and can capture local information and long-range dependencies of the input modality. The multi-scale cross-modal feature fusion method includes fusion enhancement and feature recalibration modules, which emphasize the adaptive multi-scale interaction of different complementary cues between each modality, and can suppress the noise and uncertainty in RS data, thereby improving the accuracy and robustness of remote sensing image segmentation. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes a remote sensing image semantic segmentation method based on visual structure information guidance, RS semantic segmentation is performed through the multimodal fusion framework VSGNet, and the visual structure guidance derived from the visual basic model can be used for accurate segmentation without the use of auxiliary sensors. Feature embedding is performed using a cross-modal collaborative network, which integrates convolutional neural networks and visual converters, and can capture local information and long-range dependencies of the input modality. The multi-scale cross-modal feature fusion method adopted includes fusion enhancement and feature recalibration modules, which emphasize the adaptive multi-scale interaction of different complementary cues between each modality, and can suppress the noise and uncertainty in RS data, thereby improving the accuracy and robustness of remote sensing image segmentation.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] The present invention is a remote sensing image semantic segmentation method based on visual structure information guidance, comprising the following operations:

[0008] Remote sensing datasets and tree mapping datasets were selected as experimental datasets;

[0009] Based on the experimental data set, a visual basic model is selected, and a visual structure image is extracted from the remote sensing image of the experimental data set through the visual basic model;

[0010] Based on the visual structure image, construct a feature co-embedding module to extract a visual feature image in the visual structure image;

[0011] Based on the visual feature image, construct a structure-inspired feature fusion module to perform feature fusion and structure prediction on the visual feature image to obtain a processed experimental data set;

[0012] A target loss function is constructed to predict the processed experimental data set.

[0013] A further improvement of the present invention is that after obtaining the experimental data set, data enhancement is performed on the remote sensing images in the experimental data set: the remote sensing images are windowed and cropped, that is, sub-regions are cropped on the remote sensing images according to a size of 512×512 and a step size of 256×256, and the local feature ratio in the sub-region is highlighted; the remote sensing images are randomly flipped horizontally; the remote sensing images are randomly scaled, and the scaling ratio range is 0.5-1.75.

[0014] A further improvement of the present invention is that the visual basic model includes a segmentation large model and a depth large model, the visual structure image refers to the additional modal information learned by the visual basic model from the remote sensing images of the experimental data set, the remote sensing image is input into the visual basic model and the model parameters are frozen, and only model inference is performed to obtain the visual structure image.

[0015] A further improvement of the present invention is that: the feature co-embedding module includes CNN and ViT, CNN includes 3×3 convolution, BN layer and GELU activation function, and ViT uses Dinov2 pre-training model;

[0016] Visual feature image W generated by CNN c The expression is:

[0017] W c =Conv 3×3 (σ(Conv 3×3 (X)))

[0018] Where W c is the visual feature image extracted by CNN from the remote sensing image and the visual structure image; Conv3x3 is a 3x3 convolutional layer; σ is the GELU activation function; X is the input remote sensing image;

[0019] When the ViT encoder processes the visual structure image, it tokenizes the visual structure image through the patch embedding layer to generate a visual feature image; the image passes through multiple Transformer encoding blocks, generating output data of different sizes at each layer, and there is a Patch Merging layer after each Transformer encoding block, which halves the spatial dimension and forwards the output data to the next deeper Transformer encoding block; the ViT weights obtained from Dinov2 are used to pre-train Dinov2 on the experimental dataset, and the visual feature image W embedded by ViT v The expression is:

[0020] W v =F ViT (X)

[0021] Where W vis the visual feature image extracted by ViT from the remote sensing image and the visual structure image; F ViT is the ViT feature extractor; X is the input remote sensing image;

[0022] The visual feature images of the two branches are added together and imported into the subsequent structure-heuristic feature fusion module for further processing:

[0023]

[0024] In the formula, x rgb is the remote sensing image feature; and is the visual feature image extracted by ViT from remote sensing images and visual structure images; x structure It is the visual structural feature; and It is a visual feature image extracted by CNN from remote sensing images and visual structure images.

[0025] A further improvement of the present invention is that: the structure-inspired feature fusion module includes a fusion enhancement and a feature recalibration module, the fusion enhancement includes a self-attention module and a large-kernel depth-separable convolution, and the fusion enhancement is used to promote the fusion of visual feature images; the feature recalibration module includes a channel attention and a spatial attention mechanism, and the feature recalibration module is used to improve and calibrate the visual feature image;

[0026] Fusion enhancement combines remote sensing images and visual structure images to enhance object positioning capabilities. The self-attention module fuses cross-modal inputs. The large-kernel deep separable convolution performs deep convolution on visual structure features. The convolution results are used as attention weights to reweight remote sensing image features and embed visual structure features into remote sensing image features. The large-kernel deep separable convolution is used to enhance target features in remote sensing image features.

[0027] The feature recalibration module performs feature calibration and denoising through dual-branch cross-modal interaction, applies cross-modal interaction on remote sensing images and visual structure images, and applies channel attention to convert visual feature images into attention vectors in the embedding space. The generated visual feature images are summarized and the summarized visual feature images are used as the calibration feature images under the same mode.

[0028] A further improvement of the present invention is that when the fusion enhancement merges the remote sensing image and the visual structure image, a downsampling operation is performed on the query, the query is derived from the remote sensing image and the visual structure image, K and V are derived from the remote sensing image, and the expression of the query Q is:

[0029] Q=Linear(Pool k×k ([X rgb ,Xstructure ]))

[0030] Where Q is the query, X rgb is the remote sensing image feature; X structure is the visual structure feature; [] is the connectivity in the channel dimension; Pool k×k is a k×k adaptive average pooling operation; Linear is a linear transformation;

[0031] The expression of K is:

[0032] K=Linear(X rgb )

[0033] Where, X rgb is the remote sensing image feature; Linear is the linear transformation; K is the key,

[0034] The expression of V is:

[0035] V=Linear(X rgb )

[0036] Where, X rgb is the remote sensing image feature; Linear is the linear transformation; V is the value,

[0037] The process of the feature recalibration module is:

[0038]

[0039] Where UP is the up sampling operation; Q is the query; is the output of a branch of fusion enhancement; V is the value; Softmax is the Softmax function; K is the key; C is the hidden layer; T is the transpose; d is the dimension;

[0040] Considering local information, feature weights are reshaped through large kernel depth-separable convolution, and remote sensing image information and mask information are embedded:

[0041]

[0042] In the formula, DConv is the fusion enhanced two-branch output; k×k is the depth convolution; k×k is the kernel size of the depth convolution; Linear is the linear transformation; X structure is the visual structure feature; X rgb is the remote sensing image feature; ⊙ is element-wise multiplication;

[0043] The self-attention module and the large-core depth-separable convolution function are connected and mapped to each modality respectively to form a fused enhanced output, i.e.

[0044]

[0045] Where, X FE is the output of fusion enhancement; Concat is the cascade operation; It is a branch output for fusion enhancement; It is the fusion enhanced two-branch output.

[0046] 8. A further improvement of the present invention is that when the feature recalibration module performs feature calibration and noise reduction through dual-branch cross-modal interaction, the visual feature image is superimposed:

[0047] X c =Concat(X structure ,X rgb )

[0048] Where, X c is the initial fusion feature; Concat is the cascade operation; X structure is the visual structure feature; X rgb is the remote sensing image feature;

[0049] The visual feature image is converted into the embedding space as the attention vector through the spatial attention mechanism. The expression of the attention vector is:

[0050] W rgb =σ(Mlp(F(X c )))

[0051] Where W rgb is the attention vector; σ is the activation function; Mlp is a fully connected layer, which contains two 1x1 convolutional layers; F is a long-distance context modeling operation; X c It is the initial fusion feature;

[0052] The expression of the long-distance context modeling operation F is:

[0053]

[0054] Where F is the long-distance context modeling operation; X c is the initial fusion feature; H is the feature height; W is the feature width; exp is the exponential function with the natural constant e as the base; W c is a linear transformation; is the eigenvector at position i; is the feature vector at position j;

[0055] After the attention feature is calculated, it is dot-multiplied with the original visual feature image to obtain the combined feature along the channel dimension:

[0056]

[0057] In the formula, is the calibration feature image under the same mode; ⊙ is element-wise multiplication; W rgb is the attention vector;

[0058] Cross-modal recalibration refers to the process of exchanging calibration feature images with visual structural features:

[0059]

[0060] Where, X f Recalibrate images for cross-modality; is the calibration feature image under the same mode; X structure It is the visual structural feature;

[0061] Applying the same operation in the spatial dimension, highlighting the areas with high information availability in the spatial location and suppressing the areas with low information availability, highlighting the areas with high information availability in the spatial location and suppressing the information noise:

[0062]

[0063] In the formula, is the attention feature image; G is the spatial attention operation; X c is the initial fusion feature; C is the number of channels; is the feature vector of channel i; is the feature vector of channel j; W f is the linear transformation matrix; exp is the exponential function with the natural constant e as the base;

[0064] The obtained attention feature image is weighted with the original visual feature image to highlight the information area in the spatial dimension and suppress noise to obtain the initial weighted feature image Initial weighted feature image The expression is:

[0065]

[0066] In the formula, is the initial weighted feature image; is the attention feature image; X rgb is the remote sensing image feature;

[0067] Join the original visual structure features and fuse the enhanced output:

[0068]

[0069] In the formula, is the weighted feature image; is the initial weighted feature image; X structure is the visual structure feature; X FR Output for fusion enhancement;

[0070] The feature images obtained by the channel attention and spatial attention mechanisms are concatenated to create the feature image of the feature recalibration module:

[0071]

[0072]

[0073]

[0074] Where, X FR is the feature image of the feature recalibration module; Concat is the cascade operation; is the weighted feature image; X structure It is the visual structural feature; is the output of the weighted feature image; is the output of visual structure features; and is a linear mapping;

[0075] The RGB output from the multi-scale block is sent to a multi-layer perceptron decoder to generate the final segmentation result.

[0076] The beneficial effects of the present invention are as follows: the present invention proposes a remote sensing image semantic segmentation method based on visual structural information guidance, by selecting a remote sensing data set and a tree mapping data set as an experimental data set; based on the experimental data set, a visual basic model is selected, and a visual structural image is extracted from the remote sensing image of the experimental data set through the visual basic model; based on the visual structural image, a feature co-embedding module is constructed to extract the visual feature image in the visual structural image; based on the visual feature image, a structure-inspired feature fusion module is constructed to perform feature fusion and structural prediction on the visual feature image to obtain a processed experimental data set; a target loss function is constructed to predict the processed experimental data set, so that the present invention can extract the structural image through the visual model, construct a module to extract and fuse features, and predict the processed data set, thereby improving the accuracy and robustness of remote sensing image segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A flowchart of a remote sensing image semantic segmentation method based on visual structure information guidance provided by an embodiment of the present invention;

[0078] Figure 2 A schematic diagram of the overall structure of a model provided by an embodiment of the present invention;

[0079] Figure 3 A schematic diagram of the effect of the Potsdam dataset provided by an embodiment of the present invention;

[0080] Figure 4 A schematic diagram of the effect of the Vaihingen dataset provided by an embodiment of the present invention;

[0081] Figure 5 A schematic diagram of the LoveDA dataset effect provided by an embodiment of the present invention;

[0082] Figure 6 A schematic diagram of the effect of the iSAID data set provided by an embodiment of the present invention;

[0083] Figure 7 A schematic diagram of the tree mapping dataset effect provided by an embodiment of the present invention;

[0084] Figure 8 A schematic diagram of the details of the FE module provided in an embodiment of the present invention;

[0085] Fig. 9 A schematic diagram of the details of the FR module provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0087] See also Figure 1 , a remote sensing image semantic segmentation method based on visual structure information guidance in this embodiment includes the following operations:

[0088] Remote sensing datasets and tree mapping datasets were selected as experimental datasets;

[0089] Based on the experimental data set, a visual basic model is selected, and a visual structure image is extracted from the remote sensing image of the experimental data set through the visual basic model;

[0090] Based on the visual structure image, a feature co-embedding module is constructed to extract the visual feature image in the visual structure image;

[0091] Based on the visual feature image, a structure-inspired feature fusion module is constructed to perform feature fusion and structure prediction on the visual feature image to obtain a processed experimental data set.

[0092] Construct the target loss function to predict the processed experimental data set.

[0093] After obtaining the experimental data set, the remote sensing images in the experimental data set are enhanced: the remote sensing images are windowed and cropped, that is, sub-regions are cropped on the remote sensing images with a size of 512×512 and a step size of 256×256, and the local feature ratio in the sub-region is highlighted; the remote sensing images are randomly flipped horizontally; the remote sensing images are randomly scaled, and the scaling ratio ranges from 0.5 to 1.75.

[0094] The data used in the present invention are from four well-known remote sensing datasets: ISPRS Vaihingen, ISPRS Potsdam, LoveDA and iSAID, and a tree mapping dataset.

[0095] This paper introduces two multimodal tree mapping datasets acquired from unmanned aerial vehicles (UAVs) at different times, which are used to evaluate the performance of RS semantic segmentation methods, where the precisely aligned VHR RGB images and the LiDAR-derived nDSM are canopy height models (CHMs) for forestry applications. The tree mapping datasets mainly include trees and various types of vegetation, such as shrubs and grasses, with pixel-level labels divided into two categories: trees and non-trees. See Figure 2 ,Due to the visual similarity, in distinguishing between tree and non-tree vegetation, ,in order to comprehensively evaluate the segmentation accuracy of VSGNet, ,experiments are conducted on the tree dataset and four well-known ,remote sensing datasets, including two multimodal datasets (ISPRS Vaihingen and ISPRS Potsdam) ,and two aerial RGB datasets (LoveDA and iSAID).

[0096] See also Figure 3 The Potsdam dataset consists of 38 high-resolution images of Potsdam, Germany, each with a pixel size of 6000×6000 and a ground sampling distance of 5 cm. The dataset is divided into 24 training datasets and 14 validation datasets. The Potsdam dataset includes two modalities: true orthophotos (TOP) and digital surface models (DSM). The RGB mode is used for comparative experiments of semantic segmentation methods, while the RGB and DSM modes are used for comparison with multimodal semantic segmentation methods. The Potsdam dataset includes 5 categories: buildings, low vegetation, trees, cars, and background.

[0097] See also Figure 4,The Vaihingen dataset consists of 33 high-resolution images, each with a resolution of 2494×2064 pixels, and the dataset is divided into 16 training datasets and 17 validation datasets. RGB mode is used in the comparison experiment of semantic segmentation methods, while RGB and DSM modes are used simultaneously when comparing multimodal semantic segmentation methods.

[0098] See also Figure 5 , the LoveDA dataset consists of 5897 high-resolution images, each with a pixel size of 512×512 and a sampling height of 30 cm. These data come from 18 complex urban and rural scenes in Nanjing, Changzhou and Wuhan, China. The LoveDA dataset includes seven categories: background, building, road, water, bare soil, woodland and cultivated land.

[0099] See also Figure 6 The iSAID dataset consists of 2806 high-resolution RS images, corresponding to 655451 instance objects, with resolutions ranging from 800×800 pixels to 4000×13000 pixels. It includes 15 categories: (ship, storage tank (ST), baseball dome (BD), tennis court (TC), basketball court (BC), ground track (GTF), bridge (bridge), large vehicle (LV), small vehicle (SV), helicopter (HC), swimming pool (SP), circular runway (RA), football field (SBF), airplane (plane) and port (harbour).

[0100] See also Figure 7 ,The tree mapping dataset consists of 1000 RS images, each with a resolution of 640×480. It mainly includes trees and low vegetation (no trees). The dataset is divided into 700 training images and 300 testing images.

[0101] In order to prevent overfitting and enable the model to obtain more complete sample feature information during data enhancement processing, the present invention adopts the following data enhancement methods: (1) window cropping, cropping a sub-region on the original image with a size of 512×512 and a step size of 256×256, so as to highlight the local feature ratio in the sub-region; (2) random horizontal flipping; (3) random scaling (from 0.5 to 1.75).

[0102] When setting parameters, the present invention uses a single RTX4080 GPU to train VSGNet and AdamW optimizer, with an initial learning rate of 8×10 -5, weight decay is 0.01, and batch size is 2. The first 10 epochs of training use warm-up learning rate schedule, and the subsequent epochs use multi-decay schedule, and cross entropy is used as the loss function. Dinov2 in the proposed framework adopts the provided pre-trained model, and the Dinov2 feature extractor is disabled during training. Since the cross entropy loss function is easy to implement, it can be used during training.

[0103] In addition to using VHR RGB images, the present invention also uses auxiliary RS data, which includes synthetic aperture radar images (SAR) and digital surface models (DSM), which can be used for semantic segmentation of land cover. Synthetic aperture radar data can provide geometric and contour details and are not affected by weather conditions, while digital surface models can provide vertical structure and depth information. These supplementary RS data can describe the unique attributes of the same geospatial entity and provide different perspectives for a comprehensive understanding of semantic objects. Therefore, VHR RGB and auxiliary RS data can improve the accuracy of semantic segmentation.

[0104] The visual base model includes the visual big model (SAM) and the deep big model (DAM). The visual structure image refers to the additional modal information learned by the visual base model from the remote sensing images in the experimental dataset. Visual structure extraction can be completed in the preprocessing stage. The remote sensing image is input into the visual base model and the model parameters are frozen. Only model inference is performed to obtain the visual structure image.

[0105] In the field of computer vision, visual foundation models (VFMs) include SegmentAnything (SAM) and DepthAnything (DAM), which are pre-trained with a large amount of data, so that they can have strong generalization perception capabilities, thereby enabling few-sample or zero-sample migration. Based on the image input and clues of the position and shape of the semantic object, SAM can directly generate a mask, while DAM refers to an autoencoder model that can input a two-dimensional image and generate a corresponding depth map, which can provide vertical structural knowledge of the object. Additional visual structure can be obtained through self-learning of large-scale data sets. This visual structure refers to a mechanism that provides structural priors, rather than relying on additional sensors.

[0106] The feature co-embedding module includes CNN and ViT. CNN includes 3×3 convolution, BN layer and GELU activation function, and ViT uses Dinov2 pre-trained model.

[0107] Visual feature image W generated by CNN c The expression is:

[0108] W c =Conv 3×3 (σ(Conv3×3 (X)))

[0109] Where W c is the visual feature image extracted by CNN from the remote sensing image and the visual structure image; Conv3×3 is a 3×3 convolutional layer; σ is the GELU activation function; X is the input remote sensing image;

[0110] The ViT encoder first processes the structural image, tokenizes it through a patch embedding layer, and then generates a feature map. The map can then pass through a series of Transformer encoder blocks and produce outputs of different sizes at each stage. Each Transformer encoding block is followed by a Patch Merging layer, which can halve the spatial dimension and forward the data to the next deeper Transformer encoding block. Due to the strong generalization ability of Dinov2, the present invention adopts the ViT weights obtained from Dinov2. Pre-training Dinov2 on a wide range of datasets can quickly adapt to RS images, so that richer insights can be extracted, and the resulting feature map W embedded by ViT v for:

[0111] W v =F ViT (X)

[0112] Where W v is the visual feature image extracted by ViT from the remote sensing image and the visual structure image; F ViT is the ViT feature extractor; X is the input remote sensing image;

[0113] The visual feature images of the two branches are added together and imported into the subsequent structure-heuristic feature fusion module for further processing:

[0114]

[0115] In the formula, x rgb is the remote sensing image feature; and is the visual feature image extracted by ViT from remote sensing images and visual structure images; x structure It is the visual structural feature; and It is a visual feature image extracted by CNN from remote sensing images and visual structure images.

[0116] The design of collaborative feature extractor can effectively extract long-range contextual cues and local detail information from RGB images and structure graphs, thus facilitating subsequent feature fusion. By designing cross-modal CNN and Transformer feature interaction modules, the proposed network can provide feature co-embedding capabilities, which can collaboratively collect global contextual information and local details from the visual structure graph derived from VHR RGB and VFM. Figure 8 , 9 , multi-scale structure-inspired feature fusion combines fusion enhancement (FE) and feature recalibration (FR) modules, which can extract complementary insights from different modalities. The FE module consists of two branches: global feature interaction and local feature interaction. The global feature interaction branch can improve the object localization ability from a global perspective by merging features from RGB and visual structure. The local feature interaction branch can embed the visual structure features using large kernel deep convolution and reweight the RGB features according to the embedding results, so that the target features can be improved at the detail level. The FR module adopts a dual-branch cross-modal network, which uses visual structure embedding to emphasize informative features while suppressing irrelevant noise and uncertainty.

[0117] The structure-inspired feature fusion module includes fusion enhancement and feature recalibration modules. Fusion enhancement includes self-attention module and large-kernel depth-separable convolution. Fusion enhancement is used to promote the fusion of visual feature images. The feature recalibration module includes channel attention and spatial attention mechanisms. The feature recalibration module is used to improve and calibrate visual feature images.

[0118] Fusion enhancement combines remote sensing images and visual structure images to enhance object positioning capabilities. The self-attention module fuses cross-modal inputs. The large-kernel deep separable convolution performs deep convolution on visual structure features. The convolution results are used as attention weights to re-weight remote sensing image features and embed visual structure features into remote sensing image features. The large-kernel deep separable convolution is used to enhance target features in remote sensing image features.

[0119] The feature recalibration module performs feature calibration and denoising through dual-branch cross-modal interaction, applies cross-modal interaction on remote sensing images and visual structure images, and applies channel attention to convert visual feature images into attention vectors in the embedding space. The generated visual feature images are summarized and the summarized visual feature images are used as the calibration feature images under the same mode.

[0120] After extracting feature maps from RGB and visual structure maps, the present invention creates a feature fusion module to improve information interaction and fusion. The fusion module consists of a fusion enhancement (FE) and a feature recalibration module (FR).

[0121] Fusion Enhancement (FE) can enhance object localization capabilities from both global and detailed perspectives by merging RGB and visual structures. The query comes from RGB and visual structures, which can promote feature pre-fusion along the channel dimension, thereby enhancing the saliency information in the representation. The attention strategy fuses cross-modal inputs so that features with a larger global receptive field can be accessed. In another branch, on the visual structure features, FE uses a large kernel for deep convolution and uses the convolution results as attention weights to reweight the RGB features, thereby embedding the visual structure features into the RGB features and enhancing the target features in the RGB features from a detailed perspective.

[0122] The feature recalibration module (FR) uses a two-branch cross-modal interaction to achieve feature calibration and denoising. First, cross-modal interaction is applied on RGB and visual structures. Subsequently, channel attention is applied to convert feature maps into attention vectors in the embedding space. These vectors represent confident activations in the main feature maps, which facilitates effective feature learning across information elements while reducing noise and uncertainty. The resulting feature maps are aggregated and regarded as calibrated feature maps of the same mode, and the calibrated feature maps have more powerful representation capabilities and less noise.

[0123] When fusion enhancement merges remote sensing images and visual structure images, the query is downsampled to reduce the computational complexity. The query (Q) is derived from the remote sensing image and the visual structure image, and the Key (K) and Value (V) are derived only from the RGB image. Assume that the RGB feature and the visual structure feature are X rgb and X structure , the expression of query Q is:

[0124] Q=Linear(Pool k×k ([X rgb ,X structure ]))

[0125] Where Q is the query, X rgb is the remote sensing image feature; X structure is the visual structure feature; [] is the connectivity in the channel dimension; Pool k×k is a k×k adaptive average pooling operation; Linear is a linear transformation;

[0126] The expression of K is:

[0127] K=Linear(X rgb )

[0128] Where, X rgb is the remote sensing image feature; Linear is the linear transformation; K is the key,

[0129] The expression of V is:

[0130] V=Linear(X rgb )

[0131] Where, X rgb is the remote sensing image feature; Linear is the linear transformation; V is the value,

[0132] The process of the feature recalibration module is:

[0133]

[0134] Where UP is the up sampling operation; Q is the query; is the output of a branch of fusion enhancement; V is the value; Softmax is the Softmax function; K is the key; C is the word vector / hidden layer; T is the transposition; d is the dimension;

[0135] Most existing RS methods ignore the details of objects. This paper considers local information, reshapes feature weights through large kernel depth-separable convolution, and embeds remote sensing image information and mask information:

[0136]

[0137] In the formula, DConv is the fusion enhanced two-branch output; k×k is the depth convolution; k×k is the kernel size of the depth convolution; Linear is the linear transformation; X structure is the visual structure feature; X rgb is the remote sensing image feature; ⊙ is element-wise multiplication;

[0138] The self-attention module and the large-core depth-separable convolution function are connected and mapped to each modality respectively to form a fused enhanced output, i.e.

[0139]

[0140] Where, X FE is the output of fusion enhancement; Concat is the cascade operation; It is a branch output for fusion enhancement; It is the fusion enhanced two-branch output.

[0141] The FE module is mainly used to promote feature fusion, while the FR module on the other side focuses on improving and calibrating features. Due to the symmetry of the FR operation, this paper only introduces the RGB side. When the feature recalibration module performs feature calibration and noise reduction through dual-branch cross-modal interaction, the visual feature image is superimposed:

[0142] X c =Concat(X structure ,X rgb )

[0143] Where, X c is the initial fusion feature; Concat is the cascade operation; X structure is the visual structure feature; X rgb is the remote sensing image feature;

[0144] In order to maintain the integrity of the features, the attention mechanism can convert the feature map into the embedding space as the attention vector. These vectors reflect the highly reliable activations in the feature map, which can effectively focus feature learning on the most informative aspects while reducing the impact of noise features. Taking one branch as an example, the spatial attention mechanism is used to convert the visual feature image into the embedding space as the attention vector. The expression of the attention vector is:

[0145] W rgb =σ(Mlp(F(X c )))

[0146] Where W rgb is the attention vector; σ is the activation function; Mlp is the fully connected layer, which contains two 1x1 layers; F is the long-distance context modeling operation; X c It is the initial fusion feature;

[0147] The expression of the long-distance context modeling operation F is:

[0148]

[0149] Where F is the long-distance context modeling operation; X c is the initial fusion feature; H is the feature height; W is the feature width; exp is the exponential function with the natural constant e as the base; W c is a linear transformation; is the eigenvector at position i; is the feature vector at position j;

[0150] After the attention feature is calculated, it is dot-multiplied with the original visual feature image to obtain the combined feature along the channel dimension:

[0151]

[0152] In the formula, is the calibration feature image under the same mode; ⊙ is element-wise multiplication; W rgb is the attention vector;

[0153] Due to the amplification effect of attention, the resulting feature map can contain more information elements and less noise components, which can be exchanged with features from other modalities to avoid information loss. This process is called cross-modal recalibration. Cross-modal recalibration refers to the process of exchanging calibration feature images with visual structure features:

[0154]

[0155] Where, X f Recalibrate images for cross-modality; is the calibration feature image under the same mode; X structure It is the visual structural feature;

[0156] Applying the same operation in the spatial dimension, highlighting the areas with high information availability in the spatial location and suppressing the areas with low information availability, highlighting the areas with high information availability in the spatial location and suppressing the information noise:

[0157]

[0158] In the formula, is the attention feature image; G is the spatial attention operation; X c is the initial fusion feature; C is the number of channels; is the feature vector of channel i; is the feature vector of channel j; W f is the linear transformation matrix; exp is the exponential function with the natural constant e as the base;

[0159] Different from the previous F operation, G applies softmax and weighted summation in the channel dimension, so that it can pay more attention to the spatial position and emphasize the information area. The obtained attention feature image is weighted with the original visual feature image to highlight the information area in the spatial dimension and suppress noise, and the initial weighted feature image is obtained. Initial weighted feature image The expression is:

[0160]

[0161] In the formula, is the initial weighted feature image; is the attention feature image; X rgb is the remote sensing image feature;

[0162] Similarly, in order to prevent information loss, the original feature map of the visual structure is also added, and the output of FE is also added to enable the feature map to obtain more accurate location information, adding the original visual structure features and the fusion enhanced output:

[0163]

[0164] In the formula, is the weighted feature image; is the initial weighted feature image; X structure is the visual structure feature; X FR Output for fusion enhancement;

[0165] The obtained feature images of the channel attention and spatial attention mechanisms are concatenated to create the feature image of the feature recalibration module and mapped to the corresponding mode:

[0166]

[0167]

[0168]

[0169] Where, X FR is the feature image of the feature recalibration module; Concat is the cascade operation; is the weighted feature image; X structure It is the visual structural feature; is the output of the weighted feature image; is the output of visual structure features; and is a linear mapping;

[0170] The RGB output from the multi-scale block is sent to a multi-layer perceptron decoder to generate the final segmentation result.

[0171] The present invention will be further described below in conjunction with specific embodiments:

[0172] The image dataset used in the present invention includes high-resolution remote sensing images, among which Potsdam and Vaihingen include 5 types of targets: buildings, low vegetation, trees, cars and background, LoveDA includes 7 types of targets: background, buildings, roads, water bodies, bare soil, woodland and cultivated land. iSAID contains 15 categories: ships, storage tanks (ST), baseball domes (BD), tennis courts (TC), basketball courts (BC), ground runways (GTF), bridges (bridge), large vehicles (LV), small vehicles (SV), helicopters (HC), swimming pools (SP), circular runways (RA), football fields (SBF), airplanes (plane) and harbors (harbour).

[0173] (1) The data in the data set is divided into two parts, training and testing, in proportion, and the image data of the supervision branch is enhanced before being input into the model method proposed in the present invention;

[0174] (2) inputting the image data after data enhancement into the model of the present invention respectively to extract features;

[0175] (3) Input the extracted features into the network for feature fusion and structure prediction;

[0176] (4) Experiments were conducted on the dataset, and the experimental results are as follows:

[0177] Table 1 Comparison of Potsdam dataset results

[0178]

[0179] Table 2 Comparison of the results of Vaihingen dataset

[0180]

[0181] Table 3 Comparison of LoveDA dataset results

[0182]

[0183] Table 4 Comparison of the effects of iSAID dataset

[0184]

[0185] Table 5 Comparison of tree dataset results

[0186]

[0187] Through the introduction of the above embodiments, the present invention proposes a remote sensing image semantic segmentation method based on visual structure information guidance, by selecting a remote sensing data set and a tree mapping data set as experimental data sets; based on the experimental data set, a visual basic model is selected, and a visual structure image is extracted from the remote sensing image of the experimental data set through the visual basic model; based on the visual structure image, a feature co-embedding module is constructed to extract the visual feature image in the visual structure image; based on the visual feature image, a structure-inspired feature fusion module is constructed to perform feature fusion and structure prediction on the visual feature image to obtain a processed experimental data set; a target loss function is constructed to predict the processed experimental data set, so that the present invention can extract the structure image through the visual model, construct a module to extract and fuse features, and predict the processed data set, thereby improving the accuracy and robustness of remote sensing image segmentation.

[0188] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have the same meaning as in the context of the prior art, and will not be interpreted with an idealized or overly formal meaning unless defined similarly here.

[0189] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A remote sensing image semantic segmentation method based on visual structure information guidance, characterized in that: The following operations are included: Remote sensing datasets and tree mapping datasets were selected as experimental datasets; Based on the experimental data set, a visual basic model is selected, and a visual structure image is extracted from the remote sensing image of the experimental data set through the visual basic model; Based on the visual structure image, construct a feature co-embedding module to extract a visual feature image in the visual structure image; Based on the visual feature image, construct a structure-inspired feature fusion module to perform feature fusion and structure prediction on the visual feature image to obtain a processed experimental data set; Constructing a target loss function to predict the processed experimental data set; The structural heuristic feature fusion module includes a fusion enhancement and a feature recalibration module, the fusion enhancement includes a self-attention module and a large-kernel depth-separable convolution, and the fusion enhancement is used to promote visual feature image fusion; The feature recalibration module includes a channel attention and a spatial attention mechanism, and the feature recalibration module is used to improve and calibrate the visual feature image; Fusion enhancement combines remote sensing images and visual structure images to enhance object positioning capabilities. The self-attention module fuses cross-modal inputs. The large-kernel deep separable convolution performs deep convolution on visual structure features. The convolution results are used as attention weights to reweight remote sensing image features and embed visual structure features into remote sensing image features. The large-kernel deep separable convolution is used to enhance target features in remote sensing image features. The feature recalibration module performs feature calibration and denoising through dual-branch cross-modal interaction, applies cross-modal interaction on remote sensing images and visual structure images, and applies channel attention to convert visual feature images into attention vectors in the embedding space, and summarizes the generated visual feature images, and uses the summarized visual feature images as calibration feature images under the same mode; When the fusion enhancement merges the remote sensing image and the visual structure image, a downsampling operation is performed on the query, the query is derived from the remote sensing image and the visual structure image, K and V are derived from the remote sensing image, and the expression of the query Q is: Q=Linear(Pool k×k ([X rgb ,X structure ])) Where Q is the query, X rgb is the remote sensing image feature; X structure is the visual structure feature; [] is the connectivity in the channel dimension; Pool k×k is a k×k adaptive average pooling operation; Linear is a linear transformation; The expression of K is: K=Linear(X rgb ) Where, X rgb is the remote sensing image feature; Linear is the linear transformation; K is the key, The expression of V is: V=Linear(X rgb ) Where, X rgb is the remote sensing image feature; Linear is the linear transformation; V is the value, The process of the feature recalibration module is: Where UP is the up sampling operation; Q is the query; is the output of a branch of fusion enhancement; V is the value; Softmax is the Softmax function; K is the key; C is the hidden layer; T is the transpose; d is the dimension; Considering local information, feature weights are reshaped through large kernel depth-separable convolution, and remote sensing image information and mask information are embedded: In the formula, DConv is the fusion enhanced two-branch output; k×k is the depth convolution; k×k is the kernel size of the depth convolution; Linear is the linear transformation; X structure is the visual structure feature; X rgb is the remote sensing image feature; ⊙ is element-wise multiplication; The self-attention module and the large-core depth-separable convolution function are connected and mapped to each modality respectively to form a fused enhanced output, i.e. Where, X FE is the output of fusion enhancement; Concat is the cascade operation; It is a branch output for fusion enhancement; It is the fusion enhanced two-branch output.

2. The remote sensing image semantic segmentation method based on visual structure information guidance according to claim 1, characterized in that: After obtaining the experimental data set, data enhancement is performed on the remote sensing images in the experimental data set: the remote sensing images are windowed and cropped, that is, sub-regions are cropped on the remote sensing images according to a size of 512×512 and a step size of 256×256, and the local feature ratio in the sub-region is highlighted; The remote sensing images are randomly flipped horizontally; the remote sensing images are randomly scaled, and the scaling ratio ranges from 0.5 to 1.

75.

3. The remote sensing image semantic segmentation method based on visual structure information guidance according to claim 1, characterized in that: The visual basic model includes a segmentation large model and a depth large model. The visual structure image refers to the additional modal information learned by the visual basic model from the remote sensing images of the experimental data set. The remote sensing image is input into the visual basic model and the model parameters are frozen, and only model inference is performed to obtain the visual structure image.

4. The remote sensing image semantic segmentation method based on visual structure information guidance according to claim 1, characterized in that: The feature co-embedding module includes CNN and ViT, CNN includes 3×3 convolution, BN layer and GELU activation function, and ViT uses Dinov2 pre-training model; Visual feature image W generated by CNN c The expression is: W c =Conv 3×3 (σ(Conv 3×3 (X))) Where W c is the visual feature image extracted by CNN from the remote sensing image and the visual structure image; Conv3x3 is a 3x3 convolutional layer; σ is the GELU activation function; X is the input remote sensing image; When the ViT encoder processes the visual structure image, it tokenizes the visual structure image through the patch embedding layer to generate a visual feature image; the image passes through multiple Transformer encoding blocks, generating output data of different sizes at each layer, and there is a Patch Merging layer after each Transformer encoding block, which halves the spatial dimension and forwards the output data to the next deeper Transformer encoding block; the ViT weights obtained from Dinov2 are used to pre-train Dinov2 on the experimental dataset, and the visual feature image W embedded by ViT v The expression is: W v =F ViT (X) Where W v The visual feature image extracted by ViT from remote sensing images and visual structure images; F ViT is the ViT feature extractor; X is the input remote sensing image; The visual feature images of the two branches are added together and imported into the subsequent structure-heuristic feature fusion module for further processing: In the formula, x rgb is the remote sensing image feature; and is the visual feature image extracted by ViT from remote sensing images and visual structure images; x structure It is the visual structural feature; and It is a visual feature image extracted by CNN from remote sensing images and visual structure images.

5. The remote sensing image semantic segmentation method based on visual structure information guidance according to claim 1, characterized in that: The feature recalibration module performs feature calibration and noise reduction through dual-branch cross-modal interaction, and superimposes the visual feature image: X c =Concat(X structure ,X rgb ) Where, X c is the initial fusion feature; Concat is the cascade operation; X structure is the visual structure feature; X rgb is the remote sensing image feature; The visual feature image is converted into the embedding space as the attention vector through the spatial attention mechanism. The expression of the attention vector is: W rgb =σ(Mlp(F(X c ))) Where W rgb is the attention vector; σ is the activation function; Mlp is a fully connected layer, which contains two 1x1 convolutional layers; F is a long-distance context modeling operation; X c It is the initial fusion feature; The expression of the long-distance context modeling operation F is: Where F is the long-distance context modeling operation; X c is the initial fusion feature; H is the feature height; W is the feature width; exp is the exponential function with the natural constant e as the base; W c is a linear transformation; is the eigenvector at position i; is the feature vector at position j; After the attention feature is calculated, it is dot-multiplied with the original visual feature image to obtain the combined feature along the channel dimension: In the formula, is the calibration feature image under the same mode; ⊙ is element-wise multiplication; W rgb is the attention vector; Cross-modal recalibration refers to the process of exchanging calibration feature images with visual structural features: Where, X f Recalibrate images for cross-modality; is the calibration feature image in the same mode; X structure It is the visual structural feature; Applying the same operation in the spatial dimension, highlighting the areas with high information availability in the spatial location and suppressing the areas with low information availability, highlighting the areas with high information availability in the spatial location and suppressing the information noise: In the formula, is the attention feature image; G is the spatial attention operation; X c is the initial fusion feature; C is the number of channels; is the feature vector of channel i; is the feature vector of channel j; W f is the linear transformation matrix; exp is the exponential function with the natural constant e as the base; The obtained attention feature image is weighted with the original visual feature image to highlight the information area in the spatial dimension and suppress noise to obtain the initial weighted feature image Initial weighted feature image The expression is: In the formula, is the initial weighted feature image; is the attention feature image; X rgb is the remote sensing image feature; Join the original visual structure features and fuse the enhanced output: In the formula, is the weighted feature image; is the initial weighted feature image; X structure is the visual structure feature; X FR Output for fusion enhancement; The feature images obtained by the channel attention and spatial attention mechanisms are concatenated to create the feature image of the feature recalibration module: Where, X FR is the feature image of the feature recalibration module; Concat is the cascade operation; is the weighted feature image; X structure It is the visual structural feature; is the output of the weighted feature image; is the output of visual structure features; and is a linear mapping; The RGB output from the multi-scale block is sent to a multi-layer perceptron decoder to generate the final segmentation result.

Images