Remote sensing image segmentation method based on multi-source context cues and geometric refinement network

Abstract

The application discloses a remote sensing image segmentation method based on multi-source context prompt and geometric refining network. It includes: obtaining the remote sensing image to be segmented and the digital normalized model image, inputting into the trained remote sensing image segmentation model, and generating the predicted semantic segmentation image; the remote sensing image segmentation model includes: an encoding module; a point box prompt generation module for generating a point box prompt; a cross-modal fusion module for generating cross-modal features and cross-modal fusion features; a geometric boundary prompt generation block for generating a boundary prompt vector; a prompt encoder for generating a prompt embedding and a boundary-enhanced cross-modal fusion feature; an image-text alignment module for aligning the cross-modal features; and a prompt-guided decoding module for generating the predicted semantic segmentation image according to the prompt embedding, the boundary-enhanced cross-modal fusion feature and the semantic-aligned cross-modal feature. The application enhances the robustness of remote sensing segmentation through multi-modal prompt and cross-modal enhancement cooperation.

Description

Technical Field

[0001] This invention relates to the field of deep learning and multimodal remote sensing image processing technology, and more specifically, to a large-model image segmentation method with multi-source contextual cues and geometric boundary cues. Background Technology

[0002] With the development of remote sensing technology, multimodal remote sensing image fusion has become an important means to improve the accuracy of ground object identification. Traditional semantic segmentation methods mainly rely on surface features such as image texture and color, as well as external structural information, for segmentation. The segmentation results are often coarse, with limited accuracy, and lack category labeling capabilities—they can only divide the image into several regions, but cannot automatically identify the semantic category of each region, still requiring manual judgment and labeling. Another approach combines traditional image processing techniques with deep learning: first, traditional algorithms are used to preliminarily process the image to extract candidate target regions; then, convolutional neural networks are used to learn the region features, constructing an effective classification model, ultimately achieving end-to-end semantic segmentation and automatic labeling. This type of method can not only divide the image into semantically meaningful regions, but also achieve accurate category identification for each region.

[0003] In recent years, deep learning methods, represented by convolutional neural networks, have significantly promoted the development of semantic segmentation technology. However, existing methods still face multiple challenges when processing remote sensing images. First, the scale differences and spatial structures of ground objects in remote sensing images are large, making it difficult for traditional networks to simultaneously capture global contextual information and local detailed features. Second, the spectral information provided by a single RGB modality in remote sensing images is limited, and misclassification is prone to occur in scenarios with varying illumination, shadow occlusion, or high similarity of ground objects. Although introducing auxiliary modalities such as digital surface models can provide elevation information to enhance discrimination capabilities, differences in data distribution and registration errors between different modalities often limit the fusion effect.

[0004] Recently, basic models, represented by large-scale segmentation models, have demonstrated strong generalization capabilities in the field of general image segmentation. However, directly applying these models to remote sensing image segmentation has significant limitations: their segmentation results heavily rely on manually provided points or bounding boxes, lacking automated semantic guidance; furthermore, basic models are not well-suited for the specialized field of remote sensing, making it difficult to effectively capture the subtle structure and specific semantic information of ground objects. Existing improvement schemes either only use the model output as auxiliary information or only make local fine-tuning of the model, neither of which fully solves the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a remote sensing image segmentation method based on multi-source contextual hints and geometric refinement networks. This invention can solve the problems of complex spatial structure, high intra-class variability, and insufficient cross-modal registration accuracy in remote sensing images while preserving the strong visual representation capabilities of the base model. It also overcomes the limitations of segmentation methods that still rely on manual annotation of the base model and have limited domain adaptability, significantly improving the accuracy and robustness of semantic segmentation of remote sensing images.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows: I. A remote sensing image segmentation method based on multi-source contextual cueing and geometric refinement network. The remote sensing image segmentation method includes: The remote sensing image to be segmented and the corresponding digital normalization model image are obtained, input into the trained remote sensing image segmentation model, and a predicted semantic segmentation image is generated and output. The remote sensing image segmentation model includes: The encoding module is used to acquire remote sensing features and digitally normalized features at multiple different scales, as well as single-scale digitally normalized fusion features after multi-scale fusion. The dot box prompt generation module is used to generate dot box prompts based on the digital normalization fusion features; The cross-modal fusion module is used to generate cross-modal features at the corresponding scale based on the remote sensing features and digital normalization features at each scale, and then generate single-scale cross-modal fusion features after multi-scale fusion. A geometric boundary cue generation block is used to generate boundary cue vectors based on the cross-modal fusion features; The cue encoding module is used to generate cue embeddings based on dot box cues, and to generate boundary-enhanced cross-modal fusion features based on boundary cue vectors and cross-modal fusion features; The image-text alignment module is used to perform image-text alignment processing on cross-modal features at each scale, generating semantically aligned cross-modal features at the corresponding scale. The prompt-guided decoding module is used to generate a predicted semantic segmentation image based on the prompt embedding, boundary-enhanced cross-modal fusion features, and semantically aligned cross-modal features at various scales.

[0007] The encoding module includes a remote sensing image encoding branch, which is used to encode the remote sensing image at multiple scales and extract remote sensing features at multiple different scales. Specifically, the remote sensing image encoding branch includes multiple cascaded remote sensing image encoding layers. Each remote sensing image encoding layer includes: a cue learning unit, used to generate intermediate remote sensing features based on the input of the remote sensing image encoding layer; mainly composed of a first serpentine linear layer, a first activation layer, a first linear layer, and a second activation layer connected sequentially; a query-key-value projection replacement unit, including a first low-rank adaptation layer, wherein the first serpentine linear layer and the first linear layer share the first low-rank adaptation layer; and a Transformer encoding unit, used to generate remote sensing features at the corresponding scale of the remote sensing image encoding layer based on the intermediate remote sensing features; the Transformer module is the original Transformer module in the pre-trained SAM2 encoder, and the weights of the Transformer encoding unit are frozen during training.

[0008] The encoding module further includes a digital normalized model graph encoding branch, which includes a geometric structure sub-branch and a terrain dependency sub-branch. Each sub-branch includes multiple cascaded digital normalized model graph encoding layers and an upsampling fusion block. The cascaded encoding layers of the geometric structure sub-branch and the terrain dependency sub-branch respectively employ convolutional neural networks (CNN) and Transformer networks. After the digital normalized model graph is processed by each digital normalized model graph encoding layer in sequence, multiple digital normalized features at different scales are generated. These features are then processed by the upsampling fusion block to generate digital normalized fused features. Each of the two sub-branches corresponds to a dot box prompt generation module. Each dot box prompt generation module receives digital normalized fused features from the upsampling fusion block of its own sub-branch, generates and outputs the corresponding dot box prompt.

[0009] The remote sensing image segmentation model further includes a cue quality screening layer, which is used to obtain the reliability of the bounding box cue corresponding to the two sub-branches and select the optimal sub-branch. The cue quality screening layer is also used to input the digitally normalized features of the optimal sub-branch into the cross-modal fusion module and input the bounding box cue of the optimal sub-branch into the geometric boundary cue generation block. The first input of the cue quality screening layer receives the remote sensing features output from the last remote sensing image coding layer and uses it as the backbone feature map. The second and third inputs receive bounding box cue from the two bounding box cue generation modules, respectively. The fourth input receives multiple digitally normalized features of different scales generated by the geometric structure sub-branch, and the fifth input receives multiple digitally normalized features of different scales generated by the terrain-dependent sub-branch. The first output outputs multiple digitally normalized features of different scales generated by the optimal sub-branch, specifically by outputting the digitally normalized features of different scales to the corresponding cross-modal fusion block. The second output outputs the bounding box cue of the optimal sub-branch.

[0010] In the suggestion quality screening layer, the reliability of point suggestions is obtained through the following process: extracting the feature vector corresponding to each candidate pixel from the backbone feature map, calculating the norm of the feature vector corresponding to each candidate pixel as the feature strength, and calculating the mean of all candidate pixels to obtain the reliability value. Specifically, the normalized point coordinates are mapped to the spatial dimension of the backbone feature map, the feature vector corresponding to each point position is extracted, and its L2 norm is calculated as the feature strength. Then, the average is calculated over all points and batches and multiplied by a scaling factor to obtain the evaluation score of the point prediction. The higher the score, the better the alignment between the point and the backbone features and the more accurate the prediction.

[0011] In the suggestion quality screening layer, the reliability of the bounding box suggestion is obtained through the following process: local features corresponding to each candidate pixel are extracted from the backbone feature map, and the local features are evaluated by an evaluation unit to generate a reliability value; wherein, the evaluation unit includes a convolutional layer, a fully connected layer, and a sigmoid activation layer connected in sequence. Specifically: first, a spatial scaling factor is determined based on the ratio of the backbone feature map to the original image size; local features corresponding to each bounding box are extracted from the backbone feature map through a region feature aggregation operation; then, these features are fed into an evaluation network composed of convolutional layers and fully connected layers; after sigmoid activation, a matching score between 0 and 1 is output, reflecting the degree of alignment between the bounding box and the backbone semantic features; In the prompt quality screening layer, the optimal sub-branch is selected based on the total reliability of point prompts and box prompts. Specifically, by comparing the total scores of the two branches at all the same positions in the backbone feature map, the branch with the higher total score is selected as the optimal branch and optimal sub-branch.

[0012] The dot-box hint generation module includes a dot hint generation submodule for generating dot hints and a box hint generation submodule for generating box hints. Both the dot hint generation submodule and the box hint generation submodule receive digital normalization fusion features and generate dot hints and box hints respectively.

[0013] In the dot hint generation module, the dot hint generation submodule is configured to: convert the digital normalized fusion features into a category heatmap through convolution operations, and generate the probability of each pixel belonging to each category through an activation function; filter initial candidate pixels for each category according to a preset probability threshold, and then control the number of candidate pixels for each category according to a preset quantity threshold K (if the probability exceeds K, retain the K highest probability pixels; if the probability is less than K, fill to K), so that each category includes K candidate pixels, resulting in a final candidate pixel set composed of the geometric composition of candidate pixels from all categories; normalize the pixel coordinates of the final candidate pixel set, i.e., all candidate pixels of all categories, to obtain the normalized coordinates of each candidate pixel; assign a corresponding category label to each candidate pixel, and concatenate the normalized coordinates of all candidate pixels of all categories along the batch dimension to form a point coordinate tensor and its corresponding label tensor, which together constitute the dot hint. In the bounding box prompt generation module, the bounding box prompt generation submodule is configured to: generate a center point probability map by performing convolution, activation, and max pooling operations sequentially based on the digital normalized fusion features; simultaneously, generate a size prediction map and an offset prediction map based on the digital normalized fusion features; extract candidate target centers from the center point probability map; extract the corresponding width, height, and center offset from the size prediction map and offset prediction map based on the coordinates of each candidate target center; and then calculate the coordinates of the candidate boxes (coordinates of any two diagonals of the candidate box); remove invalid boxes from all candidate boxes to obtain a set of bounding boxes as the bounding box prompt.

[0014] The cross-modal fusion module includes multiple cascaded cross-modal fusion blocks and a third upsampling fusion block. Each cross-modal fusion block receives remote sensing features at the corresponding scale and digitally normalized features of the optimal branch and optimal sub-branch at its two inputs, and outputs cross-modal features at the corresponding scale. The third upsampling fusion block receives cross-modal features at each scale at its input, generates cross-modal fusion features, and outputs them. The input of the third upsampling fusion block is connected to the output of each cross-modal fusion block, and its output is connected to the input of the geometric boundary cueing generation block. The third upsampling fusion block is used to unify the cross-modal features at each scale to the same resolution layer by layer through upsampling, and splices them along the channel dimension while upsampling the features pairwise at each layer to obtain a single-scale cross-modal fusion feature.

[0015] The cross-modal fusion module includes: a stitching unit for stitching together the remote sensing features and digitally normalized features received by the cross-modal fusion block along the channel dimension to obtain stitched features; a compression unit for processing the stitched features through sequentially connected convolutional layers and activation function layers to obtain compressed features; and a multi-branch attention module, including two parallel attention branches and an average fusion layer; the first attention branch uses an amplitude-aware linear attention (MALA) module, and the second attention branch includes two parallel depthwise separable convolutional layers, each using a 5×5 and a 7×7 convolutional kernel, respectively. The average fusion layer receives the outputs of two attention branches and performs average fusion to obtain fused attention features; the first input of the average fusion layer is connected to the output of the attention branch, and the second and third inputs are respectively connected to the outputs of the two depthwise separable convolutional layers of the second attention branch, and the output outputs the fused attention features; the fused attention features are fused with the compressed features to generate attention enhancement features; preferably, the fused attention features and the compressed features are fused through an element-wise addition operation; the residual connection unit performs residual addition of the attention enhancement features and the remote sensing features received by the cross-modal fusion block to obtain cross-modal features.

[0016] The geometric boundary hint generation block includes: a dimensionality reduction unit, which performs channel dimensionality reduction on the cross-modal fusion features through multi-layer convolution to generate a boundary response map, and then performs thresholding processing on the boundary response map of each channel to obtain a binary boundary map; a boundary hint vector generation unit, which extracts non-zero pixels from the binary boundary maps of all channels, performs coordinate normalization, and sorts them according to the normalized coordinates to obtain a set of candidate boundary points; for each candidate boundary point in the set of candidate boundary points, it obtains the Euclidean distance, cosine angle, and edge type between each candidate boundary point and its adjacent candidate boundary points according to the normalized coordinates to obtain the features of the candidate boundary points, combines the features of all candidate boundary points, and then uniformly fills or truncates them according to a preset boundary hint vector length threshold to obtain a boundary hint vector.

[0017] The image-text alignment module includes: a first convolutional layer for projecting the cross-modal features onto the text feature space to obtain a projected feature map; a fifth normalization layer for normalizing the text feature tensor to obtain normalized text features; a sixth normalization layer for normalizing the projected feature map to obtain a normalized projected feature map; a first cosine alignment layer for calculating the cosine similarity between the normalized text features and the normalized projected feature map to generate a similarity map; a third activation layer for performing Softmax normalization along the category dimension on the similarity map to generate category attention weights; a second convolutional layer for generating projected image features based on the normalized projected feature map; a first broadcast layer for performing a weighted summation of the category attention weights and the projected image features to obtain aggregated features; and a third convolutional layer for mapping the aggregated features back to the cross-modal feature dimension and then adding them to the cross-modal features through a residual connection to obtain semantically aligned cross-modal features.

[0018] Furthermore, the remote sensing image segmentation model also includes a text generation module, which generates text features, including: a first text encoder for extracting original subclass features from formatted text; a third normalization layer for normalizing the original subclass features to obtain normalized subclass features; a first averaging layer for averaging all normalized subclass features under each category to obtain aggregated features of the category; and a fourth normalization layer for performing L2 norm normalization on the aggregated features of each category to obtain normalized aggregated features; the text feature tensor is formed by stacking the normalized aggregated features of all categories along the first dimension.

[0019] The cue encoding module includes a first cue encoder, a second cue encoder, and a boundary guidance optimizer. The first cue encoder receives point cues and encodes them to generate point cue embeddings. The second cue encoder receives bounding box cues and encodes them to generate bounding box cue embeddings. The boundary guidance optimizer receives boundary cue vectors and processes them using fully connected layers to generate channel-level scaling factors and offsets. Then, it uses the scaling factors and offsets to perform channel-level linear modulation on the cross-modal fusion feature map. After performing spatial gating and convolution operations on the modulation result, it adds it to the cross-modal fusion feature map and performs convolution processing (through sequentially connected convolutional layers, batch normalization layers, and activation layers) to generate boundary-enhanced cross-modal fusion features.

[0020] The prompt guidance decoding module includes: a point prompt guidance unit, used to guide and enhance the lowest-level semantic alignment cross-modal features using point prompts, generating point prompt enhancement features; specifically: generating a corresponding Gaussian distribution heatmap based on the coordinates of each point, taking the maximum value pixel-wise for all point heatmaps to create a single-channel guidance mask, concatenating the original feature map with the mask in the channel dimension, feeding it into feature refinement convolution to obtain enhanced features, then calculating a spatial attention map through gated convolution, multiplying the enhanced features element-wise with the spatial attention map, and outputting the point prompt enhancement features; and a box prompt guidance unit, used to guide and enhance the lowest-level semantic alignment cross-modal features using box prompts; specifically: using the bounding box coordinates, generating a continuous mask for each box through sigmoid difference product, and taking the maximum value in the box dimension to obtain a single-channel spatial mask. The input features are multiplied element-wise with the mask to obtain weighted features, which are then concatenated with the original features along the channels. After double-layer convolution, the output box cue enhancement features are refined. The decoding unit is used to upsample and concatenate the multi-level alignment features output by the image text alignment module from high to low level using various cue enhancement features. It includes four layers of upsampling and concatenation operations, specifically: first, the bottom-level semantic alignment cross-modal features are concatenated with the point cue enhancement features and the box cue enhancement features respectively. The two concatenated enhancement features are fused into the bottom-level fused features. Then, they are upsampled layer by layer and concatenated with the next level features. In the last decoding layer, they are concatenated with the boundary enhancement cross-modal fused features and then convolved to extract features. The features are then upsampled to the original size to obtain the final features. The final features are passed through an output prediction head to generate a predicted semantic segmentation image.

[0021] In the predicted semantic segmentation image, each pixel is assigned a predicted category. The category is selected in advance according to the application scenario of the remote sensing image, including one or more combinations of surface object categories such as impermeable surfaces, trees, low vegetation (shrubs, grass, etc.), buildings, vehicles, and cluttered backgrounds.

[0022] II. A Remote Sensing Image Segmentation System Based on Multi-Source Contextual Cueing and Geometric Refinement Network The remote sensing image segmentation system includes: The input unit is used to acquire the remote sensing image to be segmented and the corresponding digital normalization model diagram; The prediction unit is used to input the remote sensing image to be segmented and the corresponding digital normalization model graph into the trained multi-source contextual cueing and geometric optimization network model to generate a predicted semantic segmentation image. The output unit is used to output the predicted semantic segmentation image or to visualize the predicted semantic segmentation image.

[0023] In summary, the method of this invention uses the second-generation arbitrary segmentation model as its backbone and goes through a multi-stage processing flow, including large model adaptation and optimization, dual-branch filtering and prompt generation, multimodal heterogeneous architecture feature fusion, visual-text semantic link refinement, and hierarchical decoding prediction. This process achieves semantic segmentation of remote sensing images with collaborative fusion of multi-source information and automated prompt guidance.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) This invention proposes a multi-source contextual cueing and geometry optimization network based on the second-generation arbitrary segmentation model (SAM2). This network takes RGB and digitally normalized model graphs as input. By incorporating low-rank adaptation (LoRA) and serpentine linear layer parameter-efficient adaptation modules into the large model backbone, this network enhances the model's ability to capture long-range dependencies and fine-grained textures of remote sensing images without full fine-tuning, while retaining the strong visual representation capabilities of the pre-trained model.

[0025] 2) To address the issues of uneven category distribution and insufficient extraction of normalized digital surface modal features in remote sensing images, this invention proposes a dual-branch screening mechanism for normalized digital surface modal features. It constructs heterogeneous feature extraction pathways based on convolutional neural networks (CNN) and transformer models to capture local geometric structures and global terrain dependencies, respectively. A quality-aware cue selector is then used to evaluate the accuracy and adaptively filter the point and box cue generated by the dual branches, fully mining the multi-scale information of the normalized digital surface modal data. This overcomes the limitations of traditional methods that rely on manual cueing and have insufficient reliability.

[0026] 3) To achieve accurate fusion and semantic alignment of multimodal features, this invention constructs a cross-modal fusion architecture with dual enhancement pathways. On the one hand, geometric boundary features are extracted through a polygon boundary cue generation module, transforming abstract semantic information into spatial geometric constraints to optimize structural integrity. On the other hand, a visual text semantic link engineering is constructed based on a contrastive language-image pre-training model. Fine-grained sub-type text feature generation and cross-modal semantic alignment are achieved, and language priors are used to refine feature representations. Finally, a hierarchical decoding mechanism is combined to integrate multimodal alignment features with point, box, and geometric cue types to achieve accurate end-to-end segmentation, significantly improving the segmentation accuracy and robustness of the model in complex remote sensing scenarios. Attached Figure Description

[0027] Figure 1 Block diagram of the SAM2 encoding layer adapter module; Figure 2 Overall implementation block diagram of multi-source context hints and geometric refinement network; Figure 3 This is a flowchart illustrating the overall implementation of the video-text linking project. Figure 4 This is the first test result of an embodiment of the present invention, where a is the first original remote sensing scene image of the same scene; b is... Figure 4 a) shows the ground truth map corresponding to the original remote sensing scene image; c) shows the image obtained using the multi-source contextual cueing and geometric refinement network of the present invention. Figure 4 The original remote sensing scene image shown in figure a is used to predict the semantic segmentation image. Figure 5 This is the second test result of an embodiment of the present invention, where a is the second original remote sensing scene image of the same scene; b is... Figure 5 a) shows the ground truth map corresponding to the original remote sensing scene image; c) shows the image obtained using the multi-source contextual cueing and geometric refinement network of the present invention. Figure 5 The original remote sensing scene image shown in figure a is used to predict the semantic segmentation image. Figure 6 This is the third test result of this embodiment of the invention, where a is the third original remote sensing scene image of the same scene; b is... Figure 6 a) shows the ground truth map corresponding to the original remote sensing scene image; c) shows the image obtained using the multi-source contextual cueing and geometric refinement network of the present invention. Figure 6 The original remote sensing scene image shown in figure a is used to predict the semantic segmentation image. Figure 7 This is the fourth test result of this invention embodiment, where a is the fourth original remote sensing scene image of the same scene; b is... Figure 7 a) shows the ground truth map corresponding to the original remote sensing scene image; c) shows the image obtained using the multi-source contextual cueing and geometric refinement network of the present invention. Figure 7 The original remote sensing scene image shown in figure a is used to predict the semantic segmentation image. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] This invention constructs a multi-source contextual cueing and geometric refinement network based on the SAM2 encoder, achieving effective processing of complex spatial structures in remote sensing images through a multi-dimensional guided re-optimization mechanism. First, the network integrates low-rank adaptation and serpentine linear layers into the SAM2 backbone to accurately capture long-range dependencies and fine-grained texture features in RGB images. Second, a dual-branch filtering structure is used to extract features from the normalized digital surface model. Two parallel branches, constructed based on convolutional neural networks and Transformers, extract normalized digital modal features and generate corresponding cueing. The optimal sub-branch output is adaptively selected by comparing cueing accuracy, achieving full utilization of normalized digital modal information. Then, RGB features and normalized digital modal features are fused through two parallel paths to generate geometric cueing for boundary refinement and contrastive language images for semantic alignment, i.e., pre-trained text embeddings. Finally, multi-level alignment features, including points, bounding boxes, and geometric cueing, are input into a mask decoder to achieve hierarchical prediction. Test results on the Vaihingen and Potsdam datasets show that the multi-source contextual cueing and geometric refinement network proposed in this invention achieves excellent performance in terms of average accuracy and average intersection-union ratio, verifying the robustness of the segmentation method of this invention.

[0030] Specific embodiments of the present invention are as follows: This embodiment proposes a multi-source contextual cueing and geometry optimization network, which includes two modal branches and a text-image alignment project. These are a heterogeneous network branch with digitally normalized modal input and a SAM2 network branch with RGB modal input, respectively. The overall implementation block diagram of the adapter in the SAM2 network branch is shown below. Figure 1 As shown, the overall network implementation block diagram is as follows: Figure 2 As shown, the text-image alignment process is as follows: Figure 3 As shown. The overall structure includes two processes: the training phase and the testing phase.

[0031] The specific steps of the training phase are as follows: Step 1.1: Select The training set consists of 10 original remote sensing scene images, their corresponding digitally normalized model images, and ground truth semantic segmentation images. The training set is then used to analyze the 10th and 11th remote sensing scene images. The original remote sensing scene image is denoted as Training will be focused on The corresponding real semantic segmentation image is denoted as ; Focus training on The corresponding digital normalization model diagram is denoted as follows: Among them, the remote sensing scene images are RGB color images. It is a positive integer, such as taking , It is a positive integer. , express The mid-coordinate position is The pixel value of the pixel.

[0032] The original remote sensing scene images mainly consist of RGB images taken from different scenes and digitally normalized model images. RGB images record spectral information in the red, green, and blue bands, while multispectral images record spectral information in other different bands. The digitally normalized model image is a single-channel image obtained by normalizing the multispectral image. Each band of information is equivalent to a channel component. That is, each original scene image with salient objects contains the R, G, and B channel components of the RGB image, and the digitally normalized model image has an additional channel component.

[0033] Step 1.2: Construct a SAM2 encoder enhanced with low-rank adaptation and a snake-like adapter. This mainly consists of three parts: adapter injection, multi-scale feature extraction, and frozen backbone. The network adopts a single-branch coding structure, based on a pre-trained SAM2 image encoder. First, lightweight adapter submodules are used to efficiently fine-tune the parameters, ultimately outputting multi-scale feature maps. Specifically: the adapter injection part mainly involves embedding an adapter submodule at the input of each Transformer module. The adapter submodule includes a query-key-value projection replacement unit based on low-rank adaptation (LoRA) and a cue learning unit based on a snake-like linear layer. The multi-scale feature extraction part mainly includes feature map outputs at four scales. The frozen backbone part mainly freezes all parameters of the pre-trained SAM2 image encoder during training.

[0034] The adapter module and feature extraction part mainly include the adapter sub-modules embedded in each Transformer module (encoding layer). The adapter sub-modules include a query-key-value projection replacement unit based on LoRA (first low-rank adaptation layer) and a prompt learning unit based on a serpentine linear layer (first serpentine linear layer, first activation layer, first linear layer, second activation layer).

[0035] like Figure 1 As shown, after the RGB image is input into the SAM2 encoder, it passes through the first serpentine linear layer, the first activation layer, the first linear layer, and the second activation layer of the encoding layer in sequence to obtain the RGB feature map. The weights of the first serpentine linear layer are passed to the first low-rank adaptation layer for updating. The obtained RGB feature map passes through the first attention layer, the first normalization layer, the second attention layer, and the second normalization layer in sequence to obtain the fine-tuned feature map and output it from the output layer.

[0036] Specifically, the module takes a single RGB feature image as input, with dimensions H×W. A frozen SAM2 image encoder is used as the backbone network, and parameters are efficiently fine-tuned by embedding custom adapter modules in each encoding layer. Each adapter module contains two core components: a learning unit based on a serpentine linear layer and a query-key-value projection replacement unit based on low-rank adaptation. The low-rank adapted linear layer replaces the original query-key-value projection mechanism, implicitly adjusting attention weights through low-rank adaptation, thereby achieving global feature adaptation with only a small increase in trainable parameters. The feature map is first fed into the first serpentine linear layer for fine-tuning, and then sequentially passes through the first activation layer, the first linear layer, and the second activation layer. During this stage, all linear layer weights are passed to the first low-rank adapted layer for local parameter updates. The resulting features are then passed through the first and second attention layers and a normalization layer to obtain the complete extracted features. A residual addition operation is performed each time the attention normalization operation is performed to avoid losing detailed information. Through fine-tuning and optimization in this stage, the SAM2 encoder can extract feature maps of four scales from the input image while maintaining strong visual representation capabilities, fully capturing multi-level information from high-resolution details to low-resolution semantics, and effectively avoiding the problem of small target feature loss.

[0037] Step 1.3: Constructing a Multi-Source Contextual Hints and Geometric Optimization Network. The Multi-Source Contextual Hints and Geometric Optimization Network mainly consists of several processing stages, including bimodal feature extraction (digital normalization modality dual-branch dynamic filtering and hint generation), multimodal feature fusion and geometric boundary enhancement, and visual-text alignment and semantic enhancement. Through multi-level feature interaction and enhancement, multi-task output is ultimately achieved.

[0038] The feature extraction stage is mainly implemented through RGB image input, digital normalized model graph input, twelve coding layers, three upsampling fusion blocks, two bounding box cue generation modules, one cue quality screening layer, four cross-modal fusion modules, one geometric boundary cue generation module, one image-text alignment module, and one cue encoding module. The feature reconstruction stage is mainly implemented through a cue-guided decoding module and an output prediction head.

[0039] like Figure 2 As shown, the dual-branch coding module is configured to receive the RGB image and the digital normalized model graph, and then input the remote sensing image coding branch and the digital normalized model graph coding branch respectively: The remote sensing image coding branch uses a pre-trained SAM2 encoder as its core. It achieves efficient parameter fine-tuning by injecting low-rank adaptation-based query-key-value projection units and serpentine linear layer cue learning units into the input of each SAM2 coding layer, while freezing the SAM2 encoder backbone weights. During the inference phase, the remote sensing image coding branch sequentially extracts remote sensing feature maps at four scales through the first to fourth coding layers, with sizes of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image, respectively. This hierarchically captures multi-level information from high-resolution details to low-resolution semantics. The digital normalized model graph encoding branch uses a CNN network (ResNet-18 residual network in this embodiment) and a Transformer network (DFormer-Tiny network in this embodiment) in parallel processing. Specifically, local geometric structure features and global terrain dependency features are extracted sequentially through the fifth to eighth encoders and the ninth to twelfth encoders, respectively. Then, digital normalized multi-scale fusion features are generated by passing through the first and second upsampling fusion blocks, respectively. The corresponding point box cue generation modules are then used to generate corresponding point box cuees through the first and second point box cue generation modules, respectively. The reliability of the point cuees and box cuees output by the two branches is dynamically evaluated by the first cue quality screening layer, and the optimal sub-branch output is adaptively selected.

[0040] Subsequently, the remote sensing features at various scales and the digitally normalized features of the selected optimal sub-branch are input layer by layer into the first to fourth cross-modal fusion blocks. After channel splicing, 1×1 convolution compression, amplitude-aware linear attention and depth-separable convolution with large kernels (5×5 and 7×7 convolution kernels) are jointly processed to obtain multi-level cross-modal features, thereby achieving interactive enhancement of dual-modal features.

[0041] The multi-level cross-modal features are processed by two main methods. First, the boundary response map is extracted sequentially through the third upsampling fusion block and the first geometric boundary cue generation module. After thresholding, deduplication, normalization, and neighborhood relationship encoding, a boundary cue vector is formed. Second, the first text-image alignment module performs image-text alignment projection with the multi-subtype text features generated by the text encoder of the contrastive language-image pre-trained model. Through cosine similarity calculation and category attention weighting, semantic information is integrated into the image features to obtain semantically aligned cross-modal features.

[0042] Figure 2 In the cue encoding module, the first cue encoder receives point cues and encodes them to generate point cue embeddings. The second cue encoder receives bounding box cues and encodes them to generate bounding box cue embeddings. The boundary-guided optimizer receives boundary cue vectors from the geometric boundary cue generation module and cross-modal fusion feature maps from the third upsampled fusion block. It uses the boundary cue vectors to enhance the boundaries of the cross-modal fusion feature maps, generating boundary-enhanced cross-modal fusion features.

[0043] Finally, the multi-level semantically aligned cross-modal features, along with point cue embeddings, box cue embeddings, and boundary enhancement cross-modal fusion features, are input into the cue-guided decoding module. Through progressive upsampling, they are concatenated with higher-sized features and refined by convolution to generate semantic prediction outputs at different scales. Finally, under the supervision of multiple cross-entropy losses and focus losses, end-to-end optimization is completed.

[0044] During inference, multi-level semantic alignment cross-modal features, along with point cue embedding, box cue embedding, and boundary enhancement cross-modal fusion features, are input into the cue-guided decoding module to generate a predicted semantic segmentation image and output it.

[0045] The dot hint generation submodule is configured as follows: First, the digitally normalized fusion features output from the first or second upsampled fusion block are input into the dot hint generation module. A 1×1 convolution is used to convert the feature map into a category heatmap, and an activation function is used to generate a probability score for each spatial location belonging to a category. Then, a threshold filtering is applied to the heatmap of each category, extracting all pixel coordinates with probability values ​​higher than a preset threshold as candidate points. To ensure a fixed number of dot hints per category, the module further performs quantity control: if the number of candidate points exceeds a preset K value, the top K points with the highest scores are retained; if there are fewer than K, they are filled by adding category center points or randomly repeating existing points to ensure that the number of points in each category reaches K. Next, the filtered pixel coordinates are normalized by dividing by the spatial size of the feature map to obtain continuous coordinates ranging from 0 to 1. Finally, each point is assigned its corresponding category label, and the point coordinates of all categories are concatenated along the batch dimension to form a point coordinate tensor of dimension (B, n×K, 2) and its corresponding label tensor (B, n×K), where n is the total number of categories. These tensors serve as point hint inputs for subsequent segmentation tasks.

[0046] The bounding box suggestion generation submodule is configured as follows: First, the digitally normalized fusion features output from the first or second upsampled fusion block are input into the center point heatmap prediction branch. By continuously applying 1×1 convolution, 3×3 convolution, ReLU activation, and Sigmoid activation, combined with 3×3 max pooling for local peak suppression, a center point probability map is generated, retaining only the maximum value points within the neighborhood, thus obtaining potential target center points. Next, the heatmap is flattened, and the top K points with the highest scores are selected as candidate target centers. Then, based on the candidate point coordinates, the corresponding target width, height, and center offset are extracted from the size prediction map and offset prediction map, respectively. Based on this, the center coordinates, offset, and size information are combined to calculate the coordinates of the top-left and bottom-right corners of each candidate box, completing the initial generation of the bounding box. Finally, the generated candidate boxes are filtered using a preset score threshold, eliminating invalid boxes to obtain the final set of bounding box proposals. The entire module achieves end-to-end transformation from multi-scale features to precise bounding boxes by concatenating feature extraction, candidate point filtering, and geometric decoding.

[0047] The quality screening layer is configured as follows: First, the remote sensing features output from the fourth coding layer are used as the backbone feature map. Based on the backbone feature map, the reliability of point predictions and bounding box predictions from the two branches are evaluated respectively. For point predictions, the normalized point coordinates (range 0 to 1) are mapped to the spatial dimensions of the backbone feature map. The feature vector corresponding to each point position is extracted from the backbone feature map according to the mapping index, resulting in a feature set of dimension (B4, 64, P). Then, the L2 norm of each feature vector is calculated to characterize the feature strength of the point, and the average is calculated over all points and batches. The average is then multiplied by a scaling factor to obtain the evaluation score of the point prediction. The higher the score, the better the fit between the point prediction and the backbone features. For bounding box prediction, the bounding box coordinates are first mapped to the feature map scale based on the spatial scaling factor between the backbone feature map and the original image (e.g., the ratio of image size 256 to backbone feature map size 8), and batch indexing is added to construct the tensor required for ROI Align. Then, ROI Align is used to extract the local features corresponding to each bounding box from the backbone feature map, resulting in two sets of bounding box features. These features are sequentially passed through an evaluation network consisting of convolutional and fully connected layers. After sigmoid activation, a matching score (between 0 and 1) is output for each bounding box, reflecting the degree of matching between the box region and the backbone semantic features. By comparing the total scores of the two branches at all the same locations in the backbone feature map, the branch with the higher total score is selected as the optimal sub-branch.

[0048] The cross-modal fusion block is configured as follows: First, remote sensing features and filtered digitally normalized features are concatenated along the channel dimension to obtain a joint representation. Then, a 1×1 convolution is used to perform channel compression and preliminary fusion on the concatenated features, reducing dimensionality while integrating bimodal information, and the compressed features are obtained after passing through an ELU activation function. This compressed feature is fed into a multi-branch attention module: on one hand, amplitude-aware lightweight attention (MALA) is used to capture long-range dependencies; on the other hand, 5×5 and 7×7 depthwise separable large kernel convolutions are used to extract typical long-range context and structural features in the remote sensing image. The outputs of these branches are aggregated through average fusion and then combined with the main branch features to achieve scale-adaptive enhancement for remote sensing targets. Finally, the attention-enhanced features are added to the original RGB features through residual connections, injecting global semantics into the fused feature while preserving the original modal information, thus outputting the final fused feature.

[0049] The geometric boundary cue generation block is configured as follows: First, the cross-modal fusion features output from the third upsampling fusion block are input into this module. Through three layers of convolution, the number of channels is progressively reduced from 64 to 8, generating a boundary response map. Then, each channel response map is thresholded to obtain a binary boundary map. Next, the coordinates of non-zero pixels are extracted from all channels, merged into a single set, and deduplicated. The coordinates are then normalized to the 0-1 range by dividing by 255 and sorted by coordinate value. For each point, the Euclidean distance and cosine angle with its adjacent points are calculated, and the edge type (internal edge or intersecting edge) is determined based on the channel response between the two points. The above five-tuple features (coordinates, distance, angle, type) are sequentially concatenated to form the original feature vector. Finally, they are uniformly padded or truncated to a fixed length of 128, resulting in a boundary cue tensor of shape (B, 128).

[0050] The boundary-guided optimizer in the cue encoding module is configured as follows: First, the 128-dimensional boundary cue vector is processed by a fully connected network to generate scaling factor γ and offset β for each feature channel, resulting in modulation parameters of shape (B,2,C,1,1). Then, these parameters are used to perform channel-level linear modulation on the cross-modal fusion features to achieve adaptive adjustment. Finally, the modulated feature map is refined sequentially by 3×3 convolution, batch normalization, and ReLU activation function to output an optimized feature map of the same size as the input (B,64,H,W), i.e., boundary-enhanced cross-modal fusion features, thereby effectively enhancing the feature response of the boundary region and suppressing noise.

[0051] like Figure 3As shown, the text-image alignment process is as follows: This module takes a predefined category definition dictionary as input, where each high-level category is associated with multiple sub-category descriptions. For each sub-category, the descriptive text is first formatted using the aerial view template "Aerial view of {}" to align it semantically with the remote sensing image. The formatted text is then processed by a first text encoder to extract the original sub-category features, and passed through a third normalization layer to ensure feature scale consistency. The normalized features of all sub-categories under the same category are averaged through a first averaging layer to obtain category-level aggregated features, which are then passed through a fourth normalization layer to lie on a unit hypersphere. The aggregated features of all categories are stacked along the first dimension to form a text feature tensor. Subsequently, the multi-scale image features and the pre-computed sub-category text features are fed into the text-image alignment module. Specifically, the image features are first projected onto the text feature space through a first convolutional layer to obtain a projected feature map. The projected features and text features are then L2 normalized through the fifth and sixth normalization layers, respectively, and then the cosine similarity between them is calculated through a first cosine alignment layer to generate a similarity map, representing the degree of alignment between each spatial location and each category. The similarity map is normalized along the category dimension using a third activation layer to generate category attention weights. These attention weights are then weighted and summed with the projected image features through a first broadcast layer to aggregate category-enhanced semantic features. The aggregated features are then mapped back to the original image feature dimension through a third convolutional layer and added to the input image features via residual connections. This process preserves the original spatial structure while integrating category-level semantic information into the image features, achieving cross-modal semantic alignment and fusion.

[0052] Step 1.4: Each original remote sensing scene image and its corresponding multispectral image from the training set are augmented using methods such as brightness adjustment, contrast adjustment, color adjustment, random flipping, and cropping. These are then used as the original input images and fed into the network for training to obtain the corresponding predicted remote sensing scene image, denoted as [image 1]. Within the training framework, the loss value for the final predicted remote sensing image is calculated using the cross-entropy loss function, and the true semantic segmentation image is calculated. A collection of processed real images Feature maps predicted by multi-source contextual cues and geometry-optimized networks The loss between these values ​​is used to supervise network training.

[0053] Step 1.5: Repeat step 1.4 for a total of This process is repeated until the neural network converges and the loss function value is minimized, resulting in a network classification training model. During training, a total of Q loss function values ​​are obtained. The loss function value with the smallest value is selected, and the corresponding weight vector and bias term are taken as the optimal weight vector and optimal bias term for the proposed model, denoted as... , ;in, In this embodiment, V=200.

[0054] The specific steps of the testing phase are as follows: Step 2.1: Select the Pth group of remote sensing images to be detected in the test set and label them as follows. Where, if it is the Vaihingen dataset, then 1≤p≤P, P=17; if it is the Potsdam dataset, then 1≤p≤P, P=7.

[0055] Step 2.2: Identify the various channels of the remote sensing scene image to be detected in group P ( The R-channel components, G-channel components, B-channel components, and digitally normalized model components are input into the trained model. The model network outputs a remote sensing scene semantic segmentation prediction map corresponding to the original remote sensing scene image. The resulting prediction image is denoted as... .

[0056] To further verify the feasibility and effectiveness of the method of this invention, the following tests were conducted: A heterogeneous dual-network framework for multimodal remote sensing segmentation based on co-learning was built using the deep learning library PyTorch 2.1.1. Test sets from the Vaihingen and Potsdam remote sensing scene databases were used to analyze the remote sensing scene segmentation performance using the method of this invention. Two commonly used objective parameters for evaluating semantic segmentation methods were used as evaluation metrics: Mean Class Accuracy (mAcc) and Mean Intersection over Union (MIoU) of the segmented image and the labeled image, to evaluate the segmentation performance of the predicted semantic segmentation image.

[0057] The method of this invention is used to predict each remote sensing scene image in the Vaihingen and Potsdam test sets to obtain the predicted semantic segmentation image corresponding to each remote sensing scene image. The average category accuracy mAcc and the ratio mIoU of the intersection and union of the segmented image and the label image, which reflect the semantic segmentation effect of the method of this invention, are listed in Tables 1 and 2.

[0058] Table 1 Evaluation results of the method of the present invention (MPGNet) on the Vaihingen test set. Table 2 Evaluation results of the method of the present invention (MPGNet) on the Potsdam test set. As can be seen from the data listed in Tables 1 and 2, the segmentation results of the road scene images obtained by the method of the present invention are good, indicating that it is feasible and effective to use the method of the present invention to obtain the predicted semantic segmentation images corresponding to the road scene images.

[0059] The above specific embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

[0060] The above description is only a preferred embodiment of the present invention. Therefore, all equivalent changes or modifications made to the structure, features and principles described in the claims of this patent application are included in the scope of this patent application.

Claims

1. A remote sensing image segmentation method based on multi-source contextual cues and geometric refinement networks, characterized in that, include: The remote sensing image to be segmented and the corresponding digital normalization model image are obtained, input into the trained remote sensing image segmentation model, and a predicted semantic segmentation image is generated and output. The remote sensing image segmentation model includes: The encoding module is used to acquire remote sensing features and digitally normalized features at multiple different scales; The dot box prompt generation module is used to generate dot box prompts based on the digital normalized fusion features after multi-scale fusion. The cross-modal fusion module is used to generate cross-modal features based on remote sensing features and digital normalized features at each scale, and then generate cross-modal fusion features after multi-scale fusion. A geometric boundary cue generation block is used to generate boundary cue vectors based on the cross-modal fusion features; The cue encoding module is used to generate cue embeddings based on dot box cues, and to generate boundary-enhanced cross-modal fusion features based on boundary cue vectors and cross-modal fusion features; The image-text alignment module is used to perform image-text alignment processing on cross-modal features at each scale to generate semantically aligned cross-modal features; The prompt-guided decoding module is used to generate a predicted semantic segmentation image based on the prompt embedding, boundary-enhanced cross-modal fusion features, and semantically aligned cross-modal features at various scales.

2. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: The encoding module includes a remote sensing image encoding branch composed of multiple cascaded remote sensing image encoding layers; the remote sensing image encoding layer includes: The prompting learning unit is used to receive the input of the remote sensing image coding layer and process it sequentially through the first serpentine linear layer, the first activation layer, the first linear layer, and the second activation layer to generate intermediate remote sensing features; The query-key-value projection replacement unit includes a first low-rank adaptation layer; the first serpentine linear layer and the first linear layer share the first low-rank adaptation layer; The encoding unit is used to generate remote sensing features based on the intermediate remote sensing features.

3. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: The encoding module includes a digital normalized model graph encoding branch, which comprises two sub-branches. Each sub-branch includes a cascaded encoding layer and an upsampling fusion block. The two sub-branches are a geometric structure sub-branch and a terrain-dependent sub-branch. The cascaded encoding layers of the geometric structure sub-branch and the terrain-dependent sub-branch respectively employ a convolutional neural network and a Transformer network. Each sub-branch output is configured with a bounding box prompt generation module. The outputs of both bounding box prompt generation modules are connected to a prompt quality filtering layer. The prompt quality filtering layer is used to select the optimal sub-branch based on the overall reliability of the bounding box prompts corresponding to the two sub-branches on the lowest-level remote sensing features.

4. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 3, characterized in that: In the prompt quality screening layer, the reliability of point prompts is obtained through the following process: extracting the feature vector corresponding to each candidate pixel from the backbone feature map, calculating the norm of the feature vector corresponding to each candidate pixel as the feature intensity, calculating the mean of all candidate pixels, and obtaining the reliability value; In the prompt quality screening layer, the reliability of box prompts is obtained through the following process: extracting the local features corresponding to each candidate pixel from the backbone feature map, and generating a reliability value for the local features by the evaluation unit.

5. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: The dot-box prompt generation module includes a dot prompt generation submodule and a box prompt generation submodule; The point hint generation submodule is configured to process the digital normalized fusion features through convolution and activation operations to generate the probability of each pixel belonging to each category. Initial candidate pixels for each category are selected based on a preset probability threshold. The number of candidate pixels for each category is then controlled based on a preset quantity threshold K to obtain the final set of candidate pixels. A corresponding category label is assigned to each candidate pixel, and the normalized coordinates of all candidate pixels are concatenated to form a point coordinate tensor and its corresponding label tensor, thus obtaining a point hint. The box prompt generation submodule is configured to: generate a center point probability map based on the digital normalized fusion features; extract candidate target centers from the center point probability map; extract the corresponding width, height, and center offset from the size prediction map and offset prediction map based on the coordinates of each candidate target center; and then calculate the coordinates of the candidate boxes; remove invalid boxes from all candidate boxes to obtain a set of bounding boxes as box prompts.

6. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: The cross-modal fusion module includes multiple cascaded cross-modal fusion blocks and a third upsampling fusion block; The cross-modal fusion block includes: The stitching unit is used to stitch together the remote sensing features and digitally normalized features received by the cross-modal fusion block to obtain the stitched features. The compression unit is used to process the concatenated features through sequentially connected convolutional layers and activation function layers to obtain compressed features; The multi-branch attention module includes two parallel attention branches and an average fusion layer; the first attention branch adopts an amplitude-aware linear attention module, and the second attention branch includes two parallel depthwise separable convolutional layers; the average fusion layer receives the outputs of the two attention branches and performs average fusion to obtain fused attention features; the fused attention features are fused with the compressed features to generate attention enhancement features; The residual connection unit adds the attention-enhanced features and the remote sensing features received by the cross-modal fusion block through residual addition to obtain the cross-modal features.

7. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: The geometric boundary hint generation block includes: The dimensionality reduction unit performs channel dimensionality reduction on the cross-modal fusion features through multi-layer convolution to generate boundary response maps. Then, the boundary response maps of each channel are thresholded to obtain binarized boundary maps. The boundary hint vector generation unit extracts non-zero pixels from the binary boundary map of all channels, performs coordinate normalization, and sorts them according to the normalized coordinates to obtain a set of candidate boundary points. For each candidate boundary point, it obtains the Euclidean distance, cosine angle, and edge type between each candidate boundary point and its adjacent candidate boundary points based on the normalized coordinates to obtain the features of the candidate boundary point. It combines the features of all candidate boundary points and then uniformly fills or truncates them according to a preset boundary hint vector length threshold to obtain the boundary hint vector.

8. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: The image text alignment module includes: The first convolutional layer is used to project the cross-modal features onto the text feature space to obtain a projected feature map; The fifth normalization layer is used to normalize the text feature tensor to obtain normalized text features; The sixth normalization layer is used to normalize the projection feature map to obtain a normalized projection feature map. The first cosine alignment layer is used to calculate the cosine similarity between the normalized text features and the normalized projected feature map, and generate a similarity map. The third activation layer is used to normalize the similarity map along the category dimension and generate category attention weights. The second convolutional layer is used to generate projected image features based on the normalized projected feature map; The first broadcast layer is used to perform a weighted summation of the category attention weights and projected image features to obtain aggregated features; The third convolutional layer is used to map the aggregated features back to the cross-modal feature dimension, and then add them to the cross-modal features through residual connections to obtain semantically aligned cross-modal features.

9. The remote sensing image segmentation method using multi-source contextual cueing and geometric refinement networks according to claim 1, characterized in that: In the predicted semantic segmentation image, each pixel is assigned a predicted category, which includes one or more of the following: impermeable surface, trees, low vegetation, buildings, vehicles, and background.

10. A remote sensing image segmentation system based on multi-source contextual cues and geometric refinement networks, characterized in that, include: The input unit is used to acquire the remote sensing image to be segmented and the corresponding digital normalization model diagram; The prediction unit is used to input the remote sensing image to be segmented and the corresponding digital normalization model graph into the trained multi-source contextual cueing and geometric optimization network model to generate a predicted semantic segmentation image. The output unit is used to output the predicted semantic segmentation image or to visualize the predicted semantic segmentation image.

Images