Remote sensing image segmentation method, device and equipment based on spatial affinity learning

Abstract

The application relates to a remote sensing image segmentation method, device and equipment based on spatial affinity learning. The method comprises the following steps: obtaining a multi-level feature map of a remote sensing training image through a feature extraction network and a neck network, inputting the multi-level feature map into a head network containing parallel detection and segmentation branches to obtain hierarchical classification and regression features, performing multi-level aggregation and enhancement on the two types of features through a spatial information enhancement unit, splicing the features into a bounding box feature, generating a mask feature from the multi-level feature map, guiding the enhancement of the mask feature through the bounding box feature through a double-flow residual feature fusion unit to enhance the spatial perception and semantic coherence of the mask feature, splicing the mask feature and relative coordinates to input a full convolution segmentation head to obtain an instance segmentation prediction result, calculating a detection and segmentation branch loss by taking the prediction result and a horizontal box label as input, training each module to obtain a segmentation network, processing a remote sensing image through the segmentation network to obtain a detection and segmentation result. The method improves the utilization rate of spatial information of box supervision, optimizes the box supervision remote sensing instance segmentation task, and reduces the gap with the full supervision method.

Description

Technical Field

[0001] This application relates to the field of remote sensing image segmentation technology, and in particular to a remote sensing image segmentation method, apparatus and device based on spatial affinity learning. Background Technology

[0002] Remote sensing image segmentation is a core supporting technology in fields such as land planning, agricultural monitoring, and disaster response. It provides crucial spatial information for decision-making by accurately extracting target instances (such as buildings, roads, and water bodies) from images. However, traditional instance segmentation relies on pixel-level mask annotation, which is cumbersome, time-consuming, and labor-intensive. Especially in high-resolution remote sensing images, where the number of targets is large and their scale varies greatly, the annotation cost increases exponentially, severely restricting the large-scale application of the technology.

[0003] Supervised instance segmentation with bounding boxes has emerged, which only requires horizontal rectangular bounding boxes to label the approximate range of the target, thus replacing pixel-level masks for model supervision and significantly reducing annotation costs. This has become a research hotspot in the field of remote sensing image segmentation. Currently, the mainstream methods fall into two categories: one is based on low-level visual features, using pixel color, texture, and other features to construct association constraints, transforming bounding box constraints into pixel-level supervision; the other is based on high-level semantic features, extracting target semantic information through pre-trained networks, generating pseudo-labels, and combining them with bounding box annotations for joint supervision.

[0004] However, existing box-supervised segmentation methods still suffer from core shortcomings in accurately matching subsequent technical solutions in remote sensing scenes: First, the detection and segmentation branch information is fragmented, lacking efficient feature interaction between the two. The spatial positioning information of the detection branch cannot be effectively transferred to the segmentation branch, resulting in a lack of accurate spatial guidance for pixel-level prediction. Second, they ignore spatial proximity priors, with existing methods relying heavily on color similarity to construct constraints, which easily leads to misclassification of pixels with similar colors but different instances. Third, the mask quality is poor, lacking global smoothing regularization constraints, resulting in fragmented prediction results, boundary noise, and other problems, making it difficult to adapt to the spatial structural features of targets in remote sensing images (such as the continuity of building arrangements and the linear extension of roads). Therefore, there is an urgent need for a box-supervised segmentation method that focuses on in-depth mining of spatial information and solves the problems of branch fragmentation and lack of spatial priors, in order to improve the accuracy and practicality of remote sensing image segmentation. Summary of the Invention

[0005] Therefore, it is necessary to provide a remote sensing image segmentation method, apparatus, and device based on spatial affinity learning that can enhance spatial correlation, adapt to remote sensing scenes, optimize supervised representation, and thus improve segmentation accuracy, in order to address the above-mentioned technical problems.

[0006] A remote sensing image segmentation method based on spatial affinity learning, the method comprising:

[0007] Obtain a training dataset, which includes multiple remote sensing training images and their corresponding horizontal bounding box annotations; The remote sensing training images are sequentially passed through a feature extraction network and a neck network to obtain multi-level feature maps; The multi-level feature map is input into a head network with parallel detection and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are then aggregated and enhanced by a spatial information enhancement unit. Finally, the aggregated and enhanced classification features and regression features are concatenated to obtain bounding box features. Mask features are generated from the multi-level feature maps. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain fused mask features. The result obtained by concatenating the fused mask features with the relative coordinates is input into the fully convolutional segmentation head to obtain the prediction mask; The loss functions of the detection branch and the segmentation branch are calculated based on the segmentation prediction results of the instance and the corresponding horizontal box annotations. The loss functions are then used to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit, and the fully convolutional segmentation head. A remote sensing image segmentation network is then constructed based on the trained feature extraction network, the neck network, and the head network. Acquire remote sensing images, and use the remote sensing image segmentation network to segment the remote sensing images to obtain target detection results and segmentation results.

[0008] In one embodiment, the head network includes multiple detection head branches corresponding to the processing of feature maps at various levels, and each detection head branch includes a parallel detection branch and a segmentation branch. The detection branch outputs a classification score, and the segmentation branch outputs regression coordinates, center point scores, and controller parameters for segmentation head weight allocation.

[0009] In one embodiment, when performing multi-level feature aggregation and feature enhancement on the multi-level classification features and multi-level regression features respectively through a spatial information enhancement unit: After concatenating the classification or regression features of adjacent levels, the concatenated features are then input into the spatial information enhancement unit for enhancement, resulting in aggregated enhanced classification and regression features for each level. The aggregated enhanced classification features and aggregated enhanced classification regression features at each level are aggregated in multiple levels to obtain aggregated enhanced classification features and aggregated enhanced regression features. Then, the aggregated enhanced classification features and aggregated enhanced regression features are concatenated to obtain the bounding box features.

[0010] In one embodiment, in the spatial information enhancement unit: The feature obtained by concatenating adjacent levels of classification or regression features is used as the input feature of the spatial information enhancement unit; The input features are sequentially processed by average pooling of the channel dimension, convolutional channel compression, nonlinear transformation introduced by ReLU activation, and then dimensionality increase by convolutional channels. Finally, the channel weights are mapped to the interval of 0 to 1 by the sigmoid activation function to generate feature-optimized distribution weights. The distributed weights are multiplied element-wise with the input features to obtain the enhanced features.

[0011] In one embodiment, in the dual-stream residual feature fusion unit: The bounding box features are first expanded through convolution, then captured through linear deformable convolution to capture multi-scale context, and the residuals are added and then activated by GELU to obtain the processed bounding box features. The mask features are expanded to the same channel as the bounding box features through convolution, and then multiplied element-wise with the processed bounding box features to achieve dynamic modulation and obtain fused features. The fused feature is compressed through convolution and then fused with the mask feature via residual connection to obtain the fused mask feature.

[0012] In one embodiment, the loss function of the detection branch includes classification loss, regression loss, and centrality loss; The loss function for the segmented branches includes projection loss and spatial affinity loss.

[0013] In one embodiment, the spatial affinity loss is expressed as:

[0014] In the above formula, This represents the spatial weights based on the Gaussian kernel. Represents the total variation loss. It is a color-driven pairwise affinity loss.

[0015] This application also provides a remote sensing image segmentation device based on spatial affinity learning, the device comprising: The training data acquisition module is used to acquire the training dataset, which includes multiple remote sensing training images and corresponding horizontal bounding box annotations. A multi-level feature extraction module is used to sequentially pass the remote sensing training image through a feature extraction network and a neck network to obtain a multi-level feature map; The bounding box feature acquisition module is used to input the multi-level feature map into a head network with parallel detection and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are respectively aggregated and enhanced by the spatial information enhancement unit. The aggregated and enhanced classification features and regression features are then concatenated to obtain the bounding box features. The fusion mask feature acquisition module is used to generate mask features from the multi-level feature map. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain the fused mask features. The instance segmentation prediction module is used to input the result obtained by concatenating the fused mask features and relative coordinates into the fully convolutional segmentation head to obtain the prediction mask; The network training module is used to calculate the loss function of the detection branch and the segmentation branch based on the instance segmentation prediction result and the corresponding horizontal box label, and to use the loss function to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit and the fully convolutional segmentation head, and to construct the remote sensing image segmentation network based on the trained feature extraction network, the neck network and the head network. The remote sensing image instance segmentation module is used to acquire remote sensing images, perform image segmentation on the remote sensing images using the remote sensing image segmentation network, and obtain target detection results and segmentation results.

[0016] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the remote sensing image segmentation based on spatial affinity learning described above.

[0017] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described remote sensing image segmentation method based on spatial affinity learning.

[0018] The aforementioned remote sensing image segmentation method, apparatus, and device based on spatial affinity learning obtain multi-level feature maps by sequentially passing remote sensing training images through a feature extraction network and a neck network. These multi-level feature maps are then input into a head network with parallel detection and segmentation branches to obtain multi-level classification and regression features. The multi-level classification and regression features are then aggregated and enhanced using a spatial information enhancement unit. The aggregated and enhanced classification and regression features are then concatenated to obtain bounding box features. Mask features are generated from the multi-level feature maps. Finally, a two-stream residual feature fusion unit, guided by the bounding box features, performs spatial fusion on the mask features. This method enhances perceptual ability and semantic coherence to obtain fused mask features. The fused mask features are then concatenated with relative coordinates to obtain the result, which is input into a fully convolutional segmentation head to generate a prediction mask. Loss functions for the detection and segmentation branches are calculated based on the instance segmentation prediction results and corresponding horizontal bounding box annotations. These loss functions are then used to train the feature extraction network, neck network, head network, inter-information enhancement unit, two-stream residual feature fusion unit, and the fully convolutional segmentation head. A remote sensing image segmentation network is then constructed based on the trained feature extraction network, neck network, and head network. This network is used to segment remote sensing images, yielding target detection and segmentation results. This method improves the spatial information utilization of bounding box supervision methods, performs well in bounding box-supervised remote sensing instance segmentation tasks, and narrows the gap between bounding box supervision methods and fully supervised methods. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating a remote sensing image segmentation method based on spatial affinity learning in one embodiment. Figure 2 This is a schematic diagram of the overall architecture of the method in one embodiment; Figure 3 This is a schematic diagram of the spatial information enhancement unit in one embodiment; Figure 4 This is a schematic diagram of the structure of a dual-stream residual gated fusion unit in one embodiment; Figure 5 This is a schematic diagram illustrating a pixel misclassification case caused by ignoring spatial priors in one embodiment. Figure 6 This is a schematic diagram of spatial affinity in one embodiment; Figure 7 This is a schematic diagram illustrating the confusion matrix analysis of this method on the iSAID dataset during ablation experiments; Figure 8 A schematic diagram showing the comparison between the truth map in the experiment and the instance segmentation results using the SOLO, SOLOv2, and BoxInst methods; Figure 9 For comparison experiments based onFigure 8 The truth graph in the diagram uses DiscoBox, Box2Mask, and the instance segmentation results of this method. Figure 10 This is a structural block diagram of a remote sensing image segmentation device based on spatial affinity learning in one embodiment. Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] Fully supervised instance segmentation methods have achieved high segmentation accuracy in complex remote sensing scenes thanks to fully annotated pixel-level supervision signals. However, pixel-by-pixel fine annotation of massive high-resolution remote sensing images suffers from problems such as strong reliance on expert knowledge, high manual time costs, and long data iteration cycles, which severely limit the development and application of instance segmentation methods. Weakly supervised methods use image-level labels, bounding boxes, and other weak annotations to explore the performance of instance segmentation under different weak supervision signals. Among them, box-supervised methods have advantages in balancing annotation efficiency and model performance because horizontal bounding box annotations are easy to obtain, reducing annotation costs while preserving instance-level spatial information. Existing box-supervised methods mainly convert horizontal bounding box weak supervision into pixel-level mask supervision through network architecture and loss function design. However, current box-supervised remote sensing instance segmentation methods still suffer from insufficient utilization of spatial information and weak spatial modeling capabilities, mainly reflected in the separation of spatial localization and pixel prediction, and the neglect of spatial priors.

[0022] Therefore, in this application, as Figure 1 As shown, a remote sensing image segmentation method based on spatial affinity learning is provided, including the following steps: Step S100: Obtain the training dataset, which includes multiple remote sensing training images and their corresponding horizontal bounding box annotations.

[0023] Step S110: The remote sensing training image is passed through the feature extraction network and the neck network in sequence to obtain a multi-level feature map.

[0024] Step S120: Input the multi-level feature map into the head network with parallel detection branch and segmentation branch to obtain multi-level classification features and multi-level regression features. Perform multi-level feature aggregation and feature enhancement on the multi-level classification features and multi-level regression features through the spatial information enhancement unit. Then, concatenate the aggregated and enhanced classification features and regression features to obtain bounding box features.

[0025] Step S130: A mask feature is generated from the multi-level feature map. The spatial perception capability and semantic coherence of the mask feature are enhanced by the dual-stream residual feature fusion unit under the guidance of the bounding box feature to obtain the fused mask feature.

[0026] Step S140: The result obtained by concatenating the fused mask features and relative coordinates is input into the fully convolutional segmentation head to obtain the prediction mask.

[0027] Step S150: Calculate the loss function of the detection branch and the segmentation branch based on the instance segmentation prediction result and the corresponding horizontal box annotation, and use the loss function to train the feature extraction network, neck network, head network, inter-information enhancement unit, two-stream residual feature fusion unit and fully convolutional segmentation head, and construct the remote sensing image segmentation network based on the trained feature extraction network, neck network and head network.

[0028] Step S160: Acquire remote sensing images, and use a remote sensing image segmentation network to segment the remote sensing images to obtain target detection results and segmentation results.

[0029] This application proposes a remote sensing image instance segmentation network framework (SALInst), such as... Figure 2 As shown, the framework mainly includes: a backbone network for feature extraction, a neck network for multi-scale feature processing, and a head network for generating class, bounding box, and mask predictions.

[0030] In step S100, the multiple remote sensing training images in the training dataset are the input images. ,in This refers to the batch size; 3 indicates the RGB color three-channel configuration. and These represent the height and width of the input image, respectively.

[0031] In step S110, the remote sensing training image is first fed into the backbone network ResNet for feature extraction. Then, the extracted feature maps {C3, C4, C5} are fed into the neck network FPN for multi-scale processing, resulting in fused multi-scale features {P3, P4, P5, P6, P7}. , Indicates the index of the feature layer number. This indicates the channel dimension of each layer. This represents the cumulative step size. Then, the multi-scale features are fed into a head network with parallel detection and segmentation branches for task-specific processing.

[0032] In step S120, the head network includes multiple detection head branches that process feature maps at each level. Each detection head branch includes a parallel detection branch and a segmentation branch. The detection branch outputs a classification score, and the segmentation branch outputs regression coordinates, center point scores, and controller parameters for segmentation head weight allocation.

[0033] Specifically, features P3 through P7 are fed into five shared detection heads to generate classification scores, regression coordinates, centroid scores, and controller parameters for segmentation head weight allocation. The multi-level classification and regression features processed by the detection head are used to construct the detection feature stream, and the bounding box features are realized through the spatial information enhancement unit. Fine-grained optimization is then performed, followed by a dual-stream residual gated fusion mechanism in step S130, which is combined with mask features derived from P3 to P5. Adaptive fusion is performed to generate fused mask features with spatial awareness.

[0034] Considering that bounding box supervised instance segmentation methods typically employ a dual-branch network design of detection and segmentation, where the detection branch and segmentation branch independently learn different features—that is, the detection head network processes regression features with instance spatial location information and classification features with semantic category information (collectively referred to as bounding box features), while the segmentation head network relies solely on mask features to generate pixel-level predictions—this method proposes a fusion of bounding box features and mask features to fully utilize the rich spatial information in bounding box features. By using bounding box features to provide additional spatial information guidance for mask features, this achieves a synergistic optimization of localization accuracy and segmentation detail.

[0035] Furthermore, considering that in the detection branch, five independent detection heads process multi-scale features from the FPN (P3 to P7, where high-resolution features contain fine-grained spatial details and low-resolution features contain global semantic information), the output regression and classification features are also distributed in multiple levels. If these multi-level features are directly used for fusion with mask features, the inconsistent scales of different levels will lead to spatial information misalignment, and redundant features may interfere with the semantic coherence of the mask features. Therefore, it is necessary to first aggregate the multi-level bounding box features to unify the feature scale and construct a coherent detection feature flow.

[0036] In step S130, when performing multi-level feature aggregation and feature enhancement on multi-level classification features and multi-level regression features through the spatial information enhancement unit: the classification features or regression features of adjacent levels are concatenated, and then the concatenated features are input into the spatial information enhancement unit for enhancement to obtain the aggregated enhanced classification features and regression features of each level. Then, the aggregated enhanced classification features and aggregated enhanced classification and regression features of each level are aggregated in multiple levels to obtain aggregated enhanced classification features and aggregated enhanced regression features. Finally, the aggregated enhanced classification features and aggregated enhanced regression features are concatenated to obtain the bounding box features.

[0037] In this embodiment, since the high-resolution features (P3 to P5) contain richer spatial information, and the mask features are generated from feature maps P3 to P5 through a series of convolution, upsampling, and addition operations, in order to maintain consistency with the scale of the mask features, the P3 to P5 features output by the detection head are selected to construct the detection feature stream. Taking regression features as an example, firstly, through... Convolution expands the receptive field, capturing spatial dependencies between adjacent pixels. Then, the low-resolution feature map is progressively upsampled to the same size as the highest-resolution feature map using bilinear interpolation to ensure spatial alignment. Subsequently, adjacent feature maps are... Convolution adjusts the channel dimension and outputs aggregated regression features. This involves concatenating categorical or regression features from adjacent layers. The processing flow for categorical and regression features is consistent; the output categorical and regression features are concatenated along the channel dimension to form bounding box features with localization and semantic information. Finally, this is processed... Convolution adjusts the channel dimensions to be the same as the channel dimensions of the mask features. The aggregation process of multi-level features is shown below: (1) (2) (3) In formulas (1) to (3), Indicates feature level, It is a regression feature. It is a classification feature. It is a bounding box feature. This indicates an upsampling operation, i.e., bilinear interpolation.

[0038] Furthermore, during the aggregation of multi-level features, the concatenated features are input into the spatial information enhancement unit for enhancement, resulting in aggregated enhanced classification features and regression features corresponding to each level. The aggregated enhanced classification features and aggregated enhanced classification regression features of each level are then aggregated in multiple levels to obtain aggregated enhanced classification features and aggregated enhanced regression features. Finally, the aggregated enhanced classification features and aggregated enhanced regression features are concatenated to obtain bounding box features.

[0039] In this method, a Spatial Information Augmentation Unit (SIE) is proposed to perform fine-grained optimization of bounding box features, aiming to improve the guiding performance of the detection feature flow for segmentation tasks. The structure of the Spatial Information Augmentation Unit (SIE) is as follows: Figure 3 As shown, the spatial information enhancement module mainly preserves and highlights the spatial structure information in the feature map while emphasizing spatial details and suppressing channel redundancy.

[0040] In this embodiment, in the spatial information enhancement unit: the feature obtained by concatenating adjacent layers of classification features or regression features is used as the input feature of the spatial information enhancement unit. The input feature is sequentially processed by average pooling of the channel dimension, convolutional channel compression, nonlinear transformation introduced by ReLU function activation, and then dimensionality increase by convolutional channels. Then, the channel weights are mapped to the interval of 0 to 1 by the sigmoid activation function to generate feature-optimized distribution weights. Finally, the distribution weights are multiplied element-wise with the input features to obtain the enhanced features.

[0041] Specifically, adjacent feature maps are upsampled and summed. After the convolution operation, the feature map is first subjected to channel-level average pooling to focus on the response intensity of each channel across the entire feature space. Then, the pooled feature map is further processed... Convolution performs channel compression, reducing computational complexity while avoiding interference from redundant channels on spatial information. After introducing a nonlinear transformation via ReLU activation, it is then... Convolution upscales the feature map to the original channel dimension. Convolution and activation processes generate optimized weights that adjust feature strength channel-by-channel, precisely adapting to different channels without affecting the spatial structure of the original feature map, thus emphasizing the spatial information of each channel. Finally, the sigmoid activation function maps the channel weights to the range of 0 to 1, generating distributed weights for feature optimization. These weights are multiplied element-wise with the input feature map, achieving dynamic adjustment of the features. The spatial information enhancement unit focuses on channel-dimensional feature optimization, aiming to filter and enhance the effective spatial information of key channels, indirectly improving the spatial detail recognition in the feature map, while suppressing channel redundancy, providing more accurate spatial guidance for the segmentation task.

[0042] Considering the lack of efficient feature interaction between the detection and segmentation branches in bounding box-supervised methods, although the detection branch can achieve accurate instance localization through bounding box annotations, the segmentation branch has limited semantic context and prior information on instance coordinates due to the lack of pixel-level labels, which leads to instance confusion at the feature level. Furthermore, the segmentation branch does not fully utilize the spatial information of the detection feature stream, failing to effectively associate instance regions with coordinates, thus restricting the alignment of the mask and bounding box. This method has constructed a spatially optimized detection feature stream using a spatial information enhancement unit, but simple feature fusion (such as addition or concatenation) cannot achieve dynamic adaptation between bounding box features and mask features. To fully utilize the spatial information in bounding box features, this method proposes a Dual-Stream Residual GateFusion (DSRGF) mechanism, which achieves information complementarity between detection and segmentation tasks through cross-branch feature interaction, providing additional spatial information guidance for mask features.

[0043] In step S130, in the dual-stream residual feature fusion unit: the bounding box features are first expanded through convolution, and then captured through linear deformable convolution to capture multi-scale context. The residuals are added and then activated by GELU to obtain the processed bounding box features. The mask features are expanded through convolution to the same channel as the bounding box features, and then multiplied element-wise with the processed bounding box features to achieve dynamic modulation to obtain the fused features. The fused features are compressed through convolution, and then fused with the mask features through residual connections to obtain the fused mask features.

[0044] like Figure 4 As shown, the input to the dual-stream residual feature fusion unit is dual-stream features, namely bounding box features that have undergone multi-level feature aggregation and spatial information enhancement modules. And mask features derived from feature maps P3 to P5. Bounding box features Firstly, through Convolution expands the channel dimension, followed by Linear Deformable Convolution (LD Conv) to capture multi-scale context, and then residuals are added to GELU activation to preserve the original semantics. Simultaneously, mask features... The features are expanded to the same channel dimension and then multiplied element-wise with the processed bounding box features to achieve dynamic modulation. Finally, the fused features are... Convolutional compression reduces the number of channels to the original number, and the residual connections are then added to the input mask features to form a complete gated path. The dual-stream residual gating fusion mechanism dynamically fuses bounding box features and mask features, enabling adaptive acquisition of target localization and structural priors from the detection branch. Meanwhile, the multi-layer residual connections maintain the detailed representation of features and the stability of training gradients, significantly enhancing the spatial awareness and semantic coherence of mask features.

[0045] In step S140, the fused mask features After concatenation with relative coordinates, we get , The data is fed into a fully convolutional segmentation head, which optimizes the results using projection loss and spatial affinity loss to generate accurate mask predictions. It should be noted that the fully convolutional segmentation head is integrated into the head network.

[0046] In step S150, the loss function for the detection branch includes classification loss, regression loss, and centrality loss, while the loss function for the segmentation branch includes projection loss and spatial affinity loss.

[0047] In this embodiment, the total loss is the sum of the losses of the detection branch and the segmentation branch, expressed as: (4) In formula (4), Indicates the total loss. and These represent detection loss and segmentation loss, respectively.

[0048] In this embodiment, the detection branch includes three subtasks: classification, regression, and centrality estimation. Therefore, the detection loss of SALInst is the classification loss. Regression loss and centrality loss The combination of is represented as: (5) In formula (5), Focal loss is used to effectively alleviate class imbalance in remote sensing images, and is expressed as: (6) (7) (8) In formulas (6) to (8), Represents the ground truth (GT) label. This represents the predicted probability. This is the Focal parameter used to modulate weight decay, preferably set to 2. It is the positive and negative sample sampling ratio, preferably set to 0.25.

[0049] Specifically, The GIoU loss is adopted, which effectively enhances the robustness of instance localization in remote sensing images by introducing a convex hull area penalty term between the predicted bounding box and the ground truth bounding box on top of the IoU. It is expressed as: (9) (10) (11) In formulas (9) to (11), Represents the prediction box. Represents the truth box, At the same time Include and The smallest bounding rectangle.

[0050] Specifically, It is the cross-entropy loss, which suppresses low-quality predictions of the target center by using the normalized distance from supervised feature points to the target center. It is expressed as: (12) In formula (12), Represents truth value label The total number, and They are the first The labels and scores of each prediction box.

[0051] In this embodiment, the segmentation loss introduces spatial weight modulation and total variational smoothing constraints on top of the projection loss and pairwise affinity loss of BoxInst. The segmentation loss is as follows: (13) In formula (13), This is the projection loss, which transforms the horizontal bounding box constraint into a mask region constraint, ensuring that the predicted mask region matches the ground truth bounding box. The formula for calculating the projection loss is as follows: (14) (15) (16) In formulas (14) to (16), We will continue using the Dice loss from CondInst. This represents a mask where the value is 1 within the truth box and 0 in the rest of the area. The mask representing the network prediction can be regarded as the foreground probability. and These represent the mask in and Projection along the direction, by direction and Implemented for maximum pooling operations in the direction.

[0052] Pairwise affinity loss transforms fully supervised mask annotation into weakly labeled bounding box supervision by forcing prediction consistency between each pixel and its neighbors. However, the original pairwise affinity loss is designed based on color similarity, treating all neighboring pixels equally and ignoring the spatial prior that "spatially neighboring pixels are more likely to belong to the same instance." Figure 5 As shown, ignoring spatial priors can lead to misclassification of distant pixels with similar colors but belonging to different instances, making pairwise affinity loss susceptible to noise and redundant long-range connections. Furthermore, relying solely on projection loss and pairwise affinity loss often results in mask fragmentation and boundary noise due to the lack of global smoothing regularization, making it difficult to maintain mask structural consistency. Therefore, this method proposes a spatial affinity loss, expressed as: (17) In formula (17), This represents the spatial weights based on the Gaussian kernel. Represents the total variation loss. It is a color-driven pairwise affinity loss. To achieve the transformation of the supervision method, an undirected graph was first constructed. ,in It is a collection of pixels. It is a set of edges. It should be noted that... Each pixel in it is associated with A void neighborhood is connected ( (where the neighborhood size is). Then, define... As an edge Tags: This indicates that two pixels on the same edge belong to the same truth label. This indicates they belong to different labels. Let the pixels... and For the edge The two endpoints of the network at the pixels Predicted value at location This can be considered as the probability that the pixel is in the foreground. Therefore, The probability is expressed as: (18) To achieve training that relies solely on bounding box annotations, pairwise affinity loss introduces color similarity, which is defined as: (19) In formula (19), It is the edge Color similarity, and It is a pixel and Color vectors in the LAB color space This is a hyperparameter, set to 2 in this paper. Pairwise affinity loss applies only when color similarity is greater than a threshold. The calculation on the edges is represented as: (20) In formula (20), It is a horizontal frame with at least one pixel on its edge. yes The number of middle edges, This indicates that when the color similarity is greater than the threshold... An indicator function that is 1 at time, preferably, uses a threshold. Set it to 0.3.

[0053] Furthermore, in order to make full use of spatial priors, this application proposes a spatial weighting for distance attenuation. This weight is designed based on a Gaussian kernel. For example... Figure 6 As shown, spatial weights The design concept is based on color similarity, forcing adjacent pixels within a neighborhood to have a higher probability of belonging to the same instance. The formula for calculating the spatial weight at a given location is as follows: (twenty one) In formula (21), is the radius of the Gaussian kernel, used to control the distance decay rate, preferably set to 1.0. By multiplying the spatial weights element-wise with the pairwise affinity loss, the segmentation loss function introduces spatial priors, enabling perception of spatial distances between pixels.

[0054] Furthermore, due to the lack of strong constraints from pixel-level annotations in the box-supervised model, its predicted mask is prone to noise responses and irregular pixel-level abrupt changes in local regions, affecting the visual quality and structural consistency of the segmentation results. To maintain the structural consistency of the predicted mask, a total variation loss is introduced to achieve smooth regularization constraints, expressed as: (twenty two) In Equation (22), represents the spatial location of a pixel. The total variation loss aims to suppress local discontinuities by constraining the gradient changes of the mask in the horizontal and vertical directions. This loss function penalizes the intensity differences between adjacent pixels, enabling the mask to achieve local adaptive smoothing while maintaining the main structure, thereby improving the overall quality of the prediction mask.

[0055] In this embodiment, after the model converges, a remote sensing image segmentation network is constructed based on the trained feature extraction network, neck network, and head network.

[0056] In this paper, the effectiveness of the proposed method is also demonstrated through ablation experiments and comparative experiments.

[0057] In the ablation experiments, multiple ablation experiments were designed for different modules to verify the effectiveness of the proposed method and the rationality of the module design. Due to the diverse scenes, rich scale variations, and pixel-level annotation accuracy of the iSAID dataset, all ablation experiments were conducted on the iSAID dataset. In all tables related to ablation time, the baseline model is the BoxInst method, and the optimal result is indicated in bold. √ and × represent improvements made by adding and not adding the corresponding method, respectively.

[0058] To verify the effectiveness of each component in the SALInst method, this section first conducts module ablation experiments. As shown in Table 1, by progressively adding the dual-stream residual gated fusion mechanism (DSRGF) and the spatial affinity loss function (SA Loss), it can be found that each module can work collaboratively without interfering with each other, and progressively adding each component can achieve continuous performance improvement.

[0059] Table 1. Ablation experiment results of gradually adding each component of SALInst

[0060] It should be noted that when the DSRGF mechanism is added in Table 1, the constructed detection feature stream is first enhanced by the Spatial Information Enhancement (SIE) module. The experimental results in Table 1 show that the proposed spatial information enhancement module, the dual-stream residual gated fusion mechanism, and the spatial affinity loss can achieve effective spatial relationship modeling and improve instance segmentation accuracy. Specifically, the combined effect of DSRGF and SIE results in a 1.9% mAP performance gain compared to the baseline model, while the performance improvement brought by the spatial affinity loss is the most significant, further improving the mAP segmentation accuracy by 3.8%, particularly increasing the mAP_75 index to 19.0%. To analyze the complexity of each component, the overhead of the parameters (Params) and floating-point operations (FLOPs) of each component was further quantified. The results show that all components proposed in this paper are lightweight modules. After introducing DSRGF and SIE, compared to the baseline model, the Params only increase by 1.855 (M), and the FLOPs only increase by 0.03 (T). After adding spatial affinity loss, the model's Params and FLOPs remain unchanged. This indicates that spatial affinity loss, as an optimization strategy at the loss function level, guides feature learning by simply adjusting the training objective, without introducing additional network parameters or modules, achieving a "zero-cost" performance improvement.

[0061] To verify the effectiveness and design rationality of the dual-stream residual gated fusion DSRGF mechanism, a systematic ablation experiment was conducted on the feature fusion mechanism. Specifically, based on the enhanced bounding box features of the spatial information module, four feature fusion mechanisms were evaluated: direct addition, direct concatenation, simple gated fusion, and the DSRGF mechanism. By comparing the segmentation accuracy under different feature fusion mechanisms, it can be seen that the DSRGF mechanism performs better in optimizing cross-branch feature interactions.

[0062] Table 2. Experimental results of ablation based on feature fusion mechanism

[0063] As shown in Table 2, DSRGF achieved the highest performance among all tested fusion mechanisms, with a 1.0% mAP performance gain compared to the second-best direct addition fusion method. Compared to the other three simpler feature fusion mechanisms, DSRGF introduces residual paths to preserve the original semantic information, while avoiding channel dimension expansion through dimensional expansion and compression operations. This effectively fuses bounding box features and enhances the spatial awareness of mask features. Experimental results demonstrate the superiority of the proposed dual-stream residual gated fusion mechanism.

[0064] To verify the effectiveness of the spatial affinity loss and optimize its parameter settings, ablation experiments were conducted on a BoxInst model with SIE and DSRGF added, focusing on exploring the impact of different loss function configurations on instance segmentation performance. Specifically, a basic pairwise affinity loss was first used, followed by the introduction of Gaussian kernel-based spatial weights to enhance spatial distance awareness, and finally, the total variational smoothing loss was integrated to construct a spatial affinity loss to strengthen the spatial consistency of the prediction mask.

[0065] Table 3 Ablation Experiment Results with Different Loss Function Configurations

[0066] As shown in Table 3, adding spatial weights alone, on top of the pairwise affinity loss, achieves a 3.6% improvement in mAP performance, with a significant 5.1% improvement in mAP_75. This verifies that spatial weights effectively enhance the model's ability to perceive the spatial distance between neighboring pixels. By dynamically adjusting the spatial position weights based on spatial distance using a Gaussian kernel, the model can more accurately distinguish pixel differences between instances and the background. When only total variation loss is added, mAP actually decreases. This is because total variation loss, when acting alone, excessively constrains the spatial smoothness of the mask, causing the details of the predicted mask to be blurred, thus affecting the model's segmentation accuracy. However, when spatial weights are combined with total variation loss, the model performance reaches its optimal level, with mAP reaching 23.7%, and mAP_50 and mAP_75 also improving to 50.5% and 19.0%, respectively. Spatial weights and total variation loss form a complementary and synergistic mechanism. Spatial weights ensure the accuracy of pixel-level classification by strengthening spatial distance perception, while total variation loss constrains the spatial consistency of the mask to suppress noise interference. Together, they construct a spatial affinity learning framework. Experimental results verify the effectiveness of the spatial affinity loss proposed in this paper.

[0067] To optimize the parameters of the spatial affinity loss, an additional ablation experiment was designed for the Gaussian kernel radius parameter of the spatial weights. The Gaussian kernel radius ranged from 1.0 to 3.0, with a step size of 0.5. As shown in Table 4, the segmentation effect was optimal when the Gaussian kernel radius was 1.0, with an mAP of 23.7%. Simultaneously, mAP_50 and mAP_75 reached 50.5% and 19.0% respectively, both higher than the segmentation results of other Gaussian kernel radius settings. The experimental results show that as the Gaussian kernel radius increases, the scope of the spatial weights expands, incorporating too many background pixels around the instance edges into the affinity calculation, blurring the boundary between the instance and the background. Especially under high IoU threshold (mAP_75), an excessively large Gaussian kernel radius leads to a significant decrease in segmentation accuracy. Therefore, selecting a Gaussian kernel radius of 1.0 effectively balances enhancing spatial distance perception and suppressing background pixel interference, validating the rationality of the spatial affinity loss parameter selection.

[0068] Table 4. Results of ablation experiments using spatial weight parameters

[0069] Next, to verify the effectiveness and feasibility of SALInst (the method proposed in this paper) in remote sensing instance segmentation tasks, extensive comparative experiments were conducted on the iSAID dataset, comparing SALInst with several representative box-supervised instance segmentation methods and some fully supervised instance segmentation methods. The box-supervised comparison methods included the baseline models BoxInst, DiscoBox, Box2Mask, and BoxSnake, while the fully supervised comparison methods included Mask R-CNN, CondInst, SOLO, and SOLOv2. All tables below illustrate this. The backbone network uses ResNet101, and the best results (limited to comparisons between box-supervised methods) are indicated in bold.

[0070] Table 5. Comparative experimental results of different methods on the iSAID dataset.

[0071] As shown in Table 5, the proposed SALInst method significantly outperforms existing representative box-supervised instance segmentation methods on the iSAID dataset, narrowing the performance gap between box-supervised and fully supervised methods. Specifically, taking ResNet50 as the backbone network as an example, under the same training settings, SALInst surpasses existing representative box-supervised instance segmentation methods with minimal computational overhead (params increase by only 1.855M, FLOPs increase by only 0.03T), achieving an mAP of 23.7%, a 5.7% performance improvement compared to the baseline model. Furthermore, SALInst achieves mAP_50 and mAP_75 of 50.5% and 19.0%, respectively, with mAP_50 showing a 3.6% performance improvement compared to the suboptimal BoxSnake model. Furthermore, when using ResNet101 as the backbone, SALInst achieved a segmentation accuracy (mAP) of 24.7%, with mAP_50 and mAP_75 reaching 51.3% and 20.3%, respectively. This represents a 4.5% performance gain compared to the baseline model using ResNet101 as the backbone, and its segmentation performance is close to that of some fully supervised instance segmentation methods (e.g., compared to the SOLO method using ResNet50 as the backbone, SALInst with ResNet101 as the backbone has a performance difference of only 0.8%). Experimental results demonstrate that SALInst maintains high model efficiency while reducing annotation costs and achieving the best instance segmentation performance, collectively validating its superiority. It should be noted that although SALInst's performance has reached the state-of-the-art level of box-supervised methods, it still lags significantly behind current state-of-the-art remote sensing instance segmentation methods (such as the open-vocabulary instance segmentation framework SCORE and the reinforcement learning-based iterative optimization segmentation framework RL-ISegNet). Beyond the inherent differences in the strength of the supervisory signals, the latest instance segmentation methods have achieved significant innovations in network structure design and semantic space construction, introducing a multi-granularity scene context modeling mechanism and a reinforcement learning-driven iterative optimization strategy. These findings demonstrate that fusing regional and global contextual information can significantly enhance the model's understanding of complex remote sensing scenes, while employing reinforcement learning to optimize the learning process for hard samples effectively improves data utilization efficiency, providing a promising technical path for future research in the field of instance segmentation.

[0072] Table 6. Class-by-class comparison results of different bounding box supervision methods on the iSAID dataset.

[0073] To further validate the application value of SALInst in remote sensing scenarios, a comparative analysis was conducted on the class-by-class instance segmentation results of different box-supervised methods on the iSAID dataset. As shown in Table 6, SALInst achieved good segmentation performance in most categories, especially for categories such as stadiums, large vehicles, and ports. This improvement in mAP performance is attributed to the spatial affinity learning mechanism's ability to constrain the spatial structure of geometric structures (e.g., the structured stadium). Simultaneously, the spatial information enhancement and dual-stream residual gating fusion mechanism effectively preserved the spatial structure and contour details of instances. Experimental results demonstrate that this mechanism is suitable for segmentation tasks of instances with significant geometric features in remote sensing scenarios.

[0074] The superior performance of SALInst in class-specific instance segmentation was demonstrated by analyzing the instance class prediction confusion matrix of SALInst's instance segmentation results on the iSAID dataset. Figure 7 As shown, the diagonal elements of the confusion matrix reflect higher classification accuracy, while the off-diagonal elements represent misclassification between categories. The results show that SALInst has stronger discriminative power in predicting easily confused challenging categories (such as ports and ships) and can effectively solve the semantic ambiguity problem of geometrically similar categories, which further illustrates the robustness of the spatial affinity learning framework.

[0075] Quantitative comparative analysis results show that the proposed SALInst method outperforms other bounding box supervision methods in remote sensing instance segmentation tasks, and narrows the gap between bounding box supervision methods and fully supervised methods. To further verify the superiority of SALInst, a visual qualitative comparative analysis of the instance segmentation results of different methods will be conducted. Figure 8 and Figure 9 The visualization results of instance segmentation using different methods on the iSAID dataset are presented. Figure 8 middle Figure 8 (a) is the truth value. Figure 8 (b) to Figure 8 (d) shows the instance segmentation results of the comparison methods SOLO, SOLOv2, and BoxInst, respectively. Figure 9 middle Figure 9 (a) to Figure 9(c) The instance segmentation results of DiscoBox, Box2Mask, and the proposed method SALInst are displayed respectively. The visualization results show that the comparative methods, especially the box-supervised methods, exhibit significant performance degradation in complex remote sensing scenes, suffering from issues such as missed detection of small targets, instance aggregation, and semantic ambiguity. In contrast, SALInst achieves superior instance segmentation results, particularly in scenes with complex backgrounds and densely distributed small targets, where SALInst effectively detects and segments targets at multiple scales. Through comprehensive verification using quantitative and qualitative analysis, experimental results demonstrate that the spatial affinity learning mechanism can reduce misclassification between background pixels and instances, and effectively distinguish semantically easily confused categories such as ports and ships, validating the superiority of SALInst.

[0076] Among the aforementioned remote sensing image segmentation methods based on spatial affinity learning, SALInst, a box-supervised remote sensing instance segmentation method based on spatial affinity learning, is proposed. Addressing the lack of effective utilization of spatial information from bounding box features in box-supervised methods, the idea of ​​fusing bounding box features with segmentation features is introduced. By designing spatial information enhancement strategies, a two-stream residual gated fusion mechanism, and a spatial affinity loss function, the spatial perception capability of mask features is enhanced, and a color-driven pairwise affinity loss is extended, providing a more robust supervision paradigm for box-supervised instance segmentation. Furthermore, ablation experiments and comparative experiments are conducted to verify the effectiveness and design rationality of the SALInst instance segmentation model. Experimental results show that SALInst improves the spatial information utilization rate of box-supervised methods, can better complete the box-supervised remote sensing instance segmentation task, and narrows the gap between box-supervised methods and fully supervised methods.

[0077] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0078] In one embodiment, such as Figure 10As shown, a remote sensing image segmentation device based on spatial affinity learning is provided, comprising: a training data acquisition module 200, a multi-level feature extraction module 210, a bounding box feature acquisition module 220, a fusion mask feature acquisition module 230, an instance segmentation prediction module 240, a network training module 250, and a remote sensing image instance segmentation module 260, wherein: The training data acquisition module 200 is used to acquire the training dataset, which includes multiple remote sensing training images and corresponding horizontal bounding box annotations. The multi-level feature extraction module 210 is used to sequentially pass the remote sensing training image through a feature extraction network and a neck network to obtain a multi-level feature map; The bounding box feature acquisition module 220 is used to input the multi-level feature map into a head network with parallel detection branches and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are respectively aggregated and enhanced by the spatial information enhancement unit. The aggregated and enhanced classification features and regression features are then concatenated to obtain the bounding box features. The fusion mask feature acquisition module 230 is used to generate mask features from the multi-level feature map. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain the fused mask features. The instance segmentation prediction module 240 is used to input the result obtained by concatenating the fused mask features and relative coordinates into the fully convolutional segmentation head to obtain the prediction mask; The network training module 250 is used to calculate the loss function of the detection branch and the segmentation branch based on the instance segmentation prediction result and the corresponding horizontal box label, and to use the loss function to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit and the fully convolutional segmentation head, and to construct the remote sensing image segmentation network based on the trained feature extraction network, the neck network and the head network. The remote sensing image instance segmentation module 260 is used to acquire remote sensing images, perform image segmentation on the remote sensing images using the remote sensing image segmentation network, and obtain target detection results and segmentation results.

[0079] Specific limitations regarding the remote sensing image segmentation device based on spatial affinity learning can be found in the limitations of the remote sensing image segmentation method based on spatial affinity learning mentioned above, and will not be repeated here. Each module in the aforementioned remote sensing image segmentation device based on spatial affinity learning can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0080] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a remote sensing image segmentation method based on spatial affinity learning. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0081] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0082] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Obtain a training dataset, which includes multiple remote sensing training images and their corresponding horizontal bounding box annotations; The remote sensing training images are sequentially passed through a feature extraction network and a neck network to obtain multi-level feature maps; The multi-level feature map is input into a head network with parallel detection and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are then aggregated and enhanced by a spatial information enhancement unit. Finally, the aggregated and enhanced classification features and regression features are concatenated to obtain bounding box features. Mask features are generated from the multi-level feature maps. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain fused mask features. The result obtained by concatenating the fused mask features with the relative coordinates is input into the fully convolutional segmentation head to obtain the prediction mask; The loss functions of the detection branch and the segmentation branch are calculated based on the segmentation prediction results of the instance and the corresponding horizontal box annotations. The loss functions are then used to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit, and the fully convolutional segmentation head. A remote sensing image segmentation network is then constructed based on the trained feature extraction network, the neck network, and the head network. Acquire remote sensing images, and use the remote sensing image segmentation network to segment the remote sensing images to obtain target detection results and segmentation results.

[0083] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain a training dataset, which includes multiple remote sensing training images and their corresponding horizontal bounding box annotations; The remote sensing training images are sequentially passed through a feature extraction network and a neck network to obtain multi-level feature maps; The multi-level feature map is input into a head network with parallel detection and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are then aggregated and enhanced by a spatial information enhancement unit. Finally, the aggregated and enhanced classification features and regression features are concatenated to obtain bounding box features. Mask features are generated from the multi-level feature maps. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain fused mask features. The result obtained by concatenating the fused mask features with the relative coordinates is input into the fully convolutional segmentation head to obtain the prediction mask; The loss functions of the detection branch and the segmentation branch are calculated based on the segmentation prediction results of the instance and the corresponding horizontal box annotations. The loss functions are then used to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit, and the fully convolutional segmentation head. A remote sensing image segmentation network is then constructed based on the trained feature extraction network, the neck network, and the head network. Acquire remote sensing images, and use the remote sensing image segmentation network to segment the remote sensing images to obtain target detection results and segmentation results.

[0084] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A remote sensing image segmentation method based on spatial affinity learning, characterized in that, The method includes: Obtain a training dataset, which includes multiple remote sensing training images and their corresponding horizontal bounding box annotations; The remote sensing training images are sequentially passed through a feature extraction network and a neck network to obtain multi-level feature maps; The multi-level feature map is input into a head network with parallel detection and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are then aggregated and enhanced by a spatial information enhancement unit. Finally, the aggregated and enhanced classification features and regression features are concatenated to obtain bounding box features. Mask features are generated from the multi-level feature maps. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain fused mask features. The result obtained by concatenating the fused mask features with the relative coordinates is input into the fully convolutional segmentation head to obtain the prediction mask; The loss functions of the detection branch and the segmentation branch are calculated based on the segmentation prediction results of the instance and the corresponding horizontal box annotations. The loss functions are then used to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit, and the fully convolutional segmentation head. A remote sensing image segmentation network is then constructed based on the trained feature extraction network, the neck network, and the head network. Acquire remote sensing images, and use the remote sensing image segmentation network to segment the remote sensing images to obtain target detection results and segmentation results.

2. The remote sensing image segmentation method based on spatial affinity learning according to claim 1, characterized in that, The head network includes multiple detection head branches that process feature maps at various levels, and each detection head branch includes a parallel detection branch and a segmentation branch. The detection branch outputs a classification score, and the segmentation branch outputs regression coordinates, center point scores, and controller parameters for segmentation head weight allocation.

3. The remote sensing image segmentation method based on spatial affinity learning according to claim 1, characterized in that, When performing multi-level feature aggregation and feature enhancement on the multi-level classification features and multi-level regression features respectively through the spatial information enhancement unit: After concatenating the classification or regression features of adjacent levels, the concatenated features are then input into the spatial information enhancement unit for enhancement, resulting in aggregated enhanced classification and regression features for each level. The aggregated enhanced classification features and aggregated enhanced classification regression features at each level are aggregated in multiple levels to obtain aggregated enhanced classification features and aggregated enhanced regression features. Then, the aggregated enhanced classification features and aggregated enhanced regression features are concatenated to obtain the bounding box features.

4. The remote sensing image segmentation method based on spatial affinity learning according to claim 3, characterized in that, In the spatial information enhancement unit: The feature obtained by concatenating adjacent levels of classification or regression features is used as the input feature of the spatial information enhancement unit; The input features are sequentially processed by average pooling of the channel dimension, convolutional channel compression, nonlinear transformation introduced by ReLU activation, and then dimensionality increase by convolutional channels. Finally, the channel weights are mapped to the interval of 0 to 1 by the sigmoid activation function to generate feature-optimized distribution weights. The distributed weights are multiplied element-wise with the input features to obtain the enhanced features.

5. The remote sensing image segmentation method based on spatial affinity learning according to claim 1, characterized in that, In the dual-stream residual feature fusion unit: The bounding box features are first expanded through convolution, then captured through linear deformable convolution to capture multi-scale context, and the residuals are added and then activated by GELU to obtain the processed bounding box features. The mask features are expanded to the same channel as the bounding box features through convolution, and then multiplied element-wise with the processed bounding box features to achieve dynamic modulation and obtain fused features. The fused feature is compressed through convolution and then fused with the mask feature via residual connection to obtain the fused mask feature.

6. The remote sensing image segmentation method based on spatial affinity learning according to claim 1, characterized in that, The loss function of the detection branch includes classification loss, regression loss, and centrality loss; The loss function for the segmented branches includes projection loss and spatial affinity loss.

7. The remote sensing image segmentation method based on spatial affinity learning according to claim 6, characterized in that, The spatial affinity loss is expressed as: In the above formula, This represents the spatial weights based on the Gaussian kernel. Represents the total variation loss. It is a color-driven pairwise affinity loss.

8. A remote sensing image segmentation device based on spatial affinity learning, characterized in that, The device includes: The training data acquisition module is used to acquire the training dataset, which includes multiple remote sensing training images and corresponding horizontal bounding box annotations. A multi-level feature extraction module is used to sequentially pass the remote sensing training image through a feature extraction network and a neck network to obtain a multi-level feature map; The bounding box feature acquisition module is used to input the multi-level feature map into a head network with parallel detection and segmentation branches to obtain multi-level classification features and multi-level regression features. The multi-level classification features and multi-level regression features are respectively aggregated and enhanced by the spatial information enhancement unit. The aggregated and enhanced classification features and regression features are then concatenated to obtain the bounding box features. The fusion mask feature acquisition module is used to generate mask features from the multi-level feature map. Under the guidance of the bounding box features, the spatial perception capability and semantic coherence of the mask features are enhanced by the dual-stream residual feature fusion unit to obtain the fused mask features. The instance segmentation prediction module is used to input the result obtained by concatenating the fused mask features and relative coordinates into the fully convolutional segmentation head to obtain the prediction mask; The network training module is used to calculate the loss function of the detection branch and the segmentation branch based on the instance segmentation prediction result and the corresponding horizontal box label, and to use the loss function to train the feature extraction network, the neck network, the head network, the inter-information enhancement unit, the two-stream residual feature fusion unit and the fully convolutional segmentation head, and to construct the remote sensing image segmentation network based on the trained feature extraction network, the neck network and the head network. The remote sensing image instance segmentation module is used to acquire remote sensing images, perform image segmentation on the remote sensing images using the remote sensing image segmentation network, and obtain target detection results and segmentation results.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images