A remote sensing image reference segmentation method
By generating counterfactual text and performing feature alignment and difference fusion, the problem of confusing variables in remote sensing image referential segmentation is solved, improving segmentation accuracy and interpretability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-03
AI Technical Summary
Existing remote sensing image referencing and segmentation models have poor generalization ability in complex scenes, are easily affected by visual and linguistic confounding variables, and have difficulty capturing causal features, resulting in insufficient segmentation accuracy and interpretability.
The system generates counterfactual text that is semantically opposite to the original referential text, and extracts causal features by eliminating the influence of confounding variables through dual-path feature alignment and difference fusion.
It significantly improves the accuracy and interpretability of remote sensing image referencing segmentation, effectively eliminates false associations, and enhances segmentation performance in complex remote sensing scenarios.
Smart Images
Figure CN122336288A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a remote sensing image referencing and segmentation method. Background Technology
[0002] Remote sensing image denotation segmentation aims to accurately segment corresponding target regions from high-resolution remote sensing images based on natural language descriptions. However, due to the inherent characteristics of remote sensing images, such as complex scenes, large target scale variations, and dense small targets, existing models are easily affected by confounding variables such as visual context (e.g., complex backgrounds) and co-occurring words in language. This makes it difficult to capture the true causal features of the target, resulting in poor generalization ability of the model in atypical scenarios and a tendency to produce incorrect segmentation results.
[0003] To address the generalization problem caused by the complexity of remote sensing image features, most existing remote sensing image referencing and segmentation techniques employ empirical risk minimization strategies for model training. For example, some existing techniques introduce multi-scale interactive networks or cross-scale enhancement modules to adapt to the characteristics of large target scale variations and diverse rotation directions in remote sensing images, attempting to improve the model's segmentation accuracy in complex remote sensing contexts by enhancing the scale perception and spatial structure capture capabilities of feature representations.
[0004] While existing technologies have improved scale perception, they still tend to learn statistical correlations rather than causal relationships in data, failing to effectively distinguish semantically contradictory textual descriptions, lacking interpretability, and failing to eliminate the misleading influence of visual and linguistic confounding variables on decision-making. Therefore, how to eliminate the influence of confounding variables in remote sensing image referencing segmentation, enabling segmentation to focus on causal features truly relevant to the original referential text, and thereby improving the accuracy and interpretability of remote sensing image referencing segmentation, has become an urgent problem to be solved in this field. Summary of the Invention
[0005] This invention provides a remote sensing image referencing segmentation method to solve the technical problem of how to eliminate the influence of confusing variables in remote sensing image referencing segmentation in order to improve segmentation accuracy and interpretability.
[0006] This invention provides a remote sensing image referencing segmentation method, comprising: Generate counterfactual text that is the opposite of the original referential text semantics corresponding to the remote sensing image; Align the original referential text with the visual features of the remote sensing image to obtain basic fusion features; simultaneously align the counterfactual text with the visual features to obtain counterfactual fusion features; Determine the difference features between the basic fusion features and the counterfactual fusion features, and fuse the difference features with the basic fusion features to obtain the enhanced alignment features; The enhanced alignment features are decoded to obtain the referential segmentation result of the remote sensing image.
[0007] According to a remote sensing image referential segmentation method provided by the present invention, counterfactual text with the opposite semantics to the original referential text corresponding to the remote sensing image is generated, including: The counterfactual text is obtained by replacing the foreground descriptors in the original referential text with the corresponding background descriptors, or by replacing the key attribute words in the original referential text with the corresponding antonyms.
[0008] According to a remote sensing image referential segmentation method provided by the present invention, the original referential text is aligned with the visual features of the remote sensing image to obtain basic fusion features; simultaneously, the counterfactual text is aligned with the visual features to obtain counterfactual fusion features, including: The original referential text and the counterfactual text are respectively encoded to obtain the original language features and the counterfactual language features; Attention is calculated on the original language features and the visual features to obtain the basic fusion features; simultaneously, attention is calculated on the counterfactual language features and the visual features to obtain the counterfactual fusion features.
[0009] According to a remote sensing image referencing segmentation method provided by the present invention, attention calculation is performed on the original linguistic features and the visual features to obtain the basic fusion features; simultaneously, attention calculation is performed on the counterfactual linguistic features and the visual features to obtain the counterfactual fusion features, including: The two-dimensional visual features are unfolded into a one-dimensional visual feature sequence; Using the visual feature sequence as the query feature and the original language features as the key and value features, a scaling dot product attention calculation is performed to obtain the basic fusion feature; Using the visual feature sequence as the query feature, and the counterfactual language features as the key and value features, a scaling dot product attention calculation is performed to obtain the counterfactual fusion feature.
[0010] According to a remote sensing image referencing segmentation method provided by the present invention, the method determines the difference features between the basic fusion features and the counterfactual fusion features, and fuses the difference features with the basic fusion features to obtain enhanced alignment features, including: The difference features are obtained by subtracting the basic fusion features and the counterfactual fusion features element by element. The basic fusion features are input into the gated network to calculate the weight distribution corresponding to the differential features; The weight distribution is multiplied element-wise with the difference feature, and the result of the multiplication is residually connected with the basic fusion feature to obtain the enhanced alignment feature.
[0011] According to a remote sensing image referencing segmentation method provided by the present invention, the gated network is used for: The basic fusion features are linearly transformed using a two-layer linear mapping network. The weight distribution is obtained by performing nonlinear activation calculations on the features after linear transformation using a linear rectified function and a hyperbolic tangent function.
[0012] The remote sensing image referencing segmentation method provided by the present invention further includes: Establish a training set containing sample remote sensing images, sample original referential text, and sample counterfactual text; The training set is used to train network parameters that map the sample remote sensing images, the sample original referential text, and the sample counterfactual text to the sample referential segmentation results.
[0013] According to a remote sensing image referencing segmentation method provided by the present invention, network parameters are trained using the training set to map the sample remote sensing image, the original referencing text of the sample, and the counterfactual text of the sample to the sample referencing segmentation result, including: Sample a uniformly distributed subset from the training set; Freeze the first network parameters that map the sample remote sensing image, the sample original referential text, and the sample counterfactual text to the sample enhancement alignment features, and use the subset to train the second network parameters that map the sample enhancement alignment features to the sample referential segmentation results; Unfreeze the first network parameters, and use the training set to jointly train the first network parameters and the second network parameters.
[0014] According to the remote sensing image referencing segmentation method provided by the present invention, after establishing a training set comprising sample remote sensing images, sample original referencing text, and sample counterfactual text, and before training network parameters to map the sample remote sensing images, the sample original referencing text, and the sample counterfactual text to the sample referencing segmentation results using the training set, the method further includes: Construct corresponding counterfactual mask labels for the sample counterfactual text, and add the sample counterfactual text and the corresponding counterfactual mask labels as augmented samples to the training set.
[0015] According to a remote sensing image referencing segmentation method provided by the present invention, constructing corresponding counterfactual mask labels for the sample counterfactual text includes: Extract the binary tensor annotation data corresponding to the original referential text of the sample; The pixel values representing the foreground and the pixel values representing the background in the binary tensor labeled data are swapped and inverted to obtain the counterfactual mask label.
[0016] The remote sensing image referential segmentation method provided by this invention generates counterfactual text with semantics opposite to the original referential text and performs dual-path feature alignment and difference fusion. This effectively eliminates the influence of confusing variables at the visual and linguistic levels, allowing the segmentation to focus on causal features that are truly related to the original referential text. This effectively eliminates false associations and significantly improves the segmentation accuracy and interpretability in complex remote sensing scenarios. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is a flowchart illustrating the remote sensing image referencing and segmentation method provided by the present invention.
[0019] Figure 2 This is a schematic diagram illustrating the principle of feature encoding of text provided by the present invention.
[0020] Figure 3 This is a schematic diagram illustrating the principle of generating enhanced alignment features provided by the present invention.
[0021] Figure 4 This is a schematic diagram illustrating the principle of the front-door intervention training strategy provided by the present invention.
[0022] Figure 5 This is a schematic diagram illustrating the principle of front-door intervention provided by the present invention.
[0023] Figure 6 This is a schematic diagram of a counterfactual sample provided by the present invention.
[0024] Figure 7 This is a schematic diagram of the remote sensing image referencing segmentation model provided by the present invention.
[0025] Figure 8 This is a comparison chart of the effects of different methods provided by this invention on the RefSegRS dataset.
[0026] Figure 9 This is a comparison chart of the effects of different methods provided by this invention on the RRSID dataset.
[0027] Figure 10 This is a comparison chart of the accuracy metrics of different methods provided by this invention on the RefSegRS dataset.
[0028] Figure 11 This is a comparison chart of the accuracy metrics of different methods provided by this invention on the RRSID dataset.
[0029] Figure 12 This is a schematic diagram of the remote sensing image reference segmentation device provided by the present invention.
[0030] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0032] In recent years, with the development of multimodal interaction and Transformer architecture, significant progress has been made in finger representation segmentation of natural images. However, due to the inherent characteristics of remote sensing images—such as complex scenes, large target scale variations, diverse orientations, and dense small targets—directly transferring these methods to remote sensing image tasks often yields poor results. To address this, researchers have proposed targeted improvements. For example, the LGCE module proposed in the paper "RRSIS: Referring Remote Sensing Image Segmentation" alleviates the scale variation problem through a cross-scale enhancement module, while the RMSIN framework proposed in the paper "Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation" introduces rotation and scale-aware structures to adapt to the characteristics of remote sensing images.
[0033] Nevertheless, from the perspective of causal reasoning, existing RRSIS models still have two fundamental flaws: Sensitive to confounding variables: During training, the model may capture spurious associations introduced by the context (such as background and co-occurring words) rather than the true causal features of the target itself, resulting in poor generalization ability of the model in atypical scenarios.
[0034] Insufficient interpretability: The model cannot answer counterfactual questions such as "Would the prediction result be different if the input text were to change in some way?", making it difficult to verify whether its decisions are truly driven by key semantics in the text.
[0035] Most existing models employ empirical risk minimization strategies, tending to learn statistical correlations rather than causal relationships in the data. This makes the models highly susceptible to confounding variables. For example, a model might incorrectly associate "baseball field" with "grass boundary," causing the segmentation results to fail when encountering a baseball field located on bare soil or with unclear grass boundaries. Similarly, linguistic co-occurrence biases (such as "vehicles" often co-occurring with "roads") can mislead the model, causing it to ignore crucial locational semantic information.
[0036] Existing models cannot effectively distinguish between semantically opposite textual descriptions. For example, for the mutually exclusive descriptions of "driving vehicles" and "parked vehicles," the model may output the same segmentation result because it fails to truly understand the driving role of attribute words ("driving" and "parked") in decision-making. This "black box" characteristic severely limits the deployment of models in remote sensing applications that require high reliability and accountability.
[0037] The following is combined Figures 1 to 13 The remote sensing image referencing segmentation method of the present invention is described.
[0038] Figure 1 This is a flowchart illustrating the remote sensing image referencing and segmentation method provided by the present invention, as shown below. Figure 1 As shown, the method includes, but is not limited to, steps S1, S2, S3 and S4.
[0039] Step S1: Generate counterfactual text that is semantically opposite to the original referential text corresponding to the remote sensing image.
[0040] The original referential text is a natural language sentence input by the user to describe a specific target region in a remotely sensed image. The counterfactual text is a textual description similar in lexical structure to the original referential text, but with completely opposite key semantic implications. Counterfactual text can be automatically constructed by performing part-of-speech analysis and semantic substitution on the original referential text using natural language processing algorithms.
[0041] For example, if the original referential text is "vehicles on the right", the generated counterfactual text could be "vehicles on the left".
[0042] Step S1 can provide a set of semantically mutually exclusive contrast samples to enable the model to perceive key semantic changes.
[0043] Step S2: Align the original referential text with the visual features of the remote sensing image to obtain the basic fusion features; at the same time, align the counterfactual text with the visual features to obtain the counterfactual fusion features.
[0044] Visual features are two-dimensional spatial feature matrices extracted from remote sensing images using convolutional neural networks or visual transformers. Basic fusion features are multimodal features formed by the interaction of semantic information from the original referential text and visual information from the image. Counterfactual fusion features are interfering multimodal features formed by the interaction of counterfactual text and visual information from the image. Feature alignment can be achieved by calculating the correlation between text features and image features using a cross-modal attention mechanism.
[0045] Step S2 can map information from different modalities into the same feature space, establishing a preliminary connection between vision and language.
[0046] Step S3: Determine the difference features between the basic fusion features and the counterfactual fusion features, and fuse the difference features with the basic fusion features to obtain the enhanced alignment features.
[0047] The difference features are the features that remain after removing the common information of the counterfactual fusion features from the difference features, and are truly driven by the semantics unique to the original referential text. The enhanced alignment features are multimodal features with clearer causal semantics after difference compensation. The difference between two features can be obtained through tensor subtraction, and the difference can be reinjected into the original features through feature concatenation or residual addition.
[0048] Step S3 can effectively suppress shared noise caused by confusing variables such as background and co-occurring words.
[0049] Step S4: Decode the enhanced alignment features to obtain the reference segmentation result of the remote sensing image.
[0050] The result of the denotation segmentation is a binary mask map with resolution consistent with the remote sensing image, indicating whether each pixel belongs to the target foreground. The abstract, high-dimensional enhanced alignment features can be reduced to pixel-level predicted probability maps using deconvolution or upsampling networks.
[0051] As can be seen from the above, by generating counterfactual text with semantics opposite to the original referential text and performing dual-path feature alignment and difference fusion, this invention can effectively eliminate the influence of confusing variables at the visual and linguistic levels, enabling segmentation to focus on causal features that are truly related to the original referential text, effectively eliminating false associations, and significantly improving segmentation accuracy and interpretability in complex remote sensing scenarios.
[0052] In one embodiment, step S1 may further include: By replacing foreground descriptors in the original referential text with corresponding background descriptors, or by replacing key attribute words in the original referential text with corresponding antonyms, counterfactual text can be obtained.
[0053] Foreground descriptors are nouns that describe the category of the target subject, such as "impermeable surface." Background descriptors are words that describe the area outside the target subject, such as "area other than the impermeable surface." Key attribute words are modifiers describing the target's location, size, color, etc., such as "large" or "right side." Antonyms are words whose meaning is opposite to that of key attribute words, such as "miniature" or "left side."
[0054] This invention provides a simple and efficient counterfactual text generation mechanism, ensuring the quality of counterfactual samples and helping to accurately capture the causal attributes that lead to changes in the target.
[0055] In one embodiment, step S2 may further include: Feature encoding is performed on the original referential text and the counterfactual text respectively to obtain the original language features and the counterfactual language features; Attention calculations are performed on the original linguistic features and visual features to obtain basic fusion features; simultaneously, attention calculations are performed on the counterfactual linguistic features and visual features to obtain counterfactual fusion features.
[0056] Combination Figure 2 As shown, Text is the original referential text. For counterfactual text, Text Encoder is a text encoder. F l Features of the original language This refers to counterfactual language features. Feature encoding can be achieved by extracting word vector sequences from the text using a pre-trained language model. Attention calculation can be performed by calculating the dot product similarity between the query, key, and value.
[0057] This invention ensures that the positive and negative semantics are aligned in the same visual metric space through feature encoding and attention calculation, making subsequent difference extraction more accurate and reliable.
[0058] In one embodiment, attention calculation is performed on the original linguistic features and visual features to obtain basic fusion features; simultaneously, attention calculation is performed on the counterfactual linguistic features and visual features to obtain counterfactual fusion features, which may further include: Two-dimensional visual features are unfolded into a one-dimensional sequence of visual features; Using visual feature sequences as query features and original language features as key and value features, scaling dot product attention calculation is performed to obtain basic fusion features; Using visual feature sequences as query features and counterfactual language features as key and value features, scaling dot product attention calculation is performed to obtain counterfactual fusion features.
[0059] A one-dimensional visual feature sequence is a set of pixel feature vectors arranged in spatial order. Feature unpacking can be achieved by merging the height and width dimensions through tensor reshaping operations.
[0060] Combination Figure 3 As shown, basic fusion features and counterfactual fusion features It can be calculated using the following formula: ; ; in, Visual features; , , It is a shared linear mapping matrix used to project features onto a specific subspace; flatten represents the unfolding operation; T represents the matrix transpose operation; C is the number of visual channels; softmax is the normalization exponential function. Figure 3 In the middle, Share represents shared weight.
[0061] This invention clarifies the mathematical implementation process of cross-modal attention, and through a visually guided query mechanism, it can effectively extract feature regions in images that are highly correlated with language instructions.
[0062] In one embodiment, step S3 may further include: The difference features are obtained by subtracting the basic fusion features and the counterfactual fusion features element by element. The basic fusion features are input into the gated network, and the weight distribution of the corresponding differential features is calculated. The weight distribution is multiplied element-wise with the difference features, and the result of the multiplication is residually connected with the basic fusion features to obtain the enhanced alignment features.
[0063] The gated network is a multilayer perceptron containing activation functions and linear layers. The weight distribution is an attention matrix with values between -1 and 1, used to control the proportion of differential features input.
[0064] Combination Figure 3 As shown, the formula for calculating the enhanced alignment feature A is as follows: ; Here, Gate stands for gated network. Indicates the weight distribution. This is an element-wise multiplication operation. These are the distinguishing features.
[0065] This invention, through dynamic gating mechanism and residual connection, can adaptively amplify key causal difference features while preserving the original stable semantic foundation.
[0066] In one embodiment, the gating network of the present invention can be used for: The basic fusion features are linearly transformed using a two-layer linear mapping network. The weight distribution is obtained by performing nonlinear activation calculations on the features after linear transformation using linear rectifier functions and hyperbolic tangent functions.
[0067] The formula for calculating the weight distribution is as follows: ; in, , It is a two-layer linear mapping network responsible for adjusting the feature dimensions; It is a linear rectified function used to preserve positive activation characteristics; It is a hyperbolic tangent function used to normalize the weight distribution to the interval [-1, 1] and dynamically control the proportion of differential features flowing in.
[0068] This invention enhances the expressive power of gated networks through nonlinear activation functions, making weight allocation more precise.
[0069] In one embodiment, the remote sensing image referencing segmentation method of the present invention may further include: Establish a training set containing sample remote sensing images, sample original referential text, and sample counterfactual text; The training set is used to train network parameters that map sample remote sensing images, sample original referential text, and sample counterfactual text to sample referential segmentation results.
[0070] The network parameters of the remote sensing image referencing segmentation model of this invention are obtained by mapping the sample remote sensing image, the original sample referential text, and the sample counterfactual text to the sample referencing segmentation results. The model can be trained by continuously updating the weight matrix of the neural network using the backpropagation algorithm and gradient descent optimizer, thereby providing the model with sufficient learning samples and a parameter update mechanism to ensure the acquisition of segmentation capabilities.
[0071] In one embodiment, training network parameters to map sample remote sensing images, sample original referential text, and sample counterfactual text to sample referential segmentation results using a training set may further include: Sample a uniformly distributed subset from the training set; Freeze the first network parameters that map the sample remote sensing image, the sample original referential text, and the sample counterfactual text to the sample enhancement alignment features, and use subset training to map the sample enhancement alignment features to the sample referential segmentation results. Unfreeze the first network parameters, and then jointly train the first and second network parameters using the training set.
[0072] Subsets can be collected through random sampling or cluster sampling. A well-balanced subset is a small amount of training data that contains a balanced proportion of various categories and scenes.
[0073] Combination Figure 4 As shown, the remote sensing image referencing segmentation model of the present invention includes an encoder and a decoder. After the training samples are input into the encoder, the encoder outputs sample enhancement alignment features. After the sample enhancement alignment features are input into the decoder, the decoder outputs the sample referencing segmentation result. The first network parameters are the network parameters of the encoder, and the second network parameters are the network parameters of the decoder.
[0074] This invention employs a front-door intervention training strategy to mitigate the impact of confusion factors. (In the causal graph...) Figure 5 In remote sensing image representation segmentation, uncontrollable contextual confusion factors C (such as visual background and linguistic co-occurring words) can cause spurious correlations between input X (images and text) and output Y (segmentation results). To cut off the backdoor path from C to X, this invention selects sample enhancement alignment features as mediating variables M. In remote sensing image representation segmentation tasks, the formula for front-door intervention can be expressed as: ; It's important to note that in visual tasks, the output of a certain intermediate layer in a neural network naturally possesses the potential to serve as a mediating variable. However, in the actual workflow, selecting a feature map from the output of a single layer as the mediating variable cannot sever all paths from X to Y. Therefore, the chosen mediating variable is the feature map tuple from the outputs of the four alignment modules, i.e. The present invention provides an approximate estimate. , The causal effects between the two factors are estimated to determine the composite effect, and the training strategy of the model is adjusted accordingly to simulate front door intervention.
[0075] In a feature extraction network with defined parameters, a unique input will correspond to a unique output; that is, there exists a certain... Make Therefore, the above formula can be rewritten as: ; The premise of this approximation is: ; in, This is a function of the encoder. Therefore, the causal effect between X and Y depends almost entirely on the causal effect between M and Y. When given At that time, the estimate of the output Y depends on the parameters of the decoder, and the decoder parameters are affected by... Influence: ; in The function of the decoder is subject to the input. Due to the influence of this, the above formula can be rewritten as: ; At this point, the above equation represents the expected value of the influence of a global input on the model parameters. Estimating the causal effect of X and Y can be transformed into calculating the parameters of the decoder function. The effect of the global input intervention can be used to approximate a decoder function, i.e.: ; In remote sensing image representation and segmentation tasks, if the above intervention method is directly applied, approximate parameters will be obtained. During the process, the entire training set needs to be traversed, which will reduce the overall training time complexity from... Sharply increased to Furthermore, the distribution of the training set is likely to deviate from the natural distribution and cannot play an intervention role. Therefore, it is necessary to pre-sample a small subset with a reasonable distribution and use the subset for front-door intervention training.
[0076] To approximate the aforementioned global intervention calculation in a neural network, this invention designs a two-stage sampling and fine-tuning strategy, such as... Figure 4 As shown: First, since the mediator variable M is entirely determined by the encoder, M is also unique when the input X is fixed. Freezing the first network parameters in the first stage (i.e., stopping its gradient updates) means that the mapping relationship P(M|X) is fixed. Subsequently, a small subset with a balanced data distribution is sampled from the training set, and only this subset is used to fine-tune the decoder. This process eliminates the statistical bias caused by the confusion factor C in the original data through the balanced sample distribution, thus approximately achieving global intervention on the decoder function. Finally, the first network parameters are unfrozen, gradient updates are resumed, and joint training is performed using the full training set. This strategy, without increasing computational complexity, cuts off the spurious associations of the confusion variable, enabling the model to truly learn causal features.
[0077] This invention effectively simulates causal front-door intervention without increasing computational complexity through a two-stage sampling and fine-tuning strategy, thus mitigating confusion bias in the decoding stage.
[0078] In one embodiment, after establishing a training set comprising sample remote sensing images, sample original referential text, and sample counterfactual text, and before using the training set to train network parameters that map the sample remote sensing images, sample original referential text, and sample counterfactual text to the sample referential segmentation results, the method of the present invention may further include: Construct corresponding counterfactual mask labels for sample counterfactual texts, and add the sample counterfactual texts and their corresponding counterfactual mask labels as augmented samples to the training set.
[0079] Combination Figure 6 As shown, (a) is the input sample remote sensing image, (b) is the original mask label corresponding to the original referential text of the sample (e.g., "impervious surface"), and (c) is the derived counterfactual mask label (corresponding to the counterfactual text "Areas excluding impervious surface").
[0080] In self-supervised loss function optimization, the model struggles to accurately understand the true semantics of counterfactual text when corresponding mask labels are lacking. This invention generates counterfactual mask labels and incorporates the counterfactual text and its corresponding mask labels into the training process, providing the model with direct counterfactual causal feedback (explicit supervision signals). This mechanism fundamentally solves the problem of ambiguous semantic understanding of counterfactual text in traditional self-supervised counterfactual training, forcing the model to establish a precise mapping between counterfactual semantics and counterfactual spatial regions. This significantly improves the model's ability to distinguish semantically opposite instructions and the interpretability of its decisions.
[0081] In one embodiment, constructing a corresponding counterfactual mask label for the sample counterfactual text may further include: Extract the binary tensor annotation data corresponding to the original referential text of the sample; The foreground and background pixel values in the binary tensor labeled data are swapped and inverted to obtain the counterfactual mask label.
[0082] The binary tensor annotation data consists of labels containing only 0s and 1s, where a pixel value of 1 represents the foreground and a pixel value of 0 represents the background. For counterfactual text with foreground-background transformation, the formula for calculating the counterfactual mask labels in this invention is: ; Where T is the original binary tensor labeled data, For counterfact masking labels.
[0083] For counterfactual text without a target (i.e., replacing key attributes with descriptions of objects that do not exist in the image), the corresponding counterfactual mask labels can be directly constructed as all-zero tensors.
[0084] This invention automatically generates counterfactual mask labels through mathematical operations of binary tensor interchange, which greatly reduces the cost of data labeling and can automatically generate massive amounts of counterfactual training samples for supervised training at zero cost.
[0085] Figure 7 The overall structure of the remote sensing image referencing segmentation model is shown. First, the model receives the original referential text input by the user, such as "The upper baseball field," and simultaneously inputs the corresponding remote sensing image.
[0086] The original referential text is input into the Counterfactual Generation module, which automatically generates counterfactual text that is semantically opposite, such as "The lower baseball field / Omit the upper baseball field".
[0087] The original referential text and the counterfactual text are encoded into raw linguistic features by a pre-trained language encoder (such as BERT). and counterfactual language features .
[0088] The input remote sensing image is processed by a Vision Encoder network (typically containing multiple layers, such as Layer 1 to Layer 4) to extract multi-scale visual features. .
[0089] The extracted linguistic and visual features are input into the Counterfactual Attention Multimodal Alignment (CAMA) module. In this module: primitive language features The basic fusion features are obtained by cross-modal attention computation with visual features.
[0090] Counterfactual language features The same calculations as for visual features are performed to obtain counterfactual fusion features.
[0091] The module calculates the difference between the two, extracts the causal difference features in the corresponding attention maps (A: Attentionmaps) by combining the gating network (removing common confusion noise), and performs residual fusion through element-wise product operation to output enhanced alignment features (shared weights ensure alignment in the same feature space).
[0092] The enhanced feature map is passed to the segmentation decoder predictor. A front-door intervention training strategy is applied here: the features output by CAMA are used as intermediate intervention variables.
[0093] The training phase consists of two steps. First, the network layers of the encoder stage are frozen. Then, a balanced subset is sampled from the training set to fine-tune only the decoder, thereby cutting off the paths of confusion factors. Finally, all parameters are unfrozen for joint training.
[0094] During training, in addition to using Ground Truth to supervise the original prediction, Counterfactual Targets generated by inverting the original mask will also be used as enhancement signals to jointly improve the predictor's ability to output the final high-precision segmentation mask.
[0095] Figure 8 and Figure 9 The diagrams show a comparison of the performance of different methods on the RefSegRS and RRSID datasets. The Input Image represents the input image, GT represents the ground truth, LAVT, RMSIN, and LGCE represent comparisons with existing models, and CAFI-RMSIN and CAFI-LGCE represent specific applications of this invention. The results show that the segmentation mask of this scheme is closer to the ground truth.
[0096] Figure 10 and Figure 11 The diagrams show a comparison of the accuracy metrics of different methods on the RefSegRS and RRSID datasets. It can be seen that the present invention achieves state-of-the-art performance in all key metrics, including global IoU (oIoU), average IoU (mIoU), and threshold accuracy (Pr@X), demonstrating the effectiveness and advancement of the invention.
[0097] The module proposed in this invention is a plug-and-play modular design that can be seamlessly integrated into various existing RRSIS models (such as LAVT, LGCE, RMSIN, etc.), bringing a dual improvement in performance and interpretability.
[0098] The remote sensing image identification and segmentation device provided by the present invention is described below. The remote sensing image identification and segmentation device described below and the remote sensing image identification and segmentation method described above can be referred to in correspondence with each other.
[0099] like Figure 12 As shown, the present invention provides a remote sensing image referencing and segmentation device, comprising: The generation module is used to generate counterfactual text that is semantically opposite to the original referential text corresponding to the remote sensing image; The alignment module is used to align the original referential text with the visual features of the remote sensing image to obtain the basic fusion features; at the same time, it aligns the counterfactual text with the visual features to obtain the counterfactual fusion features. The fusion module is used to determine the difference features between the basic fusion features and the counterfactual fusion features, and to fuse the difference features with the basic fusion features to obtain the enhanced alignment features; The decoding module is used to decode the enhanced alignment features to obtain the reference segmentation results of the remote sensing image.
[0100] Figure 13 A schematic diagram of the physical structure of an electronic device is provided. This electronic device may include a processor, a communications interface, memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute a remote sensing image denotation and segmentation method.
[0101] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the remote sensing image referencing and segmentation methods provided by the above methods.
[0103] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the remote sensing image referencing and segmentation methods provided by the above methods.
[0104] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0105] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A remote sensing image referencing segmentation method, characterized in that, include: Generate counterfactual text that is the opposite of the original referential text semantics corresponding to the remote sensing image; Align the original referential text with the visual features of the remote sensing image to obtain basic fusion features; simultaneously align the counterfactual text with the visual features to obtain counterfactual fusion features; Determine the difference features between the basic fusion features and the counterfactual fusion features, and fuse the difference features with the basic fusion features to obtain the enhanced alignment features; The enhanced alignment features are decoded to obtain the referential segmentation result of the remote sensing image.
2. The remote sensing image referencing segmentation method according to claim 1, characterized in that, Generate counterfactual text that is semantically opposite to the original referential text corresponding to the remote sensing image, including: The counterfactual text is obtained by replacing the foreground descriptors in the original referential text with the corresponding background descriptors, or by replacing the key attribute words in the original referential text with the corresponding antonyms.
3. The remote sensing image referencing segmentation method according to claim 1, characterized in that, Aligning the original referential text with the visual features of the remote sensing image yields basic fusion features; simultaneously, aligning the counterfactual text with the visual features yields counterfactual fusion features, including: The original referential text and the counterfactual text are respectively encoded to obtain the original language features and the counterfactual language features; Attention is calculated on the original language features and the visual features to obtain the basic fusion features; simultaneously, attention is calculated on the counterfactual language features and the visual features to obtain the counterfactual fusion features.
4. The remote sensing image referencing segmentation method according to claim 3, characterized in that, Attention is calculated on the original linguistic features and the visual features to obtain the basic fusion features; simultaneously, attention is calculated on the counterfactual linguistic features and the visual features to obtain the counterfactual fusion features, including: The two-dimensional visual features are unfolded into a one-dimensional visual feature sequence; Using the visual feature sequence as the query feature and the original language features as the key and value features, a scaling dot product attention calculation is performed to obtain the basic fusion feature; Using the visual feature sequence as the query feature, and the counterfactual language features as the key and value features, a scaling dot product attention calculation is performed to obtain the counterfactual fusion feature.
5. The remote sensing image referencing segmentation method according to claim 1, characterized in that, Determine the difference features between the basic fusion features and the counterfactual fusion features, and fuse the difference features with the basic fusion features to obtain enhanced alignment features, including: The difference features are obtained by subtracting the basic fusion features and the counterfactual fusion features element by element. The basic fusion features are input into the gated network to calculate the weight distribution corresponding to the differential features; The weight distribution is multiplied element-wise with the difference feature, and the result of the multiplication is residually connected with the basic fusion feature to obtain the enhanced alignment feature.
6. The remote sensing image referencing segmentation method according to claim 5, characterized in that, The gating network is used for: The basic fusion features are linearly transformed using a two-layer linear mapping network. The weight distribution is obtained by performing nonlinear activation calculations on the features after linear transformation using a linear rectified function and a hyperbolic tangent function.
7. The remote sensing image referencing segmentation method according to claim 1, characterized in that, Also includes: Establish a training set containing sample remote sensing images, sample original referential text, and sample counterfactual text; The training set is used to train network parameters that map the sample remote sensing images, the sample original referential text, and the sample counterfactual text to the sample referential segmentation results.
8. The remote sensing image referencing segmentation method according to claim 7, characterized in that, Training network parameters using the training set to map the sample remote sensing images, the sample original referential text, and the sample counterfactual text to the sample referential segmentation results includes: Sample a uniformly distributed subset from the training set; Freeze the first network parameters that map the sample remote sensing image, the sample original referential text, and the sample counterfactual text to the sample enhancement alignment features, and use the subset to train the second network parameters that map the sample enhancement alignment features to the sample referential segmentation results; Unfreeze the first network parameters, and use the training set to jointly train the first network parameters and the second network parameters.
9. The remote sensing image referencing segmentation method according to claim 7, characterized in that, After establishing a training set containing sample remote sensing images, sample original referential text, and sample counterfactual text, before training the network parameters that map the sample remote sensing images, the sample original referential text, and the sample counterfactual text to the sample referential segmentation results using the training set, the following steps are also included: Construct corresponding counterfactual mask labels for the sample counterfactual text, and add the sample counterfactual text and the corresponding counterfactual mask labels as augmented samples to the training set.
10. The remote sensing image referencing segmentation method according to claim 9, characterized in that, Constructing corresponding counterfactual mask labels for the sample counterfactual text includes: Extract the binary tensor annotation data corresponding to the original referential text of the sample; The pixel values representing the foreground and the pixel values representing the background in the binary tensor labeled data are swapped and inverted to obtain the counterfactual mask label.