A reference image segmentation method based on cross-modal mirror alignment and double contrast learning

Abstract

The technical scheme belongs to the field of image segmentation, and particularly relates to a kind of referential image segmentation method based on cross-modal mirror alignment and double contrast learning, and the cross-modal mirror alignment fusion module is established, the deep bidirectional semantic interaction and context information injection between visual and language features are realized using bidirectional symmetric attention mechanism, language perception visual feature and visual perception language feature are generated, the importance of feature in semantic and spatial dimension is adaptively evaluated and recalibrated through the parallel channel and spatial attention branch of space-channel gate module, noise is inhibited and key information is enhanced, high-quality enhanced cross-modal feature is output through residual connection, and fusion efficiency is optimized;At the same time, instance level and clustering level contrast are carried out, and double contrast learning mechanism with entropy regularization is supplemented, the model jointly learns more discriminative instance features and more generalizable class prototype distribution, improves the feature discrimination ability of the model in distinguishing highly similar similar interference, and improves the precision.

Description

Technical Field

[0001] This technical solution relates to the field of denotative image segmentation, specifically a denotative image segmentation method based on cross-modal mirror alignment and dual contrast learning. Background Technology

[0002] Image segmentation is a visual language task that aims to accurately segment specific target objects from images based on natural language descriptions. It requires comprehensive extraction of visual information from images, understanding of image scene distribution, in-depth mastery of semantic information in natural language, and accurate cross-modal alignment of the two.

[0003] While existing methods have evolved from fusing independently encoded visual and linguistic features at the decoder stage to extending cross-modal interaction to the encoder stage, some methods only employ a one-way interaction mechanism from language to vision, ignoring the feedback from the visual modality to language understanding; while some bidirectional interaction models often adopt parallel path designs that lack sufficient information flow, or only utilize high-level linguistic features to interact with the visual encoder in stages, ignoring the alignment opportunities between low-level features.

[0004] When faced with complex scenes containing multiple highly similar instances, denotative image segmentation models need to accurately distinguish between the specified target object and similar interference items. Existing work introduces instance-level contrastive learning to enhance local discriminative power, but the strategy of relying solely on negative sample constraints between instances is still insufficient and often fails to meet the robustness requirements of the segmentation model, resulting in a decrease in the segmentation performance of the model under visual similarity interference items.

[0005] In summary, existing methods for segmenting referential images still have the following technical shortcomings: 1. Existing cross-modal interaction mechanisms are often suboptimal, leading to inaccurate visual-language alignment; 2. The lack of an effective gating mechanism for the importance of spatial and channel dimensions leads to low efficiency in multimodal feature fusion; 3. When dealing with scenarios with highly similar interference items, the feature discrimination ability is insufficient, making it difficult to effectively distinguish similar objects through simple instance comparison learning. Summary of the Invention

[0006] This technical solution addresses the shortcomings of existing methods for representing images by providing a method for representing images based on cross-modal mirror alignment and dual contrast learning.

[0007] The purpose of this technical solution is achieved as follows: A method for segmenting referential images based on cross-modal mirror alignment and dual contrast learning, the steps of which include: S1. Obtain the image to be processed. and the corresponding referential text description ; S2, Image Input visual feature extraction model to obtain visual feature representation , to describe the referential text Input text feature extraction model to obtain language feature representation ; S3. Establish a cross-modal mirror alignment and fusion module and perform multiple operation steps to obtain cross-modal visual features. and cross-modal language features ; The specific steps of S3's operation are as follows: S31. Represent the visual features using a bidirectional symmetric cross-modal attention fusion module. Deep bidirectional interaction and semantic alignment are performed with the language feature representation L. A bidirectional attention mechanism is used to capture fine-grained associations between modalities, and cross-modal contextual information is injected into the original features respectively, thereby generating language-aware visual features V' and visual-aware language features L'. The specific steps of S31 are as follows: S311. Obtaining visual feature representation and language feature representation ; S312. Constructing a bidirectional attention matrix for semantic associations between visual and language. Bidirectional attention matrix Where H is the height of the visual feature, W is the width of the visual feature, and N is the length of the text; S313, Regarding the bidirectional attention matrix and bidirectional attention matrix transpose matrix Normalization is performed to obtain the attentional weights of language on vision. and visual attention weights to language ; S314, Weighting the attention of language to vision and visual attention weights to language Injected into the language feature representation respectively and visual feature representation Correspondingly, we obtain the language perception visual feature V' and the visual perception language feature L'.

[0008] S32. Input the language-aware visual features V' and visual-aware language features L' into the space-channel gating module for adaptive feature recalibration and noise suppression. Use parallel channel attention and spatial attention mechanisms to capture the importance weights of the features in the semantic channel dimension and spatial location dimension, respectively. Based on the importance weights, perform gating adjustment on the features to generate corresponding gated visual features. and gating language features ; The specific steps of S32 are as follows: S321. Use language-perceptual visual features V' or visual-perceptual language features L' as input features. Input features Input the parallel channel attention branch and spatial attention branch respectively; S322. Utilize channel attention branching to process input features. Global average pooling is used to aggregate global spatial information, and the channel-dimensional dependencies are extracted through the first convolutional module to generate channel attention variables. ; S323. Utilize the spatial attention branch to process the input features. Dimensionality reduction convolution and depthwise separable convolution operations are performed to capture local contextual information, and spatial dimensional dependencies are extracted through a second convolutional module to generate spatial attention variables. ; S324, Adjust channel attention variables and spatial attention variables The weights are added together and nonlinear activation is performed using the hyperbolic tangent function to generate space-channel joint gating weights. ; S325, Apply the space-channel joint gating weights Input features Perform element-wise multiplication to obtain the corresponding gated visual features. and gating language features .

[0009] S33, based on the gate control visual features respectively and gating language features Connected to visual feature representation via residuals and language feature representation Thus, cross-modal visual features are obtained. and cross-modal language features .

[0010] Preferably, the cross-modal visual features The calculation is as follows: ; The cross-modal language features The calculation is as follows: .

[0011] S4. Incorporating cross-modal visual features and cross-modal language features The input feature refinement decoder utilizes a sequential vision-language attention mechanism to perform multi-stage feature refinement to obtain segmentation visual features. and global language features ; The specific steps of S4 are as follows: S41. Construct a language-visual attention unit and utilize cross-modal visual features. As keys and values, cross-modal language features Perform a query update to obtain refined language features. ; S42. Construct visual-language attention units and utilize refined language features. As keys and values, cross-modal visual features Perform a query update to obtain segmentation visual features. ; S43. Extracting and refining language features Global semantic vectors in the language are used as global language features. ; S44. The mirror alignment and intra-class optimization comparison model is trained using the training set and the joint total loss is backpropagated to optimize the model parameters of the mirror alignment and intra-class optimization comparison model. The combined total loss of S44 is calculated as follows: S441. Segmenting visual semantic features Input the prediction header to obtain the predicted segmentation mask corresponding to the target object. The prediction head consists of a 1×1 convolutional layer and a sigmoid activation function, predicting the segmentation mask. The acquisition process is as follows: Using 1×1 convolutional layers to segment visual semantic features The channel dimension is mapped to 1. The bilinear interpolation algorithm is used to upsample the feature map to the same spatial resolution as the original input image, and the Sigmoid activation function is used to map the feature values ​​to the probability interval [0, 1]. S442. Based on the predicted segmentation mask and the real mask provided by the training set The segmentation loss is calculated using the binary cross-entropy loss function. ; S443. Define the instance-level positive sample set based on the batch index and the cluster index. Instance-level negative sample set Cluster-level positive sample set and cluster-level negative sample set , For instance-level features, the instance-level positive sample set The definition of , The instance-level negative sample set The definition of , For cluster-level features, the cluster-level positive sample set The definition of , The cluster-level negative sample set The definition of ; S444. Based on the instance feature vectors mapped to the instance comparison space, construct an instance-level similarity matrix and calculate the instance comparison loss using a logarithmic loss function. ; S445. Based on the cluster distribution probabilities mapped to the cluster comparison space, construct a cluster-level similarity matrix and calculate the cluster-level loss by combining it with the entropy regularization term. And calculate the entropy regularization term. The sum of the two is used to obtain the clustering level loss. ; S446, Based on Segmentation Loss Instance comparison loss and clustering level loss Construct a joint total loss And calculate backpropagation to optimize model parameters; Define instance comparison loss and clustering level loss Total loss : ; and total joint losses : .

[0012] S5. Segmenting visual features Corresponding output segmentation visual semantic features The predicted segmentation mask of the target object is obtained through a prediction layer and an upsampling operation, which is used to output the segmentation result of the reference image.

[0013] The key and beneficial technical effects of this technical solution compared to existing technologies are: 1. This technical solution, through the proposed cross-modal mirror alignment fusion module, achieves deep and equitable semantic interaction and contextual information injection between visual and linguistic features via a bidirectional symmetric attention mechanism. This ensures the generation of "linguistic-perceptual visual features" and "visual-perceptual linguistic features," improving the accuracy and completeness of semantic alignment between modalities. The "space-channel" gating module adaptively evaluates and recalibrates the importance of features in semantic and spatial dimensions through parallel channel and spatial attention branches, suppressing noise and enhancing key information. Finally, residual connections produce high-quality enhanced cross-modal features, thereby optimizing the efficiency and robustness of the fusion process.

[0014] 2. This technical solution employs a dual contrastive learning mechanism, which simultaneously performs instance-level and cluster-level contrastive comparisons, supplemented by entropy regularization. This enables the model to jointly learn more discriminative instance features and more generalizable category prototype distributions, thereby demonstrating better feature discrimination capabilities when faced with interference from similar objects. This comprehensively improves the accuracy and robustness of the referential image segmentation task in complex scenarios, reducing the limitations of traditional instance contrastive learning in highly similar interference scenarios. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the image referencing segmentation method in this embodiment.

[0016] Figure 2 This is a schematic diagram of the method flow for S3 in this embodiment.

[0017] Figure 3 This is a schematic diagram of the specific process of S31 in this embodiment.

[0018] Figure 4 This is a schematic diagram of the specific process of S32 in this embodiment.

[0019] Figure 5 This is a schematic diagram of the method flow for S4 in this embodiment.

[0020] Figure 6 This is a schematic diagram of the specific process for S44 in this embodiment. Detailed Implementation

[0021] A method for segmenting referential images based on cross-modal mirror alignment and dual-contrast learning, comprising the following steps: See Figure 1 S1. Obtain the image to be processed. and corresponding referential text descriptions ; Obtain raw RGB images from a denotation image segmentation dataset or actual acquisition scenarios, and preprocess them (including uniform resolution adjustment and normalization operations) to form the image to be processed. Where H and W represent the height and width of the image, respectively, and 3 represents the number of color channels of the image.

[0022] Simultaneously, natural language statements used to uniquely refer to specific targets in the image are obtained and segmented into words to form referential text descriptions. ,in, For the nth word, N represents the sequence length of the referential text description (i.e., the total number of words); by acquiring and preprocessing paired images and text, multimodal (visual and linguistic) data preparation for the referential image segmentation task is achieved.

[0023] S2. Input image X into the visual feature extraction model to obtain visual feature representation. , to describe the referential text Input text feature extraction model to obtain language feature representation ; Using visual feature extraction models to analyze images Feature extraction is performed to obtain multi-level visual feature representations. ,in, Let M be the visual feature map extracted in the i-th stage, and M be the total number of feature extraction stages. , and Let represent the channel dimension, height, and width of the visual feature map at the i-th stage, respectively. Simultaneously, a pre-trained text feature extraction model is used to describe the referential text. Feature extraction is performed to obtain language feature representations. ,in, Let N be the feature vector of the nth word in the referential text description, where N is the length of the text sequence and D represents the dimension of the language feature representation.

[0024] Both the visual feature extraction model and the text feature extraction model are pre-trained models. The visual feature extraction model is one of the following: ResNet, DenseNet, GoogleNet, or Vision Transformer, including but not limited to the above models. This embodiment shows the implementation using the GoogleNet model. The text feature extraction model is one of the following: RoBERTa, BERT, GPT, XLNet, or a language model based on the Transformer architecture, including but not limited to the above models. This embodiment shows the implementation using the RoBERTa model.

[0025] See Figure 2 and Figure 3 S3. Establish a cross-modal mirror alignment and fusion module to perform multiple operations. The specific steps are as follows: S31. A bidirectional symmetric cross-modal attention fusion module is used to perform deep bidirectional interaction and semantic alignment on the visual feature representation V and the linguistic feature representation L. The bidirectional attention mechanism is used to capture fine-grained associations between modalities, and cross-modal contextual information is injected into the original features respectively to generate linguistic-aware visual features V' and visual-aware linguistic features L', as follows: S311. Obtain visual feature representation V and linguistic feature representation L; S312. Constructing a bidirectional attention matrix for semantic associations between visual and language. ,in, The elements in the text represent the degree of response between a specific visual region and a specific language word, where H is the height of the visual feature, W is the width of the visual feature, and N is the length of the text. Bidirectional attention matrix The definition is as follows: , in, This represents the semantic similarity or response level between the i-th visual region (corresponding to the i-th row of the flattened visual features) and the j-th language word (corresponding to the j-th column of the language features). The larger the value, the stronger the semantic association between the visual region and the word.

[0026] The The value is obtained by calculating the dot product of visual features and language features in the shared subspace. The specific calculation logic is as follows: First, the visual feature representation V is input into the query projection layer. Obtain visual query vector The language features are represented by the L input key projection layer. Obtain the language query vector Then calculate the dot product of the two and divide by the scaling factor, as shown in the following formula: , in, Represents matrix multiplication. Indicates matrix transpose. This is a scaling factor used to prevent the gradient vanishing problem caused by excessively large dot product results.

[0027] S313, Regarding the bidirectional attention matrix and its transpose matrix Normalization is performed to obtain the attentional weights of language on vision. visual attention weights to language The attentional weight of language on vision The calculation formula is as follows: , in, This means performing a normalization operation on the last dimension of the matrix (i.e., the language dimension N), so that for each visual region, the sum of its attention weights for all language words is 1. The elements in the text represent the importance of specific language words to specific visual regions.

[0028] Visual attention weight to language The calculation formula is as follows: , in, This represents the transpose of the bidirectional attention matrix. This means performing a normalization operation on the last dimension of the matrix (i.e., the visual dimension HW) so that for each language word, the sum of its attention weights across all visual regions is 1. The elements in the text represent the semantic response intensity of a specific visual region to specific language words.

[0029] S314. Assign attention weights to language on vision respectively. visual attention weights to language Injected into the linguistic feature representation L and the visual feature representation V, corresponding to visual perceptual linguistic features V' and linguistic perceptual visual features L'; Visual perception of language features The calculation formula is as follows: , in, This is the attention weight matrix for language on vision. This represents the original language features. This represents the value projection weight matrix for language features; This represents a matrix multiplication operation. This step maps contextual information from linguistic features to a visual feature space, thereby endowing visual features with the ability to perceive linguistic semantics.

[0030] Language perception visual features The calculation formula is as follows: , in, This is the attention weight matrix for visual input to language. This represents the original visual features. This represents the projection weight matrix onto the values ​​of visual features; This represents a matrix multiplication operation. This step maps the contextual information in visual features to a linguistic feature space, thereby endowing linguistic features with the perceptual ability of visual images.

[0031] See Figure 2 and Figure 4 S32. Input the language-aware visual feature V' and the visual-aware language feature L' into the space-channel gating module for adaptive feature recalibration and noise suppression. Use parallel channel attention and spatial attention mechanisms to capture the importance weights of the features in the semantic channel dimension and spatial location dimension, respectively. Based on the importance weights, perform gating adjustment on the features to generate corresponding gated visual features. and gating language features The details are as follows: S321. Use language-perceptual visual features V' or visual-perceptual language features L' as input features. Input the parallel channel attention branch and spatial attention branch respectively; S322. Utilize the channel attention branch to process the input features. Global average pooling is used to aggregate global spatial information, and the channel-dimensional dependencies are extracted through the first convolutional module to generate channel attention variables. The specific formula is as follows: , in, This represents input features (i.e., language-perceptual visual features or visual-perceptual language features). This represents the global average pooling operation, used to compress input features in a spatial dimension to aggregate global information; and The first convolutional module consists of channel compression convolutional layers and channel recovery convolutional layers, both of which are 1×1 convolutional operations used to capture the non-linear dependencies between feature channels. Represents the ReLU activation function, used to introduce nonlinear factors; This represents the final generated channel attention variable.

[0032] S323. Utilize the spatial attention branch to process the input features. Dimensionality reduction convolution and depthwise separable convolution operations are performed to capture local contextual information, and spatial dimensional dependencies are extracted through a second convolutional module to generate spatial attention variables. The specific formula is as follows: , in, Represents input features; This indicates a dimension reduction convolutional layer, which is a 1×1 convolution operation used to reduce the number of feature channels and thus reduce computational cost. This represents a depthwise convolutional layer in a depthwise separable convolutional operation, using a 3×3 convolutional kernel to capture local contextual information while maintaining channel independence; This indicates the second convolutional module, which is a 1×1 convolutional operation used to extract spatial dimensional dependencies and map features back to the original or predetermined channel dimensions. This represents the ReLU activation function, used to introduce nonlinear features; This represents the final generated spatial attention variable.

[0033] S324, Adjust channel attention variables Spatial attention variables The weights are added together and nonlinear activation is performed using the hyperbolic tangent function to generate space-channel joint gating weights. The specific formula is as follows: , in, For channel attention variables, As spatial attention variables, the addition of the two achieves cross-dimensional feature fusion; The hyperbolic tangent activation function is used to map the fused feature values ​​to the interval (-1, 1), thereby introducing nonlinearity and adjusting the amplitude of the feature response; This represents the generated "spatial-channel" joint gating weights, whose dimensions are consistent with the input features, and are used for subsequent gating modulation of the features.

[0034] S325. Combine the spatial-channel joint gating weights with the input features. Perform element-wise multiplication to obtain the corresponding gated visual features. and gating language features The specific formula is as follows: Gating features The calculation is as follows: , in, This represents the input features, corresponding to language-aware visual features V' or visual-aware language features L'. For "space-channel" joint gating weights; This represents the element-wise multiplication operation of a matrix, using gated weights to weight each element of the input feature; This represents the final gated feature generated when the input is V'. This refers to the visual characteristics of gating. When the input is L', This refers to the gating language feature. .

[0035] S33, based on the gate control visual features respectively and the gated language features By connecting the residuals to the original visual feature representation V and the linguistic feature representation L, enhanced cross-modal visual features are obtained. and enhanced cross-modal language features The details are as follows: Enhanced cross-modal visual features The calculation is as follows: , Where V represents the original visual feature representation, The + indicates gated visual features, both of which have the same dimension; the + indicates an element-wise addition operation, which uses the residual connection mechanism to inject the learned cross-modal gating information into the original visual features, thereby preventing gradient vanishing and preserving the original feature information; This represents the final enhanced cross-modal visual features obtained.

[0036] Enhanced cross-modal language features The calculation is as follows: , Where L represents the original language feature representation. Indicates gating language features; This represents the final enhanced cross-modal language feature, which incorporates visually guided and gated contextual information for subsequent feature decoding and segmentation prediction.

[0037] See Figure 5 S4, Enhance the cross-modal visual features and the enhanced cross-modal language features The input feature refinement decoder utilizes a sequential vision-language attention mechanism for multi-stage feature refinement to obtain the final segmentation visual features and global linguistic features, as detailed below: S41. Construct a language-visual attention unit, using enhanced cross-modal visual features as keys and values ​​to query and update enhanced cross-modal language features, thereby obtaining refined language features. The calculation is as follows: ; in, and These represent enhanced cross-modal language features and enhanced cross-modal visual features, respectively. This represents a cross-modal attention operation, where language features are used as query vectors and visual features are used as key-value pair vectors, aiming to inject visual contextual information into language features; Language attention features that represent aggregated visual contextual information; Representation layer normalization; This represents a feedforward neural network; This indicates refined linguistic features that incorporate visual information.

[0038] S42. Construct visual-language attention units and utilize refined language features. As the key, the enhanced cross-modal visual features are queried and updated to obtain the final segmented visual features. The calculation is as follows: , In this unit, Based on visual features For the query vector, use refined language features This is a key-value pair vector designed to accurately locate target regions in an image using updated semantic cues; Visual features that incorporate key semantic cues; This represents the final segmented visual features after bidirectional sequential refinement, used for subsequent mask prediction.

[0039] S43. Extract the global semantic vector from the refined language features as global language features. The calculation is as follows: , in, This vector represents the first classification label in the refined language feature sequence. It aggregates global semantic information after sufficient interaction with the visual context and will be combined with... They are used together for subsequent segmentation prediction and contrastive learning loss calculation.

[0040] See Figure 6 S44. The mirror alignment and intra-class optimization comparison models are trained using the training set, and the joint total loss is backpropagated to optimize the model parameters of the mirror alignment and intra-class optimization comparison models. That is, the model training is jointly optimized by combining pixel-level segmentation supervision and dual contrastive learning. The aim is to enhance the model's ability to distinguish similar interference items and improve the consistency of feature space representation. The joint total loss calculation process is as follows: S441. Segmenting visual semantic features Input the prediction header to obtain the predicted segmentation mask of the target object. The prediction head includes a 1×1 convolutional layer and a sigmoid activation function. The process of obtaining the prediction segmentation mask is as follows: A 1×1 convolutional layer is used to map the channel dimension of the segmented visual semantic features to 1. A bilinear interpolation algorithm is then used to upsample the feature map to the same spatial resolution as the original input image. Finally, the Sigmoid activation function is used to map the feature values ​​to the [0,1] probability interval. The calculation is as follows: , in, This indicates a bilinear interpolation upsampling operation. This represents a 1×1 convolution operation. This is the activation function.

[0041] S442. Based on the predicted segmentation mask and the real mask provided by the training set The segmentation loss is calculated using the binary cross-entropy loss function. The calculation is as follows: , Where H and W are the height and width of the original input image, respectively. Represents pixel coordinates, Indicates the predicted segmentation mask in coordinates The predicted probability value at that location. Indicates the actual mask at coordinates The label value at the location, when the pixel belongs to the target area. When the pixel belongs to the background area .

[0042] S443. Define the instance-level positive sample set and negative sample set, as well as the cluster-level positive sample set and negative sample set, based on the batch index and cluster index. For instance-level features, the set of instance-level positive samples is... With instance-level negative sample set They are defined as follows: , in, Represents the visual instance feature vector originating from the same input sample i. With language instance feature vectors The combination of the two should maintain semantic consistency; while The combination of visual and linguistic features originating from different input samples (i.e., different batch indices) should be distributed in the feature space according to semantic distance. Furthermore, the feature representations of each element in both the instance-level positive and negative sample sets are taken from the feature matrix mapped to the instance contrast space. and .

[0043] For cluster-level features, the set of positive samples at the cluster level With cluster-level negative sample set They are defined as follows: , in, This represents the visual clustering distribution vector corresponding to the same semantic clustering cluster index k. Language clustering distribution vector The combination of (i.e., the column vectors of the matrix); and This represents the combination of distribution vectors corresponding to different semantic clustering cluster indices. Furthermore, the feature representations of each element in both the positive and negative sample sets at the clustering level are taken from the distribution matrix mapped to the clustering comparison space. and .

[0044] S444. Based on the instance feature vectors mapped to the instance comparison space, construct an instance-level similarity matrix and calculate the instance comparison loss using a logarithmic loss function. Constructing a joint instance feature matrix Based on this, the instance-level similarity matrix s is defined as follows: , in, From instance-level visual features and instance-level language features It is vertically spliced ​​together in batch dimension. This is the temperature coefficient.

[0045] Calculate the instance contrast loss using the constructed instance-level similarity matrix. The calculation is as follows: , in, For instance comparison loss, the average log loss is calculated over all samples in the batch. This represents the similarity between positive sample pairs in the similarity matrix; 2B represents the similarity between negative sample pairs in the similarity matrix; 2B is the total number of samples after concatenation.

[0046] S445. Based on the cluster distribution probability mapped to the cluster comparison space, construct a cluster-level similarity matrix and calculate the cluster-level loss by combining the entropy regularization term. Constructing a joint clustering feature matrix Based on this, a cluster-level similarity matrix is ​​defined. : , in, Clustering probability assignment matrix after transpose and It is formed by vertically splicing along the clustering dimension, where K is the number of clusters.

[0047] based on Substitute the values ​​into the loss function formula in S444 above to calculate the cluster contrast loss. To avoid cluster collapse, an entropy regularization term is defined. : , in, and represents the average allocation probability of the visual and language modalities in the kth cluster, respectively.

[0048] Based on the calculation of cluster contrast loss and entropy regularization term, the cluster-level loss is obtained. : .

[0049] S446, Based on segmentation loss and instance contrast loss and clustering level loss Construct the joint total loss and compute backpropagation to optimize the model parameters; Define instance comparison loss Clustering level loss Total loss Combined total losses : , , in, To divide the loss, The hyperparameter for balancing the weights of the segmentation task and the contrastive learning task is set to 0.01.

[0050] The training set can be any existing referential image segmentation dataset familiar to those skilled in the art, such as RefCOCO, RefCOCO+, or G-Ref datasets.

[0051] S5. The obtained segmentation visual semantic features are passed through the prediction layer and then upsampled to obtain the predicted segmentation mask of the target object, which is the image segmentation result.

[0052] Through experimental verification, this embodiment selects three standard public benchmark datasets, RefCOCO, RefCOCO+, and G-Ref, for performance verification. These datasets are all built on the large-scale MS-COCO dataset, covering a variety of language scenarios, from concise descriptions containing absolute positions (RefCOCO) to difficult descriptions that prohibit the use of locative words (RefCOCO+), and natural language descriptions containing long texts and complex sentence structures (G-Ref). They have extremely high language modality diversity and visual scene complexity, and can comprehensively and effectively verify the effectiveness and generalization ability of the proposed method in referential image segmentation under different language guidance conditions and visual interference environments.

[0053] The comparative experiment process of this scheme is as follows: To scientifically and rigorously evaluate the performance advantages of this method, four existing referential image segmentation methods—LAVT, CARIS, ReMamber, and DETRIS—were selected and systematically compared with the SAICR method proposed in this embodiment.

[0054] The experiment uses the overall intersection-union ratio (oIoU) as the core evaluation index: oIoU is calculated by the ratio of the intersection and union between the predicted mask and the real mask, which comprehensively measures the accuracy and completeness of the method in pixel-level segmentation, and pays more attention to the pixel-level matching degree of the overall dataset, as shown in Table 1. Table 1

[0055] In summary, the experimental data in Table 1 show that the SAICR method proposed in this embodiment achieves better results on all three benchmark datasets compared to the comparison methods. Its performance is particularly outstanding on the more challenging RefCOCO+ dataset, which restricts locative words. This method outperforms the previous best method by 1.07%, 1.29%, and 1.22% in the data splits of val, testA, and testB, respectively, improving the model's understanding ability in complex contexts. Furthermore, on the G-Ref dataset containing long text descriptions, this method also achieves performance gains of 0.7% and 1.48% on the val and test sets, respectively, enabling more efficient and accurate cross-modal semantic alignment and differentiation of similar interference items.

[0056] The specific work process of this plan is as follows: This technical solution proposes a cross-modal mirror alignment fusion module. Through a bidirectional symmetric attention mechanism, it achieves deep and equivalent semantic interaction and contextual information injection between visual and linguistic features, ensuring the generation of "linguistic-perceptual visual features" and "visual-perceptual linguistic features," thus improving the accuracy and completeness of intermodal semantic alignment. A "space-channel" gating module adaptively evaluates and recalibrates the importance of features in semantic and spatial dimensions through parallel channel and spatial attention branches, suppressing noise and enhancing key information. Residual connections then produce high-quality enhanced cross-modal features, optimizing the efficiency and robustness of the fusion process. Finally, a dual contrastive learning mechanism, simultaneously performing instance-level and cluster-level contrastive comparisons and supplemented by entropy regularization, enables the model to jointly learn more discriminative instance features and more generalized category prototype distributions. This results in better feature discrimination capabilities when facing interference from similar objects, comprehensively improving the accuracy and robustness of the referential image segmentation task in complex scenes, thereby reducing the limitations of traditional instance contrastive learning in highly similar interference scenarios.

[0057] The foregoing has shown and described the basic principles, main features, and advantages of this technical solution. Those skilled in the art should understand that this technical solution is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this technical solution. Various changes and modifications can be made to this technical solution without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed technical solution. The scope of protection of this technical solution is defined by the appended claims and their equivalents.

Claims

1. A method for segmenting referential images based on cross-modal mirror alignment and dual contrast learning, characterized in that, The steps of the reference image segmentation method include: S1. Obtain the image to be processed. and the corresponding referential text description ; S2, Image Input visual feature extraction model to obtain visual feature representation , to describe the referential text Input text feature extraction model to obtain language feature representation ; S3. Establish a cross-modal mirror alignment and fusion module and perform multiple operation steps to obtain cross-modal visual features. and cross-modal language features ; S4. Incorporating cross-modal visual features and cross-modal language features The input feature refinement decoder utilizes a sequential vision-language attention mechanism to perform multi-stage feature refinement to obtain segmentation visual features. and global language features ; S5. Segmenting visual features Corresponding output segmentation visual semantic features The predicted segmentation mask of the target object is obtained through a prediction layer and an upsampling operation, which is used to output the segmentation result of the reference image.

2. The method for referential image segmentation based on cross-modal mirror alignment and dual contrast learning according to claim 1, characterized in that, The specific steps of S3's operation are as follows: S31. Represent the visual features using a bidirectional symmetric cross-modal attention fusion module. Deep bidirectional interaction and semantic alignment are performed with the language feature representation L. A bidirectional attention mechanism is used to capture fine-grained associations between modalities, and cross-modal contextual information is injected into the original features respectively, thereby generating language-aware visual features V' and visual-aware language features L'. S32. Input the language-aware visual features V' and visual-aware language features L' into the space-channel gating module for adaptive feature recalibration and noise suppression. Use parallel channel attention and spatial attention mechanisms to capture the importance weights of the features in the semantic channel dimension and spatial location dimension, respectively. Based on the importance weights, perform gating adjustment on the features to generate corresponding gated visual features. and gating language features ; S33, based on the gate control visual features respectively and gating language features Connected to visual feature representation via residuals and language feature representation Thus, cross-modal visual features are obtained. and cross-modal language features .

3. The method for segmenting referential images based on cross-modal mirror alignment and dual contrast learning according to claim 2, characterized in that, The specific steps of S31 are as follows: S311. Obtaining visual feature representation and language feature representation ; S312. Constructing a bidirectional attention matrix for semantic associations between visual and language. Bidirectional attention matrix Where H is the height of the visual feature, W is the width of the visual feature, and N is the length of the text; S313, Regarding the bidirectional attention matrix and bidirectional attention matrix transpose matrix Normalization is performed to obtain the attentional weights of language on vision. and visual attention weights to language ; S314, Weighting the attention of language to vision and visual attention weights to language Injected into the language feature representation respectively and visual feature representation Correspondingly, we obtain the language perception visual feature V' and the visual perception language feature L'.

4. The method for referential image segmentation based on cross-modal mirror alignment and dual contrast learning according to claim 2, characterized in that, The specific steps of S32 are as follows: S321. Use language-perceptual visual features V' or visual-perceptual language features L' as input features. Input features Input the parallel channel attention branch and spatial attention branch respectively; S322. Utilize channel attention branching to process input features. Global average pooling is used to aggregate global spatial information, and the channel-dimensional dependencies are extracted through the first convolutional module to generate channel attention variables. ; S323. Utilize the spatial attention branch to process the input features. Dimensionality reduction convolution and depthwise separable convolution operations are performed to capture local contextual information, and spatial dimensional dependencies are extracted through a second convolutional module to generate spatial attention variables. ; S324, Adjust channel attention variables and spatial attention variables The weights are added together and nonlinear activation is performed using the hyperbolic tangent function to generate space-channel joint gating weights. ; S325, Apply the space-channel joint gating weights Input features Perform element-wise multiplication to obtain the corresponding gated visual features. and gating language features .

5. The method for segmenting referential images based on cross-modal mirror alignment and dual contrast learning according to claim 2, characterized in that: The cross-modal visual features The calculation is as follows: ； The cross-modal language features The calculation is as follows: 。 6. The method for segmenting referential images based on cross-modal mirror alignment and dual contrast learning according to claim 1, characterized in that, The specific steps of S4 are as follows: S41. Construct a language-visual attention unit and utilize cross-modal visual features. As keys and values, cross-modal language features Perform a query update to obtain refined language features. ; S42. Construct visual-language attention units and utilize refined language features. As keys and values, cross-modal visual features Perform a query update to obtain segmentation visual features. ; S43. Extracting and refining language features Global semantic vectors in the language are used as global language features. ; S44. The mirror alignment and intra-class optimization comparison model is trained using the training set and the joint total loss is backpropagated to optimize the model parameters of the mirror alignment and intra-class optimization comparison model.

7. The method for segmenting referential images based on cross-modal mirror alignment and dual contrast learning according to claim 6, characterized in that, The combined total loss of S44 is calculated as follows: S441. Segmenting visual semantic features Input the prediction header to obtain the predicted segmentation mask corresponding to the target object. The prediction head consists of a 1×1 convolutional layer and a sigmoid activation function, predicting the segmentation mask. The acquisition process is as follows: Using 1×1 convolutional layers to segment visual semantic features The channel dimension is mapped to 1. The bilinear interpolation algorithm is used to upsample the feature map to the same spatial resolution as the original input image, and the Sigmoid activation function is used to map the feature values ​​to the probability interval [0, 1]. S442. Based on the predicted segmentation mask and the real mask provided by the training set The segmentation loss is calculated using the binary cross-entropy loss function. ; S443. Define the instance-level positive sample set based on the batch index and the cluster index. Instance-level negative sample set Cluster-level positive sample set and cluster-level negative sample set , For instance-level features, the instance-level positive sample set The definition of ， The instance-level negative sample set The definition of ， For cluster-level features, the cluster-level positive sample set The definition of ， The cluster-level negative sample set The definition of ； S444. Based on the instance feature vectors mapped to the instance comparison space, construct an instance-level similarity matrix and calculate the instance comparison loss using a logarithmic loss function. ; S445. Based on the cluster distribution probabilities mapped to the cluster comparison space, construct a cluster-level similarity matrix and calculate the cluster-level loss by combining it with the entropy regularization term. And calculate the entropy regularization term. The sum of the two is used to obtain the clustering level loss. ; S446, Based on Segmentation Loss Instance comparison loss and clustering level loss Construct a joint total loss And calculate backpropagation to optimize model parameters; Define instance comparison loss and clustering level loss Total loss : ； and total joint losses : 。

Images