Text-guided image anaphora segmentation method

Abstract

The invention discloses a text-guided image anaphora segmentation method. The image anaphora segmentation method comprises the following steps: acquiring a to-be-segmented target image; respectively adopting a visual encoder and a text encoder to generate image features and text features of the initial stage; capturing global context and local boundary information under multiple scales by adopting a multi-scale attention feature enhancement method to obtain fusion features; a dynamic weight distribution mechanism is used to adaptively adjust the output proportion between the fusion features; the predicted mask size is adjusted to be consistent with the input picture through decoder calculation and up-sampling processing; training the segmentation model by using the DiceFocal loss to obtain a predicted mask sequence; and selecting a mask sequence which is most matched with the text description from the predicted mask sequences. The phenomenon of missing detection or false detection of segmentation is avoided, and the continuity and accuracy of the target edge and the stability and robustness of segmentation are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a text-guided image referent segmentation method. Background Art

[0002] The image referent segmentation task is to accurately segment the corresponding object or region in the image, given an input image and a referent description, by analyzing and understanding the referent description. This type of visual task primarily involves three aspects: generation, understanding, and segmentation. The generation task is similar to image description generation, but requires the generated text description to be specific to a specific target in the image rather than the entire image. Understanding focuses primarily on locating the target region, that is, determining the position and size of the target in the image based on the description. Segmentation, on the other hand, is a more refined analysis based on the understanding task. It not only determines the target location but also refines the target bounding box to its outline to obtain pixel-level precise positioning. Compared with semantic segmentation and instance segmentation tasks, image referent segmentation focuses more on segmenting the specific region specified in the description rather than segmenting all regions of the same category. Research on image referent segmentation not only has significant practical value but also presents broad application prospects, and is an important component of intelligent life.

[0003] Hu et al. first proposed a method for image referent segmentation, using CNN and LSTM to extract features from the description and the entire image, respectively, and then using a fully convolutional segmentation network to output the final result. The CNN-LSTM framework is the most traditional and intuitive fusion method. This method concatenates image and text features with normalized spatial coordinates, uses a series of convolutional layers and upsampling similar to FCN-32S to obtain a predicted mask, and then calculates the loss with the ground-truth annotations. Liu et al. proposed a recurrent multimodal interaction network, using a multimodal convLSTM network to incorporate image features into the encoding of each word in the text, thereby obtaining richer multimodal features. Ye et al. proposed a cross-modal self-attention network. Through self-attention, this network model can simultaneously and adaptively focus on important regions in the image and informative keywords in the linguistic description, effectively capturing the long-term dependencies between linguistic and visual features, fully capturing global interaction information, and better understanding the semantics of the two modal data and segmenting related regions.

[0004] Although the above methods can effectively improve the performance of referential image segmentation, the following shortcomings still exist. The interactive modeling of global information and local information in the fusion features in the existing methods is difficult to form complementarity in the model, which weakens the model's perception of complex scenes. In addition, they rely on fixed weights or linearly weighted fusion features, lack the ability to model dynamic correlations between features, and are difficult to adapt to changes in feature distribution of different input samples. Traditional loss functions are easily dominated by background areas during training. When the target ratio is too small, the model finds it difficult to learn effective target features, resulting in problems of missed segmentation or increased false detection rates. Summary of the Invention

[0005] In view of the above problems, the present invention is proposed to provide a text-guided image referent segmentation method that overcomes the above problems or at least partially solves the above problems.

[0006] According to one aspect of the present invention, a text-guided image referent segmentation method is provided, the image referent segmentation method comprising:

[0007] Obtain the target image to be segmented;

[0008] Visual encoder and text encoder are used to generate image features and text features in the preliminary stage respectively;

[0009] A multi-scale attention feature enhancement method is used to capture global context and local boundary information at multiple scales to obtain fused features.

[0010] Use dynamic weight allocation mechanism to adaptively adjust the output ratio between fusion features;

[0011] The decoder is used to calculate and upsample the predicted mask to be consistent with the input image size.

[0012] Use DiceFocal loss to train the segmentation model and obtain the predicted mask sequence;

[0013] Select the mask sequence that best matches the text description among the predicted mask sequences.

[0014] Optionally, the use of a visual encoder and a text encoder to generate preliminary image features and text features specifically includes: selecting Swin Transformer as a visual feature network to extract visual features of different granularities in four stages, and selecting Bert as a text feature network to extract word vectors of the input text L.

[0015] Optionally, the multi-scale attention feature enhancement method is used to capture global context and local boundary information at multiple scales, and the fusion feature is obtained specifically including:

[0016] The visual encoding network is divided into four stages by the Swin Transformer to extract visual features of different granularities. Each stage first performs self-attention calculation on the pixels within the local window in the image;

[0017] Calculate the correlation of visual features between different windows by moving the windows;

[0018] Input a three-channel RGB image with height H and width W respectively, and divide the window into 4×4 pixel groups through the Patch Partition module, and obtain a window area of ​​size 48-dimensional feature map;

[0019] The number of channels is converted to C through linear projection of the Linear Embedding module, and then the size is obtained through the 2-layer window transformation self-attention module. C-dimensional feature map;

[0020] The window transformation self-attention extracts the smallest granularity features in the image in stage 1;

[0021] In stages 2, 3, and 4, the feature maps are downsampled through the Patch Merging module, merging four 2×2 windows into one window, reducing the number of windows and increasing the granularity of features.

[0022] The four main features S are obtained by extracting visual features of increasing granularity from the attention module through 2, 18, and 2 window transformations respectively. i (i∈{1,2,3,4}) is

[0023] In SwinTransformerBlock, the number of stacked modules is an even number, and the modules with two different structures are operated alternately. The visual features f l-1 After the LayerNorm linear normalization layer and window attention, we get Then the visual feature f is obtained through the linear normalization layer and the multi-layer perceptron l , complete the self-attention calculation of the features within the window, similar to the convolution operation using the convolution kernel in the convolutional neural network to extract local features;

[0024] Visual feature f l After the second window transformation self-attention module, the window transformation self-attention layer first splits and recombines the window, so that the features of different windows perform mutual self-attention calculations, and then the output visual feature f is obtained through the linear normalization layer and the multi-layer perceptron. l+1 ,Stacking multiple layers of window transformation self-attention module ,is intended to extract the global correlation of visual features;

[0025] For the text feature extraction process, use Bert to extract the word vector of the input text L Among them C t is the number of channels, and T is the number of words in the description.

[0026] Optionally, the multi-scale attention feature enhancement method is used to capture global context and local boundary information at multiple scales, and obtaining fusion features further includes:

[0027] A dual-branch structure is used to process input features;

[0028] In the local branch, the convolution method is used to capture the detailed information of the local area through the effective combination of shared weights and context-aware weights;

[0029] In the global branch, the attention mechanism is adopted to enhance the model's ability to capture long-range dependencies by modeling global context information;

[0030] The outputs of local features and global features are synthesized through a direct fusion method to achieve effective integration of multi-level feature information.

[0031] Optionally, the use of a dynamic weight allocation mechanism to adaptively adjust the output weights between fusion features specifically includes:

[0032] The mechanism uses a set of self-learning weight maps to adjust the fusion ratio between the two and dynamically control the flow of local features and global features;

[0033] The weight distribution mechanism adjusts the way features flow to the next stage by learning the weighted graph, preventing the transfer of local or global features from being overly constrained;

[0034] The adaptive adjustment mechanism ensures the effective fusion of local and global features and optimizes the feature flow according to the needs of specific tasks.

[0035] Optionally, the step of using DiceFocal loss to train the segmentation model and obtain the predicted mask sequence specifically includes:

[0036] Dice loss evaluates the segmentation effect by measuring the overlap between the predicted and true segmentation areas;

[0037] The focal loss adjusts the weights of easy-to-classify regions and difficult-to-classify regions, allowing the model to pay more attention to difficult-to-classify samples during training.

[0038] Optionally, obtaining the target image to be segmented specifically includes:

[0039] Select the base version of Swin Transformer and Bert as the visual encoder and language encoder;

[0040] Training is performed on three datasets: RefCOCO, RefCOCO+, and G-Ref. The batch size is set to 8, the Adam optimizer is used, the initial learning rate is set to 5×10-5, the weight decay is 0.01, and a total of 40 epochs are performed.

[0041] The present invention provides a text-guided image referent segmentation method, which includes: obtaining a target image to be segmented; using a visual encoder and a text encoder to generate preliminary image features and text features, respectively; using a multi-scale attention feature enhancement method to capture global context and local boundary information at multiple scales to obtain fused features; using a dynamic weight allocation mechanism to adaptively adjust the output ratio between fused features; using decoder calculation and upsampling processing to adjust the predicted mask size to be consistent with the input image; using DiceFocal loss to train a segmentation model to obtain a predicted mask sequence; and selecting the mask sequence from the predicted mask sequence that best matches the text description. This method avoids segmentation omissions or false detections, improves the continuity and accuracy of target edges, and improves the stability and robustness of segmentation.

[0042] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of a text-guided image reference segmentation method provided by an embodiment of the present invention;

[0045] Figure 2 The overall framework diagram of the model provided by the embodiment of the present invention;

[0046] Figure 3 Schematic diagram of a local-global collaborative attention module provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of a dynamic weight allocation mechanism provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0049] The terms "comprises" and "comprising" and any variations thereof in the description, embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0050] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0051] like Figure 1 As shown, a text-guided image referent segmentation method is provided, which includes: obtaining a target image to be segmented; using a visual encoder and a text encoder to generate preliminary image features and text features respectively; using a multi-scale attention feature enhancement method to capture global context and local boundary information at multiple scales to obtain fused features; using a dynamic weight allocation mechanism to adaptively adjust the output ratio between fused features; using decoder calculation and upsampling processing to adjust the predicted mask size to be consistent with the input image; using DiceFocal loss to train a segmentation model to obtain a predicted mask sequence; and selecting a mask sequence that best matches the text description in the predicted mask sequence.

[0052] This paper proposes a method for image referent segmentation based on text-guided dynamic feature collaboration. The method consists of four main components: First, it achieves deep interaction and effective fusion of global contextual information and local boundary details. Second, it adaptively adjusts the output weights of the fused features. Third, it effectively suppresses the influence of background on small objects and enhances the model's attention to difficult-to-classify samples.

[0053] like Figure 2The figure shows the overall framework of the model. The network model takes images and text as input. It first uses SwinTransformer and BERT to extract visual features and text features respectively. After preliminary feature fusion, features are generated and then input into the Local Global Synergistic Attention Module (LGSA) to obtain the enhanced features of the first stage. The weights between the fused features are then adjusted through the Dynamic Weight Allocation Mechanism (DW), and finally the fused semantic features are obtained. The above operation is repeated to gradually obtain the visual features and fused features of the remaining stages. Finally, the fused features at all levels are input into the lightweight decoder for processing. The decoder adjusts the predicted mask size to be consistent with the input image through calculation and upsampling operations, and finally obtains a refined segmentation mask through the joint loss function.

[0054] First, the input features are linearly transformed and fully connected to obtain the query, key, and value. While maintaining the integrity of feature information, the ability to extract fine-grained features is further improved.

[0055] In the global branch, K and V are average pooled to aggregate the global context, reducing the amount of attention matrix calculation from O(H 2 W 2 ) down to To effectively reduce the number of parameters required to calculate the attention mechanism, thereby reducing computational complexity and enhancing the global receptive field of the model, the attention process is then performed on Q, K, and V to extract global features, capturing the overall semantic information and long-range dependencies of the image. The formula is as follows.

[0056] Q x ,K x ,V x =FC(Linear(X in ))

[0057] X global =A x Pool(V x )

[0058]

[0059] In the local branch, depthwise convolution (DWconv) is used to aggregate local information, where DWconv is divided into two steps. The first is channel-by-channel convolution. Each input channel uses a convolution kernel independently to extract the local spatial features of each channel, followed by point-by-point convolution. The weights of point-by-point convolution are globally shared and used to fuse cross-channel information to combine features of different channels. Depthwise separable convolution ensures efficient extraction of local features. For context-aware enhancement of local features, after integrating the local information and shared weights of V, the combination of Q and K is further combined to generate a context-aware weight map. Specifically, DWconv is used to aggregate local features of Q and K respectively, and their element-by-element product is calculated to obtain the context-aware weight between [-1,1]. Finally, the weight is used to enhance the local features. The formula is as follows.

[0060] Q,K,V=FC(Linear(X in ))

[0061] V z =Depth(V; Θ d )⊙Point(V;Θ p )

[0062] Q i =DWconv(Q; Θ q )

[0063] Attn i =W2·(Swish(W1·(Q i ⊙K i )))

[0064] X local =Attn⊙V z

[0065] The dynamic weight allocation mechanism consists of a two-layer perceptron. The first layer extracts key information between channels or spatial locations through 1×1 convolution operations, uses activation functions to extract and activate features, and adjusts nonlinear characteristics through Gaussian distribution to make gradient propagation smoother and more stable. The second layer further refines features and generates a weight map through 1×1 convolution operations with a post-activation function. The function output range is [-1, 1] and the gradient changes smoothly. This design can provide a wider dynamic range for features to effectively capture important feature information. Finally, the weights obtained in the previous step are multiplied element-by-element to achieve feature scaling or suppression. The residual form is then used to enable the model to selectively strengthen or weaken features while retaining the original features. This design can effectively avoid interfering with the initial weights of the original fused features. The formula is shown below.

[0066]

[0067] Y i =γ i (G i )

[0068] F=S i ⊙G i ′+Y i ⊙G i

[0069] A joint loss function optimizes the segmentation boundary by measuring the degree of overlap between the predicted segmentation area and the true segmentation area. To a certain extent, it can alleviate the problem of small targets being ignored. If the overlapping area is large and the error rate is low, the model can more directly optimize the ability to capture the target during training, penalize the difference between the predicted area and the true area, and make the model better fit the object contour and shape, thereby improving the complete cutting quality of the target. If the target area is occluded or the language description is vague, the weights of easy-to-separate samples and difficult-to-separate samples are dynamically adjusted to suppress the loss of easy-to-separate samples and amplify the loss of difficult-to-separate samples, so that the model pays more attention to target areas with ambiguous or complex backgrounds. Since the number of background pixels is often much larger than the target pixels, the model's learning of small or complex targets is improved by generating larger gradients at small pixels. In multiple rounds of segmentation iterations, the attention to difficult-to-separate samples is increased. The formula is shown below.

[0070] l Combine =l D α D +l F β F

[0071] where α D and β F is a hyperparameter that adjusts the importance of the two loss functions. D and β F The ratio between them determines whether the model is more inclined to improve the global overlap or the ability to identify difficult examples. If the foreground is extremely small or severely occluded, increasing β F , impose stronger penalties on targets that are uncertain or easily confused, reducing misclassification and omission; if the model can better identify the target but the edge is not accurate, you can increase α D , improve the IoU between the predicted mask and the true target, and maintain a stable gradient even when the target is small or extremely unbalanced.

[0072] The datasets constructed in this paper include RefCOCOg, RefCOCO, and RefCOCO+, all of which were collected on the Microsoft COCO image collection. The RefCOCOg dataset was collected in non-interactive mode, while the other two datasets, RefCOCO and RefCOCO+, were collected interactively in a two-player game.

[0073] The language used in the RefCOCO and RefCOCO+ datasets tends to be more concise than that used in RefCOCOg, with an average length of 5.57 for RefCOCO, 5.85 for RefCOCO+, and 11.72 for RefCOCOg. The RefCOCOg dataset contains 85,474 Chinese reference representations for 54,822 objects in 26,711 images. Images are selected to contain two to four objects of the same category. RefCOCO contains 142,209 Chinese representations for 50,000 objects in 19,994 images, while RefCOCO+ contains 141,564 Chinese representations for 49,856 objects in 19,992 images.

[0074] The present invention provides a local-global collaborative attention module that can fully capture global contextual information and local boundary features at different scales, thereby significantly improving the model's overall understanding of complex scenes and fine-grained segmentation capabilities; at the same time, the constructed dynamic weight allocation mechanism realizes adaptive regulation of the output proportion of fusion features, further enhancing the model's attention to fine-grained targets; a joint loss function is proposed to address the segmentation challenge of difficult-to-distinguish targets, which not only effectively reduces the interference of the background on the segmentation of small targets, but also achieves precise focus on difficult samples by assigning differentiated weights to easy-to-classify and difficult-to-classify samples, thereby improving the model's robustness in small sample and data imbalance situations. After testing on the constructed dataset, the present invention verified the effectiveness of the method.

[0075] The present invention provides an image referent segmentation method based on text-guided dynamic feature collaboration, such as Figure 3 As shown in , a local-global collaborative attention module is designed to achieve deep interaction and effective fusion of global context information and local boundary details through global branches and local branches, thereby improving the model's overall understanding and fine segmentation capabilities of target areas in complex scenes; Figure 4 As shown in the figure, a dynamic weight distribution mechanism is introduced to adaptively adjust the output ratio between fused features, improve the attention of fused features and enhanced features to fine-grained targets, and avoid redundant information interference; a joint loss function is designed to optimize the segmentation performance of difficult-to-distinguish targets, effectively suppress the impact of background on small targets and enhance the model's attention to difficult-to-classify samples, focusing on difficult samples and improving the robustness of the model.

[0076] Step 1: Data Preprocessing

[0077] (1) Select the basic version of Swin Transformer and BERT as the visual encoder and language encoder;

[0078] (2) Use the Swin Transformer model pre-trained on the ImageNet-22K dataset as the visual encoder and initialize it with the official pre-trained weights;

[0079] (3) Use the BERT model as the language encoder and initialize it with the official pre-trained weights;

[0080] Step 2: Model training

[0081] (1) Use the above preprocessed data and perform model training;

[0082] (2) Using the Adam optimizer for gradient updates during training;

[0083] (3) Use the set initial learning rate to set to 5×10 -5 Optimized with weight decay 0.01;

[0084] (4) Training for 40 epochs on the RefCOCO, RefCOCO+, and G-Ref datasets;

[0085] (5) End-to-end training is performed without any segmentation-related pre-training processing.

[0086] Beneficial effects: It provides a local-global collaborative attention module, builds global branches and local branches in parallel, and effectively realizes the complementarity of feature map information through the deep interaction of global and local features, so that the model can better understand the interaction between global background and local details in complex scenes, significantly improve the segmentation accuracy of target boundary areas, and enhance the adaptability and robustness to complex scenes. It proposes a dynamic weight distribution mechanism to adjust the influence between features in real time to ensure that different features do not interfere with each other, avoid the negative impact of redundant information on model training, prevent gradient disappearance, accelerate model convergence, and improve the overall perception ability of fusion features. It proposes a joint loss function that reduces the interference of background areas on model training by assigning a higher penalty coefficient to target boundary pixels. In scenes where the target accounts for a small proportion, it avoids segmentation omissions or false detections, improves the continuity and accuracy of target edges, and improves the stability and robustness of segmentation.

[0087] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A text-guided image referent segmentation method, characterized in that: The image reference segmentation method includes: Obtain the target image to be segmented; Visual encoder and text encoder are used to generate image features and text features in the preliminary stage respectively; A multi-scale attention feature enhancement method is used to capture global context and local boundary information at multiple scales to obtain fused features. Use dynamic weight allocation mechanism to adaptively adjust the output ratio between fusion features; The decoder is used to calculate and upsample the predicted mask to be consistent with the input image size. Use DiceFocal loss to train the segmentation model and obtain the predicted mask sequence; Select the mask sequence that best matches the text description among the predicted mask sequences.

2. The text-guided image referent segmentation method according to claim 1, characterized in that: The method of using a visual encoder and a text encoder to generate image features and text features in the preliminary stage specifically includes: using SwinTransformer as a visual feature network to extract visual features of different granularities in four stages, and using Bert as a text feature network to extract word vectors of the input text L.

3. The text-guided image referent segmentation method according to claim 2, characterized in that: The multi-scale attention feature enhancement method is used to capture global context and local boundary information at multiple scales, and the fusion features obtained specifically include: The visual encoding network is divided into four stages by the Swin Transformer to extract visual features of different granularities. Each stage first performs self-attention calculation on the pixels within the local window in the image; Calculate the correlation of visual features between different windows by moving the windows; Input a three-channel RGB image with height H and width W respectively, and divide the window into 4×4 pixel groups through the Patch Partition module, and obtain a window area of ​​size 48-dimensional feature map; The number of channels is converted to C through linear projection of the Linear Embedding module, and then the size is obtained through the 2-layer window transformation self-attention module. C-dimensional feature map; The window transformation self-attention extracts the smallest granularity features in the image in stage 1; In stages 2, 3, and 4, the feature maps are downsampled through the Patch Merging module, merging four 2×2 windows into one window, reducing the number of windows and increasing the granularity of features. The four main features S are obtained by extracting visual features of increasing granularity from the attention module through 2, 18, and 2 window transformations respectively. i (i∈{1,2,3,4}) is In SwinTransformerBlock, the number of stacked modules is an even number, and the modules with two different structures are operated alternately. The visual features f l-1 After the LayerNorm linear normalization layer and window attention, we get Then the visual feature f is obtained through the linear normalization layer and the multi-layer perceptron l , complete the self-attention calculation of the features within the window, similar to the convolution operation using the convolution kernel in the convolutional neural network to extract local features; Visual feature f l After the second window transformation self-attention module, the window transformation self-attention layer first splits and recombines the window, so that the features of different windows perform mutual self-attention calculations, and then the output visual feature f is obtained through the linear normalization layer and the multi-layer perceptron. l+1 ,Stacking multiple layers of window transformation self-attention module ,is intended to extract the global correlation of visual features; For the text feature extraction process, use Bert to extract the word vector of the input text L Among them C t is the number of channels, and T is the number of words in the description.

4. The text-guided image referent segmentation method according to claim 1, characterized in that: The multi-scale attention feature enhancement method is used to capture global context and local boundary information at multiple scales, and the fusion feature is obtained further including: A dual-branch structure is used to process input features; In the local branch, the convolution method is used to capture the detailed information of the local area through the effective combination of shared weights and context-aware weights; In the global branch, the attention mechanism is adopted to enhance the model's ability to capture long-range dependencies by modeling global context information; The outputs of local features and global features are synthesized through a direct fusion method to achieve effective integration of multi-level feature information.

5. The text-guided image referent segmentation method according to claim 1, characterized in that: The use of a dynamic weight allocation mechanism to adaptively adjust the output weights between fusion features specifically includes: The mechanism uses a set of self-learning weight maps to adjust the fusion ratio between the two and dynamically control the flow of local features and global features; The weight distribution mechanism adjusts the way features flow to the next stage by learning the weighted graph, preventing the transfer of local or global features from being overly constrained; The adaptive adjustment mechanism ensures the effective fusion of local and global features and optimizes the feature flow according to the needs of specific tasks.

6. The text-guided image referent segmentation method according to claim 1, characterized in that: The method of using DiceFocal loss to train the segmentation model and obtain the predicted mask sequence specifically includes: Dice loss evaluates the segmentation effect by measuring the overlap between the predicted and true segmentation areas; The focal loss adjusts the weights of easy-to-classify regions and difficult-to-classify regions, allowing the model to pay more attention to difficult-to-classify samples during training.

7. The text-guided image referent segmentation method according to claim 1, characterized in that: The obtaining of the target image to be segmented specifically includes: Select the base version of Swin Transformer and Bert as the visual encoder and language encoder; Training is performed on three datasets: RefCOCO, RefCOCO+, and G-Ref. The batch size is set to 8, the Adam optimizer is used, the initial learning rate is set to 5×10-5, the weight decay is 0.01, and a total of 40 epochs are performed.

Images