Multi-modal language model assisted category guide image segmentation method and system

Abstract

The invention discloses a multi-modal language model-assisted category guide image segmentation method and system, and relates to the technical field of image processing, and the method comprises the steps: respectively extracting the original image features and user interaction click features of a to-be-segmented image through an image encoder and a prompt encoder; coding the true value mask by using a convolutional network, splicing the true value mask with original image features, and then carrying out cross attention fusion to generate mask image features; extracting original text features through a text encoder, and aligning the original text features with mask image features based on comparative learning; and inputting the aligned text features, original image features and interactive click features into a mask decoder, and performing cross attention operation and normalized focus loss calculation to obtain segmentation loss and optimize an overall training target. According to the method, through normalization of focus loss and cross entropy loss functions, accurate alignment of a region-level visual mode and a text mode is ensured, and the accuracy and comprehensiveness in an image segmentation task are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and specifically relates to a category-guided image segmentation method and system assisted by a multimodal language model. Background Art

[0002] Interactive image segmentation (IIS) aims to segment objects in an image using interactive cues provided by the user. IIS has important application values in many fields such as image editing, autonomous driving, and medical image processing. Interactive segmentation methods have significantly promoted the expansion of image segmentation datasets.

[0003] In recent years, the SAM model has attracted much attention for its ability to achieve high-quality interactive segmentation in real time, which not only helps with segmentation annotation but also widely supports various vision tasks. Although SAM demonstrates powerful interactive segmentation capabilities in natural images, there is still room for further improvement in its performance on multiple interactive segmentation datasets.

[0004] However, interactive segmentation faces ambiguities during the interaction process. For example, when the annotator intends to segment a person's backpack, the model may interpret it as a cue to segment the entire person, and vice versa. Such ambiguities are common in various interaction methods such as clicks, bounding boxes, and scribbles, which increases the complexity of the interaction and reduces the segmentation accuracy. Therefore, a click-based segmentation method focusing on the mainstream is needed to resolve the ambiguities and improve the performance of the IIS task. Summary of the Invention

[0005] To solve the above problems, the present invention proposes a category-guided image segmentation method and system assisted by a multimodal language model, which realizes the efficient alignment of image and text features and significantly improves the user interaction efficiency during the interactive image segmentation process by combining ground truth mask encoding, self-attention mechanism, contrastive learning training, and category label preprocessing strategy.

[0006] The specific solutions are as follows:

[0007] On the one hand, a category-guided image segmentation method assisted by a multimodal language model includes:

[0008] S1, extracting the original image features of the image to be segmented through an image encoder, and extracting the interactive click features input by the user through a prompt encoder;

[0009] S2, encoding the ground truth mask corresponding to the original image of the image to be segmented through a convolutional network to obtain mask features, splicing the mask features with the original image features of the image to be segmented, and performing cross-attention fusion operations to obtain mask image features;

[0010] S3. Extract the original text features of the image to be segmented through a text encoder, and based on contrastive learning training, align the original text features in the image to be segmented with the mask image features to obtain aligned text features;

[0011] S4. Perform cross-attention operation on the aligned text features and the original image features to obtain image features guided by the aligned text features. Input the image features guided by the aligned text features, the interactive click features, and the aligned text features into a mask decoder for normalized focal loss calculation to obtain a segmentation loss. Calculate the overall training objective based on the segmentation loss, and guide the model to segment specific objects through the overall training objective.

[0012] Further, the S2 specifically includes:

[0013] Align the ground truth mask M with the original image of the corresponding image to be segmented through bilinear interpolation to obtain the aligned ground truth mask M;

[0014] Encode the aligned ground truth mask M through a convolutional network by a mask encoder to obtain mask features F M , and the calculation formula is as follows:

[0015] F M = MaskEncoder(M);

[0016] where MaskEncoder() represents the operation of encoding the ground truth mask into mask features F through convolutional encoding by the mask encoder M , F M ∈R B×c×H×W , B represents the number of original images of the images to be segmented in a batch, C represents the feature dimension, H represents the height of the original image of the image to be segmented, and W represents the width of the original image of the image to be segmented;

[0017] Mix the obtained mask features F M with the original image features of the image to be segmented;

[0018] The specific steps of the mixing are as follows: First, concatenate the mask features F M with the original image features F I of the image to be segmented along the channel dimension to obtain mixed features F IM ∈R B×2C×H×W ; Convert the shape of F IM to the shape of R B×HW×2C , and concatenate the converted F IM with the K-dimensional learnable tensor Q M ∈1R B×K×2C used to extract mask image features to obtain the concatenated mask features F M concat ∈RB×(HW+K)×2C Perform cross-attention fusion on the spliced mask feature F M concat to obtain the mask image feature.

[0019] Furthermore, the step S3 specifically includes:

[0020] Encode the category text T through a text encoder, and the calculation formula is as follows:

[0021] F T = TextEncoder(T);

[0022] where F T is the original text feature of the image to be segmented extracted by encoding; TextEncoder() represents encoding the text T to extract the original text feature of the image to be segmented;

[0023] Calculate the similarity S T between F M and the learnable tensor Q tm extracted with the mask image feature mt :

[0024] S tm = F T · Q M ;

[0025] S mt = Q M · F T ;

[0026] where S tm represents the similarity matrix between the original text feature of the image to be segmented and the mask image feature, and S mt represents the similarity matrix between the mask image feature and the original text feature of the image to be segmented;

[0027] Align the original text feature of the image to be segmented with the mask image feature, and calculate the mask-text contrast loss of the mask image feature and the original text feature of the image to be segmented under contrastive learning training The calculation formula is as follows:

[0028]

[0029] where represents the cross-entropy loss function, and T tm represents the label where the mask-text match is correct;

[0030] Calculate the mask-text matching loss:

[0031]

[0032] Among them, Q c represents the correctly matched prediction result, and T c represents the correctly matched label. represents the mask-text matching loss; based on and the original text features of the image to be segmented are aligned with the mask image features.

[0033] Furthermore, in S4, the image features guided by the aligned text features, the interactive click features, and the aligned text features are input into the mask decoder to calculate the normalized focal loss, and the segmentation loss is obtained as follows:

[0034] F I = VisualEncoder(I);

[0035] F P = PromptEncoder(P);

[0036] Y = MaskDecoder(F I , F P );

[0037] Among them, P represents the received interactive click input; I represents the original image to be segmented; VisualEncoder() represents the image encoder; F I represents the original image features of the image to be segmented extracted by the image encoder; PromptEncoder() represents the prompt encoder; F P represents the interactive click features extracted by the prompt encoder; MaskDecoder() represents the prediction mask generation by the mask decoder; Y represents the prediction mask generated based on F I and F P ;

[0038] The formula for the normalized focal loss is as follows:

[0039]

[0040] Among them, represents the obtained segmentation loss, Y represents the prediction mask, represents the normalized focal loss calculation function, and M represents the ground truth mask.

[0041] Furthermore, the overall training objective, the calculation formula is as follows:

[0042]

[0043] Among them, represents the overall training objective; Represents the segmentation loss; w mtc Represents the learnable weight for balancing the mask-text contrast loss; Represents the mask-text contrast loss; w mtm Represents the learnable weight for balancing the mask-text matching loss; Represents the mask-text matching loss; w mc Represents the learnable weight for balancing the class matching loss; Represents the class matching loss; w mtc 、w mtm and w mc are used to balance the mask-text contrast loss, mask-text matching loss, and class matching loss.

[0044] On the other hand, the multi-modal language model-assisted class-guided image segmentation system includes:

[0045] A feature extraction module for extracting the original image features of the image to be segmented through an image encoder and extracting the interactive click features of the user input through a prompt encoder;

[0046] A mask image feature acquisition module for encoding the ground truth mask corresponding to the original image of the image to be segmented through a convolutional network to obtain mask features, splicing the mask features with the original image features of the image to be segmented and performing cross-attention fusion operations to obtain mask image features;

[0047] A feature alignment module for extracting the original text features of the image to be segmented through a text encoder and aligning the original text features in the image to be segmented with the mask image features based on contrastive learning training to obtain aligned text features;

[0048] An overall training objective calculation module for performing cross-attention operations on the aligned text features and the original image features to obtain image features guided by the aligned text features, inputting the image features guided by the aligned text features, interactive click features, and aligned text features into a mask decoder for normalized focal loss calculation to obtain a segmentation loss, calculating the overall training objective based on the segmentation loss, and guiding the model to segment specific objects through the overall training objective.

[0049] The present invention adopts the above technical solutions and has beneficial effects:

[0050] (1) The present invention realizes the alignment of mask-image features by encoding the ground truth mask into mask features and performing self-attention operations after splicing with the image embedding features, and further enhances the segmentation performance of the model through the contrastive learning training of text features and mask-image features, effectively improving the alignment accuracy between the regional-level visual modality and text modality in interactive image segmentation, thereby improving the segmentation accuracy of the model when processing specific objects;

[0051] (2) The present invention encodes the category text using the CLIP text encoder of the freezing parameters to obtain text features, and then combines the text features with the image features through a cross-attention mechanism to generate image features guided by the text features, significantly enhancing the model's perception ability of specific object categories, reducing the ambiguity that may occur during the interaction process, and improving the overall segmentation effect.

[0052] (3) The present invention introduces a category label preprocessing strategy. By pre-generating all possible category labels in the image, the originally cumbersome keyboard input operation is simplified to a single click operation for selecting category text, integrating text prompts into the interactive segmentation field at a low cost, enabling users to complete the target segmentation task with higher efficiency and lower interaction cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flowchart of a multi-modal language model assisted category-guided image segmentation method according to an embodiment of the present invention;

[0054] Figure 2 is a schematic diagram of the process for obtaining a prediction mask according to an embodiment of the present invention;

[0055] Figure 3 is a system diagram of a multi-modal language model assisted category-guided image segmentation according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The present invention will be further described in detail below in conjunction with embodiments and the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0057] As Figure 1 shown, the multi-modal language model assisted category-guided image segmentation method of the present invention includes:

[0058] S1, extracting the original image features of the image to be segmented through an image encoder, and extracting the interactive click features input by the user through a prompt encoder.

[0059] Specifically, the original text of the image to be segmented is pre-generated by a multi-modal large language model. The multi-modal language model pre-generates the category text of all possible objects in the image to be segmented, and the user interactively selects the category of the object to be segmented currently as the original text, which is used as the input of the text encoder to extract the original text features.

[0060] S2, encoding the ground truth mask corresponding to the original image of the image to be segmented through a convolutional network to obtain mask features, splicing the mask features with the original image features of the image to be segmented and performing a cross-attention fusion operation to obtain mask image features.

[0061] Specifically, S2 specifically includes:

[0062] Align the ground truth mask M with the original image of the corresponding image to be segmented through bilinear interpolation to obtain the aligned ground truth mask M;

[0063] Encode the aligned ground truth mask M through a convolutional network of the mask encoder to obtain the mask feature F M , and the calculation formula is as follows:

[0064] F M = MaskEncoder(M);

[0065] where MaskEncoder() represents the operation of convolutional encoding through the mask encoder to encode the ground truth mask into the mask feature F M , F M ∈R B×C×H×W , B represents the number of original images of the images to be segmented in a batch, C represents the feature dimension, H represents the height of the original image of the image to be segmented, and W represents the width of the original image of the image to be segmented;

[0066] Mix the obtained mask feature F M with the original image feature of the image to be segmented;

[0067] The specific steps of the mixing are as follows: First, concatenate the mask feature F M with the original image feature F I of the image to be segmented along the channel dimension to obtain the mixed feature F IM ∈R B×2C×H×W ; Convert the shape of F IM to the shape of R B×HW×2C , and concatenate the converted F IM with the K-dimensional learnable tensor Q M ∈1R B×K×2C used to extract the mask image feature to obtain the concatenated mask feature F M concat ∈R B×(HW+K)×2C , and perform cross-attention fusion on the concatenated mask feature F M concat to obtain the mask image feature.

[0068] S3, extract the original text feature of the image to be segmented through the text encoder, and align the original text feature in the image to be segmented with the mask image feature based on contrastive learning training to obtain the aligned text feature.

[0069] Specifically, S3 specifically includes:

[0070] Encode the category text T through a text encoder, and the calculation formula is as follows:

[0071] F T = TextEncoder(T);

[0072] Among them, F T is the original text feature of the image to be segmented obtained by encoding extraction; TextEncoder() represents encoding the text T to extract the original text feature of the image to be segmented;

[0073] Calculate the similarity S T between F M and the learnable tensor Q tm extracting the masked image feature mt :

[0074] S tm = F T ·Q M ;

[0075] S mt = Q M ·F T ;

[0076] Among them, S tm represents the similarity matrix between the original text feature of the image to be segmented and the masked image feature, and S mt represents the similarity matrix between the masked image feature and the original text feature of the image to be segmented;

[0077] Align the original text feature of the image to be segmented with the masked image feature, and calculate the mask-text contrast loss of the masked image feature and the original text feature of the image to be segmented under contrastive learning training The calculation formula is as follows:

[0078]

[0079] Among them, represents the cross-entropy loss function, and T tm represents the correct label of mask-text matching;

[0080] Calculate the mask-text matching loss:

[0081]

[0082] Among them, Q c represents the correct prediction result of matching, and T c represents the correct label of matching, represents the mask-text matching loss; Based on and Align the original text features of the image to be segmented with the mask image features.

[0083] S4. Perform cross-attention operation on the aligned text features and the original image features to obtain image features guided by the aligned text features. Input the image features guided by the aligned text features, the interactive click features, and the aligned text features into the mask decoder for normalized focal loss calculation to obtain the segmentation loss. Calculate the overall training objective based on the segmentation loss, and guide the model to segment specific objects through the overall training objective.

[0084] Specifically, input the image features guided by the aligned text features, the interactive click features, and the aligned text features into the mask decoder for normalized focal loss calculation to obtain the segmentation loss, as follows:

[0085] F I = VisualEncoder(I);

[0086] F P = PromptEncoder(P);

[0087] Y = MaskDecoder(F I , F P );

[0088] where P represents the received interactive click input; I represents the original image to be segmented; VisualEncoder() represents the image encoder; F I represents the original image features of the image to be segmented extracted by the image encoder; PromptEncoder() represents the prompt encoder; F P represents the interactive click features extracted by the prompt encoder; MaskDecoder() represents the prediction mask generation through the mask decoder; Y represents the prediction mask generated based on F I and F P ;

[0089] The formula for the normalized focal loss is as follows:

[0090]

[0091] where represents the obtained segmentation loss, Y represents the prediction mask, represents the normalized focal loss calculation function, and M represents the ground truth mask.

[0092] Specifically, the formula for the overall training objective is as follows:

[0093]

[0094] Among them, represents the overall training objective; represents the segmentation loss; w mtc represents the learnable weight of the balanced mask-text contrast loss; represents the mask-text contrast loss; w mtm represents the learnable weight of the balanced mask-text matching loss; represents the mask-text matching loss; w mc represents the learnable weight of the balanced class matching loss; represents the class matching loss; w mtc 、w mtm and w mc are used to balance the mask-text contrast loss, the mask-text matching loss, and the class matching loss.

[0095] Specifically, in this embodiment, the cross-attention operation is performed on the aligned text features and the original image features, and the calculation formula is as follows:

[0096]

[0097] Y = MaskDecoder(F I CTI , F P );

[0098]

[0099] Among them, Softmax() represents the normalization function; F I CTI represents the image features guided by the text features; represents the class matching loss; Q c represents the correct prediction result of the match; T c represents the true class matching label; FFN() represents the feed-forward network; MLP() represents the multi-layer perceptron; F T represents the text features extracted by the encoding; MaskDecoder() represents the mask decoder; C represents the feature dimension; represents the class matching loss.

[0100] Specifically, as Figure 2 shown, the image extracts the categories of each object in the image through the multi-modal large model. The image inputs the image encoder to extract the image features, and the mask features extracted from the ground truth mask by the mask encoder perform cross-attention. The mask features and the text features of the mixed image features are aligned, and the image features and the text features perform cross-attention again to obtain the text-guided image features and the interactive click features extracted by the prompt encoder, and input them into the mask decoder to obtain the predicted mask.

[0101] Specifically, the present invention is developed on the Ubuntu platform, and the developed deep learning framework is based on Pytorch. The main language used in the present invention is Python. The experimental settings of this embodiment are as follows: a. Use the AdamW optimizer for network training, and the initial learning rate is 1×10 4 ; b. The size of the input image is uniformly adjusted to 640×640 pixels to adapt to the ViT-B model; c. When training, the batch size is 32, and the training period is 80000 iterations. Take the COCO-LVIS dataset, train the network according to the above steps and experimental settings, and use the test set for testing. In order to verify the effectiveness of the method, two metrics are adopted: the mean Intersection over Union (mIoU) and NoC@X (Number of Clicks@X); mIoU calculates the average of the ratio of the intersection area to the union area between the predicted segmentation mask and the true annotation; NoC represents the number of clicks required to reach a certain IoU threshold X. For example, NoC@90 refers to the number of clicks required to reach an IoU threshold of 0.90. The present invention mainly uses NoC90 and NoC95 to measure the model.

[0102] Table 1 Comparison table with the existing state-of-the-art methods;

[0103]

[0104] Table 1 shows the test results of the present invention on the test dataset. The present invention uses DAVIS, HQSeg44K, PartImageNet, PascalPart, and MGSeg as the test sets for testing. It can be found that the present invention has the best effect on the DAVIS, HQSeg44K, PartImageNet, PascalPart, and MGSeg datasets compared with other methods.

[0105] Generally speaking, the present invention introduces a class label preprocessing strategy to pre-generate all possible class labels in the image, simplifies the originally cumbersome keyboard input operation into a single click operation to select class text, thereby integrating text prompts into the interactive segmentation field at a relatively low cost.

[0106] As Figure 3 shown, this embodiment also discloses a class-guided image segmentation system diagram assisted by a multimodal language model, including:

[0107] A feature extraction module 31, configured to extract the original image features of the image to be segmented through an image encoder, and extract the interactive click features of the user input through a prompt encoder;

[0108] The mask image feature acquisition module 32 is configured to encode the ground truth mask corresponding to the original image of the image to be segmented through a convolutional network to obtain mask features, splice the mask features with the original image features of the image to be segmented, and perform cross-attention fusion operations to obtain mask image features;

[0109] The feature alignment module 33 is configured to extract the original text features of the image to be segmented through a text encoder, and align the original text features in the image to be segmented with the mask image features based on contrastive learning training to obtain aligned text features;

[0110] The overall training objective calculation module 34 is configured to perform cross-attention operations on the aligned text features and the original image features to obtain image features guided by the aligned text features, input the image features guided by the aligned text features, the interactive click features, and the aligned text features into a mask decoder to calculate the normalized focal loss, obtain the segmentation loss, calculate the overall training objective based on the segmentation loss, and guide the model to segment specific objects through the overall training objective.

[0111] The specific implementation of the multi-modal language model-assisted category-guided image segmentation system is the same as that of the multi-modal language model-assisted category-guided image segmentation method, and will not be repeated in this embodiment.

[0112] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, those skilled in the art should understand that various changes can be made to the present invention in terms of form and details without departing from the spirit and scope of the present invention defined by the appended claims, and all such changes are within the protection scope of the present invention.

Claims

1. A category-guided image segmentation method assisted by a multi-modal language model, characterized in that, Including: S1. Extract the original image features of the image to be segmented through an image encoder, and extract the interactive click features of the user input through a prompt encoder; S2. Encode the ground truth mask corresponding to the original image of the image to be segmented through a convolutional network to obtain mask features, splice the mask features with the original image features of the image to be segmented, and perform cross-attention fusion operations to obtain masked image features; S3. Extract the original text features of the image to be segmented through a text encoder, and align the original text features in the image to be segmented with the masked image features based on contrastive learning training to obtain aligned text features; S4. Perform cross-attention operations on the aligned text features and the original image features to obtain image features guided by the aligned text features, input the image features guided by the aligned text features, the interactive click features, and the aligned text features into a mask decoder for normalized focal loss calculation to obtain a segmentation loss, calculate the overall training objective based on the segmentation loss, and guide the model to segment specific objects through the overall training objective.

2. The category-guided image segmentation method assisted by a multi-modal language model according to claim 1, wherein The S2 specifically includes: Align the ground truth mask M with the original image of the corresponding image to be segmented through bilinear interpolation to obtain the aligned ground truth mask M; The aligned ground truth mask M is encoded through a convolutional network by a mask encoder to obtain a mask feature F M , and the calculation formula is as follows: F M = MaskEncoder(M); Among them, MaskEncoder() represents the convolutional encoding operation through the mask encoder to encode the ground truth mask into the mask feature F M , F M ∈R B×C×H×W , B represents the number of original images of the images to be segmented in a batch, C represents the feature dimension, H represents the height of the original image of the images to be segmented, and W represents the width of the original image of the images to be segmented; Mix the obtained masked feature F M with the original image feature of the image to be segmented; The specific steps of the mixing are as follows: First, along the channel dimension, the masked feature F M is concatenated with the original image feature F I of the image to be segmented to obtain the mixed feature F IM ∈R B×2C×H×W ; the shape of F IM is converted to the shape of R B×HW×2C , and the converted F IM is concatenated with the K-dimensional learnable tensor Q M ∈1R B×K×2C for extracting the masked image feature to obtain the concatenated masked feature F M concat ∈R B×(HW+K)×2C , and the concatenated masked feature F M concat is subjected to cross-attention fusion to obtain the masked image feature.

3. The category-guided image segmentation method assisted by a multi-modal language model according to claim 1, wherein The S3 specifically includes: Encode the category text T through a text encoder, and the calculation formula is as follows: F T = TextEncoder(T); Among them, F T is the original text feature of the image to be segmented obtained by encoding extraction; TextEncoder() represents encoding the text T to extract the original text feature of the image to be segmented; Calculate F T and the learnable tensor Q that extracts the masked image features M to obtain the similarity S tm between F and S mt : S tm = F T · Q M ; S mt = Q M · F T ; Among them, S tm represents the similarity matrix of the original text features and the mask image features of the image to be segmented, and S mt represents the similarity matrix of the mask image features and the original text features of the image to be segmented; Align the original text features of the image to be segmented with the mask image features, and calculate the mask-text contrast loss between the mask image features and the original text features of the image to be segmented under contrastive learning training The calculation formula is as follows: Among them, represents the cross-entropy loss function, and T tm represents the correct label for mask-text matching; Calculate the mask-text matching loss: Among them, Q c represents the correctly matched prediction result, and T c represents the correctly matched label, represents the mask-text matching loss; based on and the original text features of the image to be segmented are aligned with the mask image features.

4. The category-guided image segmentation method assisted by a multimodal language model according to claim 1, wherein In S4, input the image features guided by the aligned text features, the interactive click features, and the aligned text features into a mask decoder for normalized focal loss calculation to obtain a segmentation loss, specifically as follows: F I = VisualEncoder(I); F P = PromptEncoder(P); Y = MaskDecoder(F I , F P ); Among them, P represents the received interactive click input; I represents the original image to be segmented; VisualEncoder() represents the image encoder; F I represents the original image feature of the image to be segmented extracted by the image encoder; PromptEncoder() represents the prompt encoder; F P represents the interactive click feature extracted by the prompt encoder; MaskDecoder() represents the prediction mask generation by the mask decoder; Y represents based on F I and F P generated prediction mask; The formula for the normalized focal loss is as follows: Among them, represents the obtained segmentation loss, Y represents the predicted mask, represents the normalized focal loss calculation function, and M represents the ground truth mask.

5. The category-guided image segmentation method assisted by a multimodal language model according to claim 1, wherein The formula for the overall training objective is as follows: Among them, represents the overall training objective; represents the segmentation loss; w mtc represents the learnable weight of the balanced mask-text contrast loss; represents the mask-text contrast loss; w mtm represents the learnable weight of the balanced mask-text matching loss; represents the mask-text matching loss; w mc represents the learnable weight of the balanced class matching loss; represents the class matching loss; w mtc 、w mtm and w mc are used to balance the mask-text contrast loss, the mask-text matching loss, and the class matching loss.

6. A category-guided image segmentation system assisted by a multi-modal language model, characterized in that, Including: A feature extraction module for extracting the original image features of the image to be segmented through an image encoder and extracting the interactive click features of the user input through a prompt encoder; A masked image feature acquisition module for encoding the ground truth mask corresponding to the original image of the image to be segmented through a convolutional network to obtain mask features, splicing the mask features with the original image features of the image to be segmented, and performing cross-attention fusion operations to obtain masked image features; A feature alignment module for extracting the original text features of the image to be segmented through a text encoder and aligning the original text features in the image to be segmented with the masked image features based on contrastive learning training to obtain aligned text features; An overall training objective calculation module for performing cross-attention operations on the aligned text features and the original image features to obtain image features guided by the aligned text features, inputting the image features guided by the aligned text features, the interactive click features, and the aligned text features into a mask decoder for normalized focal loss calculation to obtain a segmentation loss, calculating the overall training objective based on the segmentation loss, and guiding the model to segment specific objects through the overall training objective.

Images