Weakly supervised directional segmentation method based on semantic and detail collaboration

Abstract

The application belongs to the technical field of artificial intelligence, and discloses a weakly supervised directional segmentation method based on semantic and detail cooperation. First, two Transformer-based encoders are used to extract the features of input images and text, and then the two features are input into a semantic perception module and a detail cognition module. The two modules focus on high-level semantics and low-level details respectively and are used for a collaborative learning strategy, which integrates cross-modal internal loss, matching invariance loss and region contrast loss to realize sufficient visual language alignment. The semantic perception module and the detail cognition module generate an activation map respectively, and then the activation maps are integrated by a cooperation module to output an accurate segmentation mask. The application only uses image text pairs to supervise the training of the model, and does not rely on dense pixel-level labels, thereby significantly reducing the cost of manual labeling while realizing good image segmentation performance of natural language instructions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a weakly supervised referential segmentation method based on semantic and detail cooperation. BACKGROUND

[0002] The referential segmentation task refers to given an image and a language description of a target object, according to the input natural language description, the pixel region related to the description in the image is segmented out. This task is different from the semantic segmentation of the image, and the image segmentation is combined with the natural language. Due to the increasing demand for visual language understanding in practical applications, this is a promising task, such as text-based image editing, human-computer interaction and visual navigation. Compared with the single-modal visual segmentation task, the referential segmentation is more difficult because it has to deal with richer vocabulary and complex syntax in the natural language, and at the same time understand the relationship between the text and the corresponding instance. The difficulty of this task lies in locating the relevant target object and segmentation under the given language condition. Recently, the fully supervised referential segmentation technology has made significant progress, but the high acquisition cost of the pixel-level annotation of each target object limits the applicability and scalability of the technology. Therefore, the weakly supervised referential segmentation gradually attracts the attention of researchers, which uses the useful knowledge hidden in the image-text pair to effectively identify the alignment between the language and the image region.

[0003] Existing weakly supervised referential segmentation methods can be divided into two categories: attention-based methods, such as “Dongwon Kim, Namyup Kim, Cuiling Lan, and Suha Kwak. Shatter and Gather: Learning Referring Image Segmentation with Text Supervision [C]. In International Conference on Computer Vision, IEEE, 2023”, and contrastive learning-based methods, such as “Fang Liu, Yuhao Liu, Yuqiu Kong, et al. Referring Image Segmentation Using Text Supervision [C]. In International Conference on Computer Vision, IEEE, 2023”. The former captures long-range dependencies between image-text modalities and generates cross-attention maps rich in high-level semantic information. However, this attention map often focuses on only a small number of objects because it does not explicitly utilize the underlying detailed information of the scene. To alleviate this problem, researchers use the attention map to select candidate masks generated by unsupervised object discovery techniques as the final result. However, the performance of such methods is significantly affected by the quality of the candidate masks. Contrastive learning-based methods mainly constrain the alignment between image features and text features by calculating the affinity of cross-modal features and produce activation maps. Since these methods implicitly strengthen the consistency between image and text features, they do not introduce high-level semantics into cross-modal embeddings, resulting in activation maps that focus more on low-level details and incorrectly respond to irrelevant regions. These methods perform poorly when the referred object exhibits a similar visual appearance to the background or the reference expression is ambiguous.

[0004] Methods based on semantic and detail collaboration explicitly utilize and combine high-level semantics and low-level details during cross-modal feature learning, which can well solve the above problems. Therefore, the present application proposes a weakly supervised referential segmentation method based on semantic and detail collaboration. SUMMARY

[0005] The technical problem to be solved by the present application is to make up for the shortcomings of current natural language instruction image segmentation methods under weak supervision, and to propose a weakly supervised referential segmentation method based on semantic and detail collaboration. The present method involves deep learning and computer vision content, and is based on a semantic perception module, a detail cognition module, and a collaborative learning strategy to realize weakly supervised natural language instruction image segmentation, in order to improve the accuracy of image segmentation.

[0006] The technical scheme of the present application is as follows: a weakly supervised directional segmentation method based on semantic and detail cooperation, the steps being as follows:

[0007] Step 1, constructing an image encoder to extract image features Wherein one image is segmented into N v non-overlapping patches, D v represents the channel dimension of the image features; a text encoder is constructed; text features are extracted The input is a sentence containing N t words, which is labeled as a label sequence, and a [CLS] label is added to the beginning of the sequence; D t represents the channel dimension of the text features;

[0008] Step 2, constructing a semantic perception module

[0009] The semantic perception module captures the long-term dependency relationship between image features and text features through an attention mechanism, and fuses the two features together to obtain an activation map M sem ;

[0010] Step 3, constructing a detail perception module

[0011] The detail perception module integrates image features and text features by exploring the low-level details of cross-modal features, and obtains an activation map M det ; the image features are weighted through M det to obtain cross-modal features F det ;

[0012] Step 4, constructing a cooperative learning strategy

[0013] A cooperative learning strategy is adopted to understand the complex correlation between images and texts, and a cooperation module is used to generate a segmentation mask using activation maps that highlight high-level semantics and low-level details respectively;

[0014] Subsequently, the cooperative learning strategy utilizes cross-modal information from three perspectives to ensure comprehensive image-text alignment: cross-modal internal loss, matching invariance loss, and region contrast loss; in addition, a total variation regularization loss is introduced to improve segmentation performance; a unified loss function simultaneously constrains the semantic perception module and the detail perception module, ensuring that the cross-modal features are aligned at two different levels;

[0015] Step 5, after the training of steps 1-4 is completed, input an image and a short natural language sentence, and send the image into the image encoder to extract image features, and send the short natural language sentence into the text encoder to extract text features; send the image features and the text features into the semantic perception module and the detail perception module to generate cross-modal features Fsem and cross-modal features F focusing on low-level details det At the same time, two activation maps are generated; the two activation maps are input into the cooperation module, and the segmentation result is output.

[0016] Further, a new negative text generation method is added to ensure accurate understanding of the referring expression;

[0017] The new negative text generation method refers to introducing more difficult-to-identify negative text by replacing the key words in the paired text; first, parse the text, and assign a part-of-speech tag to each word using a text parser; then, two strategies are used to generate challenging negative text; the first is to randomly mask an adjective or noun and fill it with RoBERTa; if there is no adjective in the original sentence, a color word or a location word is randomly added to the sentence; the second is to exchange the positions of two nouns in the text; if there is only one noun, a class name is randomly selected from the COCO dataset and added to the text, and the positions of the two nouns are exchanged; by modifying the key words of the sentence, the segmentation model perceives the subtle differences in semantic information in natural language when learning the matching between images and text, thereby effectively improving its ability to distinguish real targets from image backgrounds.

[0018] The semantic perception module uses the gradient-weighted class activation mapping technique to determine the area in the image that has the greatest impact on the final classification result using the gradient of image-text similarity, and then distinguishes the target object from a global perspective;

[0019] The semantic perception module is composed of 6 stacked network layers, each containing a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network, with image features and text features as inputs; for the cross-attention layer, the text features are used as queries, and the image features are used as keys and values; the semantic perception module outputs the cross-modal feature representation as where D h is the channel dimension of the cross-modal feature; the cross-modal feature is constrained by the matching invariance loss to ensure alignment between image-text pairs; the gradient-weighted class activation mapping technique is used to generate an activation map based on the cross-modal feature F sem The activation map indicates the image region that is most semantically related to the referring expression,

[0020]

[0021] where, is the affinity matrix of the cross-attention layer of the semantic perception module, represents the nth column element taken from the matrix A cross ​​reflects the contribution of the affinity matrix to the similarity score S sem ; the similarity score is used to evaluate the matching degree between image features and text features, and represents the element-wise multiplication operation.

[0022] The detail-aware module first extracts local features from the image using a convolutional neural network, and uses an upsampling technique to improve the resolution of the image features, obtaining a feature where N up represents the number of non-overlapping patches; the detail-aware module generates an activation map by calculating the affinity between normalized image features and text features

[0023]

[0024] where and are linear projection layers that map input image features and text features to a dimension D h ; is the [CLS] embedding of the text features, and norm(·) represents L2 normalization; the image features are weighted using M det to obtain cross-modal features

[0025]

[0026] where is a linear projection layer that maps input image features to a dimension D h .

[0027] The collaboration module jointly integrates the high-level semantic information of M sem and the low-level details of M det ; first, considering that M sem generated by the semantic-aware module focuses on a limited object area, a block similarity propagation method is used to expand the effective activation area, and the activation score of each block in M sem is extended to its adjacent blocks according to its similarity;

[0028] First, calculate the similarity matrix between image blocks

[0029]

[0030] where and are linear projection layers; then, calculate the corrected activation map

[0031]

[0032] where N(n) is the set of eight blocks spatially adjacent to the nth image block in the 2D plane, and is the softmax function; M rec is obtained by merging M det and M final

[0033] M rec = Up(M det ) ^ M rec

[0034] where Up(·) denotes upsampling M det to the same size as M cmi .

[0035] The cross-modal internal loss enables the method to directly use cross-modal feature information to distinguish whether an image-text pair matches or not; the loss is used to learn the [CLS] embedding of the cross-modal feature, and the true value label is obtained by judging the matching of the image-text pair:

[0036] L sem = CE(MLP(F label [CLS]), y det ) + CE(MLP(F label ), y sem )

[0037] where MLP is a multi-layer perceptron used to reduce the channel number of the multi-modal feature to 2; when the cross-modal features F det and F label are obtained from a correctly aligned image-text pair, the value of y j is 1, otherwise it is 0; CE(·, ·) denotes the cross-entropy loss;

[0038] The matching invariance loss is used to pull the cross-modal features generated from real image-text pairs closer to the text features, while pushing the cross-modal features from mismatched image-text pairs away from the text features; if the ith image and the jth text match, the cross-modal features and will be closely aligned with the text feature T j ; on the contrary, the distance between them will be enlarged; first, the similarity between and T j is calculated,

[0039]

[0040] where m and n represent the mth word and the nth word, respectively; the matching invariance loss is represented by the InfoNCE loss:

[0041] L miv = InfoNCE(Ssem )+InfoNCE(S det )

[0042]

[0043] where τ represents a learnable temperature parameter, B represents the size of the training batch; by applying the matching invariance loss to the cross-modal features containing high-level semantics and low-level details, the alignment between images and texts is distinguished at various levels;

[0044] In order to further distinguish highly similar negative image-text pairs, the influence of negative image-text pairs showing high matching similarity is increased during the training process, and the similarity score is used to select image-text pairs; then, the negative texts of each image in the batch are sampled according to the similarity score, and the mismatched images of each original text are sampled according to the similarity score; further, the matching invariance loss is modified as:

[0045] L′ miv =L miv +InfoNCE(S sem′ )+InfoNCE(S det′ )

[0046] where S sem′ is a similarity graph obtained by setting the elements corresponding to the mismatched image-text pairs not sampled from S sem to -10000, and S det′ is a similarity graph obtained by setting the elements corresponding to the mismatched image-text pairs not sampled from S det to -10000;

[0047] The region contrast loss is used to align image features and text features at a fine-grained level; a segmentation mask generated during the training phase is used to facilitate region-level contrast learning; the loss formula is:

[0048] L rct =InfoNCE(S mask )

[0049] where S mask represents the cosine similarity graph between the image features V mask of the mask region in the i-th image in the same batch and the features T of the j-th text;

[0050]

[0051] where and are linear projection layers that map input features to a dimension D h ; a segmentation mask M binaryand the original image I to obtain a mask image feature is denoted as

[0052] V mask = E v (I binary )

[0053] where UP(·) denotes upsampling M binary to the size of I,

[0054] M binary = Gumbel-Max(reshape 2d (M final ))

[0055] where Gumbel-Max(·) is a hard assignment method that ensures the binary mask remains differentiable and the whole process is end-to-end trainable; reshape 2d (·) denotes reshaping M final from 1D to 2D;

[0056] Applying a total variation regularization loss makes V up and V more smooth in the feature space:

[0057] L reg = ψ(V up ) + ψ(V)

[0058] where ψ(·) is an anisotropic total variation norm.

[0059] The image encoder adopts a Swin-Transformer network structure as the backbone network; and the text encoder adopts a 6-layer Transformer network structure as the backbone network.

[0060] The beneficial effects of the present application are: the present application solves the referential segmentation task only through text supervision; it uses a semantic perception module and a detail perception module to integrate image features and text features, and explores high-level semantics and low-level details respectively; the collaborative learning strategy aims to enhance the interaction between the two modules, and realizes comprehensive visual language alignment; in addition, the present application introduces a new negative text generation method to ensure accurate understanding of the referential expression; extensive experiments on three benchmarks show that the present application is superior to the current most advanced weakly supervised method. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 It is a weakly supervised referential segmentation method training framework based on semantic and detail collaboration.

[0062] Figure 2 It is an internal specific structure diagram of the semantic perception module.

[0063] Figure 3 The internal specific structure of the detail cognition module is shown.

[0064] Figure 4 The schematic diagram of the new negative text generation method is shown. DETAILED DESCRIPTION

[0065] The specific embodiments of the present application are further illustrated below in combination with the drawings and technical solutions.

[0066] The method uses a semantic perception module and a detail cognition module, and explores the potential coordination of semantics and details through a collaborative learning strategy; the semantic perception module uses a cross-modal interaction method based on attention and generates an attention map indicating the image region using gradient weighting technology; this top-down method encourages accurate positioning of the target object, but cannot detect the entire object region; the detail cognition module explores the potential relevant region in a bottom-up manner by calculating the similarity between image and text features, thereby generating an activation map; this two-pronged approach ensures comprehensive information extraction and lays a solid foundation for accurate segmentation guided by text description; the collaborative learning strategy uses a collaboration module to integrate the advantages of the above two modules; in addition, unlike traditional contrast learning-based methods that directly align visual and text features, our strategy limits the potential consistency of cross-modal features from three perspectives: cross-modal feature internal alignment, cross-modal feature and text feature alignment, and mask region image feature and text feature alignment; to further encourage fine-grained alignment between cross-modal features, we propose a negative text generation method that is applied to the collaborative learning stage; it guides the model to focus on more subtle basic elements such as attributes and relationships, improving the model's performance in distinguishing and understanding natural language.

[0067] Figure 1 The schematic diagram of the network training framework based on semantic and detail collaboration is shown; the network includes an image encoder, a text encoder, a semantic perception module, a detail cognition module, and a collaboration module; where V i represents the i-th image feature, T i represents the text feature matched with the picture, T j does not match the picture; the semantic perception module and the detail cognition module are used for cross-modal feature fusion, which integrates image features and text features by exploring high-level semantics and low-level detail information, while generating two complementary activation maps; the collaborative learning strategy uses the collaboration module to combine the activation maps and generate the final segmentation mask; it mainly uses three losses to jointly train the entire network, namely the cross-modal internal loss L cmi , the matching invariance loss L miv , and the region contrast loss L rctto achieve comprehensive visual language alignment.

[0068] Figure 2 for the detailed structure of the semantic perception module, wherein denotes the cross-modal feature containing high-level semantics generated by the paired image feature and text feature, Conversely.

[0069] Figure 3 for the detailed structure of the detail perception module, wherein denotes the image feature after upsampling, refers to the matrix multiplication operation; denotes the cross-modal feature containing low-level details generated by the paired image feature and text feature, Conversely.

[0070] Figure 4 The negative text generation method is introduced, and "girl under a green umbrella" is an example.

[0071] A weakly supervised directional segmentation method based on semantic and detail collaboration, specifically including the following steps:

[0072] Step 1, build an image encoder and a text encoder;

[0073] The Swin-Transformer network structure is used as the backbone network to build the image encoder, which is used to extract image features Where one image is divided into N v non-overlapping patches, D v denotes the channel dimension of the visual feature; the input of the image encoder is a three-channel RGB image, and the output image feature corresponds to a channel number of 1024.

[0074] The 6-layer Transformer network structure is used as the backbone network to build the text encoder, which is used to extract text features The input is a sentence containing N t words, which is marked into a token sequence, and a [CLS] token is added to the beginning of the sequence; D t denotes the channel dimension of the text feature, and its value is 768.

[0075] Step 2, build a semantic perception module;

[0076] The semantic perception module captures the long-term dependency between image features and text features through an attention mechanism, and fuses the two features together; it uses gradient-weighted class activation mapping technology to determine which regions in the image have the greatest impact on the final classification result using the gradient of image-text similarity, and then identifies the target object from a global perspective.

[0077] The semantic perception module is composed of 6 stacked network layers, each containing a multi-head self-attention layer, a multi-head cross-attention layer, and a feedforward network. The module takes image features and text features as input, as shown in Figure 2 For the cross-attention layer, the text features are used as queries, and the image features are used as keys and values; the semantic perception module outputs a fused feature representation This feature is constrained by a matching invariance loss to ensure alignment between image-text pairs; then, using gradient-weighted class activation mapping technology, based on F sem Generate activation map This activation map indicates the visual regions that are most semantically relevant to the reference expression,

[0078]

[0079] where is the affinity matrix of the cross-attention layer of the semantic perception module; reflects the contribution of the affinity matrix to the similarity score S sem , which evaluates the degree of matching between image features and text features; ⊙ denotes the element-wise multiplication operation.

[0080] The activation map M sem highlights the image regions most relevant to the reference expression; a global perspective helps capture a broader contextual framework and overall semantic content; however, this top-down localization method mainly focuses on the most prominent regions when determining the matching of cross-modal features, resulting in M sem often overlooks subtle details, especially at object boundaries; therefore, more detailed information about the referenced object is needed.

[0081] Step 3, build the detail perception module;

[0082] The detail perception module integrates image features and text features by exploring the low-level details of cross-modal features, as shown in Figure 3 First, it uses a convolutional neural network to extract local features from the image and uses upsampling techniques to improve the resolution of visual features; it obtains a feature where N updenotes the number of non-overlapping patches; the upsampled image features contain more detailed information, such as color and texture, thus facilitating subsequent low-level perception; then, the module generates an activation map by computing the affinity between the normalized image features and the text features

[0083]

[0084] wherein and is a linear projection layer that maps the input to a dimension D h ; is the [CLS] embedding of the text features, and norm(·) denotes L2 normalization; this method can utilize the alignment of low-level image features and text details to recognize the entire object from the bottom up; the activation map generated by the shallow interaction between the two modalities mainly highlights low-level information and makes a more extensive recognition of the entire object.

[0085] Subsequently, the image features are weighted using M det to obtain cross-modal features

[0086]

[0087] wherein is a projection that maps the input to a dimension D h .

[0088] Step 4, constructing a collaborative learning strategy

[0089] The present application adopts a collaborative learning strategy to understand the complex correlation between images and texts, such as Figure 1 as shown in the right half; it uses a collaboration module to generate a segmentation mask using two activation maps that highlight high-level semantics and low-level details, respectively.

[0090] The collaboration module jointly integrates the high-level semantic information of M sem and the low-level details of M det ; first, considering that M sem generated by the semantic perception module is concentrated in a limited object area, the present application uses a block similarity propagation method to expand the effective activation area; this involves extending the activation score of each block in M sem to its neighboring blocks according to its similarity.

[0091] First, calculate the similarity matrix S between image blocks

[0092]

[0093] wherein and is the linear projection layer; then, the corrected activation map is calculated

[0094]

[0095] where N(n) is the set of eight blocks spatially adjacent to the nth block in the 2D plane, and σ(·) is the softmax function; compared with the previous method, this approach eliminates the interference produced by the image blocks far from the nth block, because the distant image blocks are not always highly relevant to the nth block; however, M rec still activates the background region incorrectly; this problem can be solved by merging M rec with M det The final activation map is represented as:

[0096] M final = Up(M rec )⊙M det

[0097] where Up(·) denotes up-sampling M rec to the same size as M det ;

[0098] Subsequently, the cooperative learning strategy utilizes cross-modal information from three perspectives to ensure comprehensive visual-linguistic alignment: cross-modal internal loss, matching invariance loss, and region contrast loss; in addition, a total variation regularization loss is introduced to improve segmentation performance; the unified loss function simultaneously constrains the semantic perception module and the detail perception module, ensuring that cross-modal features are aligned at two different levels; the semantic perception module has a more significant impact on the loss components related to high-level semantic information, while the detail perception module tends to have a more obvious impact on the alignment of low-level features.

[0099] The cross-modal internal loss enables the model to directly use cross-modal feature information to distinguish whether an image-text pair matches; this loss is used to learn the [CLS] embedding of cross-modal features, and the true value label is obtained by judging the matching of the image-text pair:

[0100] L cmi = CE(MLP(F sem [CLS]), y label )+ CE(MLP(F det ), y label )

[0101] where MLP is a multi-layer perceptron that reduces the channel number of multi-modal features to 2; when the cross-modal features F sem and F det are obtained from a correctly aligned image-text pair, y labelis 1, otherwise 0; CE(·, ·) denotes the cross-entropy loss.

[0102] The application also uses a matching invariance loss to pull the cross-modal feature generated by the real image-text pair closer to the text feature, while pushing the cross-modal feature from the mismatched image-text pair away from the text feature; intuitively, the cross-modal feature can be regarded as a language feature enriched in the visual context; therefore, if the ith image and the jth text match, the cross-modal feature and will be closely aligned with the text feature T j ; on the contrary, the distance between them will be enlarged; first, the similarity between the cross-modal feature and the text feature is calculated,

[0103]

[0104] where m and n represent the mth and nth words respectively; then, the matching invariance loss can be represented by the InfoNCE loss:

[0105] L miv = InfoNCE(S sem ) + InfoNCE(S det )

[0106]

[0107] where τ represents a learnable temperature parameter, and B represents the size of the training batch; by applying the matching invariance loss to the cross-modal feature containing high-level semantics and low-level details, the model can distinguish the alignment between images and texts at various levels, thereby improving the performance of the model;

[0108] In order to further distinguish highly similar but mismatched image-text pairs, the application increases the influence of negative image-text pairs that exhibit high matching similarity during the training process, that is, the similarity score is used to select image-text pairs; then, the negative texts for each image in the batch are sampled according to the similarity score, and the mismatched images for each original text are sampled according to the similarity score; therefore, the matching invariance loss is further modified as:

[0109] L′ miv = L miv + InfoNCE(S sem′ ) + InfoNCE(S det′ )

[0110] where S sem′ is a similarity map obtained by setting the elements corresponding to the mismatched image-text pairs not sampled from S sem to -10000, and S det′ is a similarity map obtained by setting the elements corresponding to the mismatched image-text pairs not sampled from Sdet a similarity map obtained by setting the corresponding element of the sampled mismatched image-text pair to -10000;

[0111] The third loss used by the present application is the region contrast loss; this loss is used to align image features and text features at a fine-grained level; compared with the traditional contrast learning method of feature alignment at the image level, the present application uses the segmentation mask generated in the training stage to promote region-level contrast learning; this aims to make the image features of the mask region closer to the corresponding text features, while ensuring a clear distinction from the mismatched text features; the loss formula is:

[0112] L rct =InfoNCE(S mask )

[0113] where S mask represents the cosine similarity map between the image features V mask of the mask region in the i-th image in the same batch and the features T of the j-th text.

[0114]

[0115] where and are linear projections that map the input to a dimension D h ; the mask image features V binary are obtained using M mask and the original image I, and are represented as

[0116] V mask =E v (I⊙UP(M binary ))

[0117] where UP(·) represents up-sampling M binary to the size of the original image I, and M binary represents the segmentation mask,

[0118] M binary =Gumbel-Max(reshape 2d (M final ))

[0119] where Gumbel-Max(·) is a hard assignment method that ensures that the binary mask remains differentiable and the entire process is end-to-end trainable; reshape 2d (·) represents changing M final from 1 dimension to 2 dimensions.

[0120] In addition, the embedding similarity between adjacent blocks of images should be stronger; the present application applies a total variation regularization loss to Vup and V is more smooth in the embedding space:

[0121] L reg = ψ(V up )+ ψ(V)

[0122] where ψ(·) is an anisotropic total variation norm.

[0123] Further, the negative text generation method is introduced

[0124] Aligning the entire reference text directly with the image may not effectively identify the subtle details of basic elements such as attributes and relationships; In order to avoid learning a bag-of-words representation, the present application uses a negative text generation technique in collaborative learning, which helps to understand the reference expression in a fine-grained manner, as shown in Figure 4

[0125] In addition to the negative text in the batch that does not match the image, the present application also introduces more difficult-to-identify negative text by replacing keywords in the paired text; First, parse the text, that is, use the text parser Spacy to assign a part-of-speech tag to each word; Then adopt two strategies to generate challenging negative text; The first is to randomly mask an adjective or noun and fill it with RoBERTa; If there is no adjective in the original sentence, a color word or a location word is randomly added to the sentence; The second is to exchange the positions of two nouns in the text; If there is only one noun, a class name is randomly selected from the COCO dataset and added to the text, and the positions of the two nouns are exchanged; By modifying the key words of the sentence, the model can perceive the subtle differences in semantic information in natural language when learning the matching between the image and the text, thereby effectively improving its ability to distinguish real targets from image backgrounds.

[0126] Construct the overall network structure; input the image and a short natural language sentence, send the image into the image encoder to extract the image features, and send the short natural language sentence into the text encoder to extract the text features; Send the image features and the text features into the semantic perception module and the detail cognition module to respectively generate cross-modal features focusing on high-level semantics and low-level details, and simultaneously generate two activation maps; Send the two activation maps into the collaborative module to output the segmentation result.

[0127] Training stage: use Swin-Transformer-Bas e ​As an image encoder, take 6-layer Transformer as text encoder, train initial load pre-training weight, train initial load X-VLM weight, as“Yan Zeng, Xinsong Zhang, Hang Li. Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts [C]. In International Conference on Machine Learning, ACM, 2022”; Use overall loss L = L reg +λ1L′ miv +λ2L cmi +λ3L rct Train the model, where λ1 = 1.0, λ2 = 2.0 and λ3 = 0.2 are the super parameters of balancing loss importance; The model is trained on RefCOCO, RefCOCO+, RefCOCOg datasets, and finally validated on the corresponding test sets respectively; Train for 15 rounds, a total of 56000 iterations, batch size is 32, input image resolution is 224x224; The network is optimized by AdamW optimizer, weight decay is 1e -2 -4; The learning rate is increased from 1e -8 -4 to 1e -5 -2 in the first round, and linearly decays to 1e -8 -4 in subsequent rounds.

[0128] The model performance is improved by 6.21%, 7.25%, 6.11% and 5.88% respectively on the validation sets of RefCOCO, RefCOCO+, G-Ref(Google partition) and G-Ref(UMD partition) compared with the prior art.

Claims

1. A weakly supervised directional segmentation method based on semantic and detail collaboration, characterized in that, The steps are as follows: Step 1, construct image encoder to extract image features where one image is split into N v non-overlapping patches, D v represents the channel dimension of image features; construct text encoder; extract text features input is a sentence containing N t words, which is labeled into a token sequence, a [CLS] token is added to the beginning of the sequence; D t represents the channel dimension of text features; Step 2, constructing a semantic perception module; The semantic perception module captures long-term dependency relationship between image features and text features through an attention mechanism, and fuses the two features together to obtain an activation map M sem ; Step 3, constructing a detail perception module; The fine-grained cognition module integrates the image features and the text features by exploring the low-level details of the cross-modal features to obtain an activation map M det ; the image features are weighted by M det to obtain cross-modal features F det ; Step 4, constructing a collaborative learning strategy A collaborative learning strategy is adopted to understand the complex correlation between images and texts, and a collaborative module is used to generate a segmentation mask by using an activation map that highlights high-level semantics and low-level details respectively; Subsequently, the collaborative learning strategy utilizes cross-modal information from three perspectives to ensure comprehensive image-text alignment: cross-modal internal loss, matching invariance loss, and region contrast loss; In addition, a total variation regularization loss is introduced to improve segmentation performance; The unified loss function constrains the semantic perception module and the detail perception module, ensuring that the cross-modal features are aligned at two different levels; Step 5, after training is completed through steps 1-4, input an image and a short sentence in natural language, send the image into an image encoder to extract image features, and send the short sentence in natural language into a text encoder to extract text features; send the image features and the text features into a semantic perception module and a detail cognition module to respectively generate a cross-modal feature F sem paying attention to high-level semantics and a cross-modal feature F det paying attention to low-level details; simultaneously generate two activation maps; send the two activation maps into a collaboration module to output a segmentation result.

2. The method of claim 1, wherein, A new negative text generation method is also added to ensure accurate understanding of the referential expression; The new negative text generation method refers to introducing more difficult-to-identify negative texts by replacing the key words in the paired text; First, parse the text and assign a part-of-speech tag to each word using a text parser; Then, two strategies are used to generate challenging negative texts; The first is to randomly mask an adjective or noun and fill it with RoBERTa; If there is no adjective in the original sentence, a color word or a location word is randomly added to the sentence; The second is to exchange the positions of two nouns in the text; If there is only one noun, a class name is randomly selected from the COCO dataset and added to the text, and the positions of the two nouns are exchanged; By modifying the key words of the sentence, the segmentation model can perceive the subtle differences in semantic information in natural language when learning the matching between images and texts, thereby effectively improving its ability to distinguish real targets from image backgrounds.

3. The method according to claim 1 or 2, wherein, The semantic perception module uses gradient-weighted class activation mapping technology to determine the areas in the image that have the greatest impact on the final classification result using the gradient of image-text similarity, and then distinguishes the target object from a holistic perspective; The semantic perception module is composed of 6 stacked network layers, each containing a multi-head self-attention layer, a multi-head cross-attention layer and a feedforward network, taking image features and text features as input; for the cross-attention layer, the text features are taken as the query, and the image features are taken as the key and value; the semantic perception module outputs the cross-modal feature representation as where D h represents the channel dimension of the cross-modal feature; the cross-modal feature is constrained by the matching invariance loss to ensure the alignment between the image-text pair; using the gradient-weighted class activation mapping technique, the cross-modal feature F sem generates an activation map The activation map indicates the image region that is most semantically relevant to the quoted expression, wherein, is an affinity matrix of a cross-attention layer of the semantic perception module, denotes the nth column element taken from the matrix A cross ; and reflects the contribution of the affinity matrix to a similarity score S sem that assesses the degree of matching between the image features and the text features, and ⊙ denotes an element-wise multiplication operation.

4. The method of claim 1 or 2, wherein, The detail-aware module first extracts local features from the image using a convolutional neural network, and uses an upsampling technique to improve the resolution of the image features, obtaining a feature with higher resolution where N up represents the number of non-overlapping patches; the detail-aware module generates an activation map by computing the affinity between the normalized image features and the text features wherein and is a linear projection layer that maps the input image features and text features to a dimension D h ; is the [CLS] embedding of the text features, norm(·) denotes L2 normalization; using M det to weight the image features, resulting in cross-modal features wherein is a linear projection layer that maps input image features to dimension D h D.

5. The method of claim 1 or 2, wherein, The cooperation module joint integrates M sem high-level semantic information and M det low-level details; first, considering that the M sem generated by the semantic perception module focuses on a limited object area, a block similarity propagation method is used to expand the effective activation area, and the activation score of each block in M sem is extended to its adjacent blocks according to its similarity; First, a similarity matrix between image blocks is calculated wherein and is a linear projection layer; Then, the corrected activation map is calculated where N(n) is the set of eight blocks spatially adjacent to the nth image block in the 2D plane, σ(·) is the softmax function; and M rec is the final activation map. det The final activation map is obtained by merging as follows: M final = Up(M rec )⊙M det where Up(·) denotes upsampling of M rec to the same size as M det .

6. The method of claim 1 or 2, wherein, The cross-modal internal loss enables the method to directly use cross-modal feature information to distinguish whether an image-text pair matches; This loss is used to learn the [CLS] embedding of cross-modal features and obtain the true value label by judging the matching of image-text pairs: L cmi = CE(MLP(F sem [CLS]), y label ) + CE(MLP(F det ), y label ) where MLP is a multi-layer perceptron used to reduce the number of channels of the multi-modal feature to 2; when the cross-modal feature F sem and F det is obtained from a correctly aligned image-text pair, the value of y label is 1, otherwise 0; CE(·, ·) denotes the cross-entropy loss; The matching invariance loss is used to pull the cross-modal features generated by the real image-text pair closer to the text features, while pushing the cross-modal features from the mismatched image-text pair away from the text features; If the ith image and the jth text match, then the cross-modal feature and will be closely aligned with the text feature T j ; instead, the distance between them will be pulled apart; first, the similarity between and T j is computed, Where m and n represent the mth word and the nth word respectively; The matching invariance loss is represented by the InfoNCE loss: L miv = InfoNCE(S sem ) + InfoNCE(S det ) Where τ represents a learnable temperature parameter, and B represents the size of the training batch; By applying the matching invariance loss to cross-modal features containing high-level semantics and low-level details, the alignment between images and texts at various levels is identified; In order to further distinguish the highly similar negative image-text pairs, the influence of the negative image-text pairs showing high matching similarity is increased in the training process, and the similarity score is used to select the image-text pairs; then, the negative text of each image in the batch is sampled according to the similarity score, and the non-matching image of each original text is sampled according to the similarity score; further, the matching invariance loss is modified as: L' miv = L miv + InfoNCE(S sem′ ) + InfoNCE(S det′ ) where S sem′ is a similarity map obtained by setting elements corresponding to mismatched image-text pairs not sampled from S sem to -10000, and S det′ is a similarity map obtained by setting elements corresponding to mismatched image-text pairs not sampled from S det to -10000. The region contrast loss is used to align the image features and the text features at the fine-grained level; the segmentation mask generated in the training stage is used to promote the region-level contrast learning; and the loss formula is: L rct = InfoNCE(S mask ) where S mask represents the image feature V of the mask region in the i-th image in the same batch mask cosine similarity map between the feature T of the j-th text wherein and is a linear projection layer mapping the input features to a dimension D h Using the segmentation mask M binary and the original image I, the masked image features are represented as V mask = E v (I⊙UP(M binary )) where UP(·) denotes upsampling the M binary to the size of I, M binary = Gumbel-Max(reshape 2d (M final )) wherein Gumbel-Max (·) is a hard assignment method that ensures the binary mask remains differentiable and the whole process is end-to-end trainable; reshape 2d (·) denotes reshaping M final from 1D to 2D; The V up and V are smoother in the feature space: L reg = ψ(V up )+ ψ(V) Wherein, ψ(·) is an anisotropic total variation norm.

7. The method of claim 1 or 2, wherein, The image encoder adopts a Swin-Transformer network structure as a backbone network; and the text encoder adopts a 6-layer Transformer network structure as a backbone network.

Images