Weak supervision semantic segmentation method and related device

Abstract

The invention discloses a weak supervision semantic segmentation method and a related device, and the method comprises the steps: building a foreground text prompt according to the similarity between a foreground auxiliary text feature and an image X feature, screening out similar words for a foreground category name, and helping a visual language pre-training model to activate pixels during the generation of a semantic segmentation pseudo tag, the problem of insufficient activation of the target pixel is solved, and the background text prompt is constructed according to the common background category name feature of the image X, the similarity of the image X feature and the screened common background category name, so that the sensitivity of the visual language pre-training model to the co-occurrence background is improved, and the problem of the co-occurrence background is solved.

Description

Technical Field

[0001] The invention relates to a weakly supervised semantic segmentation method and a related device, and belongs to the field of image semantic segmentation. Background Art

[0002] Image semantic segmentation is a fundamental research direction in the field of computer vision. The purpose of this task is to predict a category label for each pixel of a given image, thereby obtaining a pixel-level label. Thanks to the development of deep learning algorithms, semantic segmentation has also made great progress in recent years. However, semantic segmentation training requires a large amount of pixel-level annotations. Obtaining these pixel annotations is a laborious and time-consuming task, which incurs high labor and time costs.

[0003] The emergence of weakly supervised semantic segmentation solves the above problems, because weakly supervised semantic segmentation only relies on weakly annotated labels to train the segmentation model. Some common weak annotations include bounding boxes, graffiti, points, and images. Image-level annotations are easier to obtain than other methods. At the same time, since image-level annotations only give specific target category information in the image but do not indicate the specific location of the target category, the weakly supervised semantic segmentation method based on image-level annotations is very challenging. Therefore, there have always been two problems in the process of weakly supervised semantic segmentation tasks: 1) Co-occurrence background problem: In the training data set used for weakly supervised semantic segmentation, the target object and certain background elements will frequently appear at the same time, such as rails and trains, boats and lakes, etc. This may cause the model to mistake these common background elements as part of the target object when performing segmentation tasks, thereby affecting the segmentation accuracy; 2) Insufficient activation of target pixels: During the training process, the pixels used to represent the target objects are not fully identified and activated, which affects the quality of the pseudo-labels and causes the target objects to be incorrectly segmented in the final segmentation results. Summary of the invention

[0004] The present invention provides a weakly supervised semantic segmentation method and related devices, which solve the problems disclosed in the background technology.

[0005] According to one aspect of the present disclosure, a weakly supervised semantic segmentation method is provided, comprising: Obtain image A; wherein image A is the image to be semantically segmented; Using the trained semantic segmentation model, perform semantic segmentation on image A; wherein the semantic segmentation model is trained using the semantic segmentation pseudo labels generated by the visual language pre-training model; The process of generating semantic segmentation pseudo labels using the visual language pre-training model is as follows: A foreground auxiliary text is constructed using a similar word list and a preset context of the foreground category name of image X, where image X is an image for generating semantic segmentation pseudo labels; According to the similarity between the foreground auxiliary text feature and the image X feature, similar words are filtered, and the foreground text prompt is constructed using the preset context, the foreground category name and the filtered similar words; According to the similarity between the common background category name features of image X and the features of image X, common background categories are screened, and background text prompts are constructed using the preset context and the screened common background category names; According to the foreground text cues and background text cues, pseudo labels for semantic segmentation are obtained.

[0006] In some embodiments of the present disclosure, similar words are filtered according to the similarity between the foreground auxiliary text feature and the image X feature, and a foreground text prompt is constructed using the preset context, the foreground category name and the filtered similar words, including: A text encoder is used to extract the features of the foreground auxiliary text, and an image encoder is used to extract the features of image X; Calculate the similarity between the foreground auxiliary text features and the image X features; Construct a foreground text hint based on the preset context, the foreground category name, and the similar words corresponding to the highest similarity.

[0007] In some embodiments of the present disclosure, based on the similarity between the common background category name features of image X and the features of image X, common background categories are screened, and a background text prompt is constructed using a preset context and the screened common background category names, including: A text encoder is used to extract features of common background category names of image X, and an image encoder is used to extract features of image X; Calculate the similarity between the common background category name features and the image X features; Construct background text prompts based on common background category names corresponding to preset contexts and similarities not less than a threshold.

[0008] In some embodiments of the present disclosure, obtaining a pseudo label for semantic segmentation according to a foreground text prompt and a background text prompt includes: According to the foreground text prompt and the background text prompt, the gradient-based class activation mapping is used to obtain the initial class activation map; The initial class activation map is improved by using category-aware attention affinity; According to the improved class activation map, the pseudo label of semantic segmentation is obtained.

[0009] In some embodiments of the present disclosure, when using gradient-based class activation mapping, softmax is introduced into the class activation mapping, and the class probability adopted by the softmax score is calculated. Y The formula is: ; In the formula, F T To use the text prompt features extracted by the text encoder, F I is the image X feature extracted by the image encoder, is the temperature factor, and cos represents the cosine similarity.

[0010] According to another aspect of the present disclosure, a weakly supervised semantic segmentation apparatus is provided, comprising: An acquisition module acquires image A, wherein image A is an image to be semantically segmented; The semantic segmentation module uses a trained semantic segmentation model to perform semantic segmentation on image A; wherein the semantic segmentation model is trained using semantic segmentation pseudo labels generated by a visual language pre-training model; The process of generating semantic segmentation pseudo labels using the visual language pre-training model is as follows: A foreground auxiliary text is constructed using a similar word list and a preset context of the foreground category name of image X, where image X is an image for generating semantic segmentation pseudo labels; According to the similarity between the foreground auxiliary text feature and the image X feature, similar words are filtered, and the foreground text prompt is constructed using the preset context, the foreground category name and the filtered similar words; According to the similarity between the common background category name features of image X and the features of image X, common background categories are screened, and background text prompts are constructed using the preset context and the screened common background category names; According to the foreground text cues and background text cues, pseudo labels for semantic segmentation are obtained.

[0011] In some embodiments of the present disclosure, in the visual language pre-training model of the semantic segmentation module, similar words are screened according to the similarity between the foreground auxiliary text features and the image X features, and the foreground text prompt is constructed using the preset context, the foreground category name and the screened similar words, including: A text encoder is used to extract the features of the foreground auxiliary text, and an image encoder is used to extract the features of image X; Calculate the similarity between the foreground auxiliary text features and the image X features; Construct a foreground text hint based on the preset context, the foreground category name, and the similar words corresponding to the highest similarity.

[0012] In some embodiments of the present disclosure, in the visual language pre-training model of the semantic segmentation module, common background categories are screened according to the similarity between the common background category name features of image X and the features of image X, and background text prompts are constructed using the preset context and the screened common background category names, including: A text encoder is used to extract features of common background category names of image X, and an image encoder is used to extract features of image X; Calculate the similarity between the common background category name features and the image X features; Construct background text prompts based on common background category names corresponding to preset contexts and similarities not less than a threshold.

[0013] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform a weakly supervised semantic segmentation method.

[0014] According to another aspect of the present disclosure, a computer device is provided, comprising one or more processors and one or more memories, wherein one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing a weakly supervised semantic segmentation method.

[0015] The beneficial effects achieved by the present invention are as follows: when generating a semantic segmentation pseudo label, the present invention selects similar words for the foreground category name according to the similarity between the foreground auxiliary text feature and the image X feature, constructs a foreground text prompt, helps the visual language pre-training model to activate pixels, and solves the problem of insufficient activation of target pixels; according to the common background category name feature of the image X and the similarity between the image X feature, selects the common background category name, constructs a background text prompt, and improves the sensitivity of the visual language pre-training model to the co-occurrence background, and solves the problem of the co-occurrence background. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flowchart of the weakly supervised semantic segmentation method; Figure 2 Schematic diagram of the visual language pre-training model; Figure 3 Schematic diagram for similar word filtering and common background category name filtering; Figure 4 This is a comparison chart of the visualization results with the existing methods; Figure 5 Block diagram of the weakly supervised semantic segmentation device. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. It is obvious that the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0018] Unless otherwise specified, the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0019] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0020] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0021] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0022] It should be noted that similar symbols and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it does not require further discussion in subsequent figures.

[0023] In order to solve the co-occurrence background problem and the problem of insufficient activation of target pixels, the present disclosure proposes a weakly supervised semantic segmentation method and related devices, which specifically use an improved visual language pre-training model to generate semantic segmentation pseudo labels.

[0024] Figure 1 Schematic diagram of an embodiment of the weakly supervised semantic segmentation method disclosed in the present invention. Figure 1 The embodiment can be executed by an image semantic segmentation terminal.

[0025] like Figure 1 As shown, in step 1 of the embodiment, image A is obtained; wherein image A is the image to be semantically segmented.

[0026] It should be noted that there are no special requirements for the images to be segmented, and they can be images of subway arrivals, images of trains running on rails, and other images that are prone to co-occurrence background problems.

[0027] return Figure 1In step 2 of the embodiment, a trained semantic segmentation model is used to perform semantic segmentation on image A; wherein the semantic segmentation model is trained using semantic segmentation pseudo labels generated by a visual language pre-training model.

[0028] It should be noted that the Contrastive Language-Image Pre-Training (CLIP) model launched by OpenAI has a strong ability to generate text to images, indicating that there is a great connection between the text and the corresponding parts in the image.

[0029] CLIP consists of an image encoder and a text encoder. The image encoder and the text encoder can convert the input image and text into vectors respectively, so that the image and text can be represented in the same vector space. The task of combining CLIP with weakly supervised semantic segmentation has achieved remarkable results, but there is also room for improvement. For example, the image category information input to CLIP is generally fixed and unique, which will result in less semantic information provided by the text, easy to disagree, and ultimately lead to insufficient activation of the target pixel. In addition, since CLIP has a text encoder, for background pixels, you can choose to directly input a specific background text group into CLIP for suppression. However, using the same background text group for background pixel suppression for all images will make CLIP's suppression of background pixels lack of specificity, making it difficult to separate background pixels, especially when facing images that are prone to co-occurrence background problems. This phenomenon will be more obvious.

[0030] See Figure 2 In the embodiment 3, the process of generating semantic segmentation pseudo labels using the visual language pre-training model can be as follows: Define image X as an image for generating semantic segmentation pseudo labels, use the similar word list of the foreground category name of image X and the preset context to construct the foreground auxiliary text, filter similar words based on the similarity between the foreground auxiliary text feature and the image X feature, use the preset context, the foreground category name and the filtered similar words to construct the foreground text prompt, filter common background categories based on the similarity between the common background category name feature of image X and the image X feature, use the preset context and the filtered common background category name to construct the background text prompt, and obtain the pseudo labels for semantic segmentation based on the foreground text prompt and the background text prompt.

[0031] It should be noted that Figure 2The input image X has foreground and background. Assuming that the foreground category name is defined as [CLS], ChatGPT can be used to obtain a list of similar words to the foreground category name of image X. These similar words have a certain semantic similarity with the foreground category. Specifically, the following requirement is made to ChatGPT: "Give me m words that are semantically similar to [CLS] and use the CLIP model to calculate the cosine similarity score." This query will return a list of m words and their similarity scores for a specific [CLS]. This means that for [CLS], a set of m most semantically similar words (i.e., similar words) can be obtained, which can be represented by a list as S=[CLS1,CLS2,...,CLS m ].

[0032] It should be noted that in the field of semantic segmentation, common contexts currently include "a photo of {}.", "an image of {}.", etc. Recent studies have found that using "a clean origami {}." as the context has better results. Therefore, "a clean origami {}." can be further combined with similar words in the list to construct foreground auxiliary text. For example, "a clean origami {}." and CLS1 are combined to construct the foreground auxiliary text "aclean origami { CLS1}.".

[0033] m similar words can construct m foreground auxiliary texts, which can be expressed as T=[t1,t2,t3,... ,t m ]. The text encoder is used to extract the features of the foreground auxiliary text. Specifically, T is input into the text encoder to obtain the features of each foreground auxiliary text, which can be expressed as V=[v1,v2,v3,... ,v m ].

[0034] In order to select the most suitable similar words, the similarity between the foreground auxiliary text features and the image X features can be calculated. Specifically, the image encoder is used to extract the feature F of image X, and the similarity between F and v is calculated. k The similarity of (k∈{1,2,3,...,m}) can be expressed as: ; In the formula, Sim is the abbreviation of Similarity, which means finding similarity, and || || means finding the modulus of the vector. The modulus is a measure of the length of the vector, which represents the distance from the origin to its end point in space.

[0035] We can further use the argmax function to find the highest similarity and construct the foreground text prompt based on the preset context, the foreground category name, and the similar words corresponding to the highest similarity. Assume that the similar words corresponding to the highest similarity are represented as [CLS] s , then the foreground text hint can be expressed as "a clean origami { [CLS][CLS] s}.".

[0036] When generating semantic segmentation pseudo labels, similar words are filtered out for the foreground category names based on the similarity between the foreground auxiliary text features and the image X features, and foreground text hints are constructed to help the visual language pre-training model activate pixels, solving the problem of insufficient activation of target pixels.

[0037] In the task of combining weakly supervised semantic segmentation with CLIP, the input to the text encoder is generally a text consisting of contextual cues plus foreground class names. However, in weakly supervised semantic segmentation, the name of each foreground target is fixed and unique. A single category name can provide less semantic information and is prone to disagreement. After adding a similar class for assistance, more semantic information can be provided without affecting CLIP's judgment of the true category.

[0038] It should be noted that suppressing the activation of background pixels is a necessary consideration in the weakly supervised semantic segmentation process. In the task combined with CLIP, due to the existence of the text encoder, the background category can be concretized as text for suppression, which is also the common practice in most related work. Recent studies have found that CLIP is more sensitive to co-occurring background categories than some common background categories. The images in the VOC dataset (specifically The PASCAL Visual Object Classes 2012, PASCAL VOC 2012, which has no official Chinese translation, and literally means PASCAL 2012 Visual Object Classes) are matched with some common background categories for similarity. The results show that in images that are prone to co-occurring backgrounds, such as trains, ships, motorcycles, etc. as foreground images, the similarity between co-occurring backgrounds such as rails, lakes, and ground and images is much higher than other background categories. Inspired by this, we can combine the common background category names of image X, use a text encoder to extract the features of the common background category names of image X, calculate the similarity between the common background category name features and the image X features, and construct background text prompts based on the common background category names corresponding to the preset context and the similarity not less than the threshold.

[0039] It should be noted that common background category names may include ground, sky, rails, etc. Assuming there are n background category names, the background category names can be combined to be expressed as M=[B1,B2,B3,...,B n ], due to the zero-shot learning capability of CLIP, the content in M ​​can be changed at any time without the need to retrain the classification network based on training methods, making the model more flexible.

[0040] See Figure 2 In the above example, similar to the processing of foreground auxiliary text, M is input into the text encoder to obtain the feature of each common background category name, and the similarity between each common background category name feature and the image X feature is calculated. The n similarities can be expressed as L=[L1,L2,L3,...,L n ].

[0041] Define a threshold α and use it to filter the similarity. The formula can be expressed as: ; In the formula, G represents the set of common background category names after screening, B i is the name of the i-th common background category; that is, when L i If B is greater than or equal to the threshold, i Add G. After screening, for an image, the corresponding common background class can be obtained, especially for some categories that are prone to co-occurrence background problems, such as trains, ships, etc., through this method, their co-occurrence background can be accurately found.

[0042] Combining the context with the content in G, we can construct a background text prompt, which can be specifically expressed as “a cleanorigami { G1,G2,G3,...,G g}."; where G1, G2, G3, ..., G g are the g elements in G, i.e., the names of the common background categories after screening.

[0043] When generating semantic segmentation pseudo labels, the common background category names are screened out according to the similarity between the common background category name features of image X and the features of image X, and background text prompts are constructed to increase the sensitivity of the visual language pre-training model to the co-occurring background, thus solving the problem of co-occurring background.

[0044] The weakly supervised semantic segmentation task combined with CLIP generally manually constructs a background text group to suppress background pixels. However, since these background text information is for the entire dataset rather than each image, CLIP's suppression of background pixels lacks specificity, making it difficult for the model to separate co-occurring background pixels from the target object. Here, targeted background text is screened out for each image, allowing CLIP to specifically suppress background pixels. In particular, in images where co-occurring backgrounds are likely to occur, the effect of targeted suppression of background pixels is significantly improved.

[0045] After obtaining the foreground text prompt and the background text prompt, we can enter the second stage of CLIP, which is to generate pseudo labels. Specifically, we can use gradient-based class activation mapping to obtain the initial class activation map according to the foreground text prompt and the background text prompt, and use category-aware attention affinity to improve the initial class activation map. According to the improved class activation map, we can obtain the pseudo label of semantic segmentation.

[0046] It should be noted that by adding GradCAM (Gradient-weighted Class Activation Mapping) to CLIP, you can directly use CLIP to get the initial class activation map, and then use the carefully selected text to help CLIP activation.

[0047] For an image and its corresponding foreground text prompt and background text prompt, similar to the above, a text encoder is used to extract the features of the text prompt, denoted as F T , use the image encoder to extract the features of image X, denoted as F I , calculate the class probability Y : ; In the formula, is the temperature factor, and cos represents the cosine similarity.

[0048] It should be noted that the image encoder here is not exactly the same as the previous one. The previous image encoder only needs to obtain image features to calculate the similarity to obtain text. Here, the image encoder needs to be used to generate pseudo labels. Therefore, GradCAM is added to the image encoder used for the second time, so that pseudo labels can be directly generated from features.

[0049] It should be noted that when using gradient-based class activation mapping, softmax is introduced into the class activation mapping, so Y , the softmax score can be calculated as: ; In the formula, p is the foreground class presented in the image, B For the common background categories screened out, Y c For the c The class probabilities, For the The probability of a class.

[0050] It should be noted that the last multi-head attention (MHSA) layer of the image encoder outputs image features, corresponding to the first k Category on 1 channel (such as c categories), the formula for weight activation calculation can be expressed as: ; In the formula, For the k 1st channel c The activation weights of the categories, H is the height of the image, W is the width of the image, F k1,u,v For the k The height of one channel is u , width is v Features of the image.

[0051] Then c The initial class activation map of categories can be expressed as: ; In the formula, For the c The initial class activation map of the categories, RELU is the ReLU function, K The maximum number of channels for the image After obtaining the initial class activation map, it is refined using category-aware attention affinity (CAA), which proposes to use the attention weights of the last MHSA layer ( Refers to a size of HW × HW A real-valued matrix, which is used to represent the attention weights between each pixel in the feature map and all other pixels) to improve the initial class activation map, which can be expressed as: ; In the formula, is the improved initial class activation map, M c is the initial class activation map before improvement, B c For the cThe box mask obtained by retrieving the initial class activation map of each category, ⊙ is the element-by-element product, t is the number of iterations, A is the symmetric affinity matrix obtained from the attention weights, vec Represents the vectorization of a matrix.

[0052] It should be noted that in the pseudo-label generation stage, since CLIP is a pre-trained model that does not need to be trained, pseudo-labels are generated instantly, which can save a lot of training time. For example, for the 10,582 images in the VOC dataset, it takes about 1 hour to get the pseudo-labels corresponding to all images.

[0053] It should be noted that the semantic segmentation model can use DeepLabV2 based on the deep residual network ResNet 101 as the segmentation model. DeepLab V2 (semantic segmentation model) is the second generation of DeepLab segmentation model. ResNet101 (residual network 101 layer) is widely used due to its outstanding performance in balancing computational efficiency and segmentation accuracy. DeepLab V2 has good scalability and can be well adapted to ResNet101. ResNet101 as the backbone network, coupled with the hole convolution module of DeepLab v2, can achieve excellent performance on large-scale data sets.

[0054] For the parameters of semantic segmentation model training, SGD (Stochastic Gradient Descent) is used as the optimizer, the initial learning rate is set to 2.5e-4, the batch size is set to 6, and after multiple rounds of training, the parameters corresponding to the best round of training results are saved by observing the results of the validation set. After the model training is completed, the best saved parameters are loaded into the model, and then the test set data is input into the model. The trained model can generate a relatively accurate segmentation map, see Table 1 and Table 2.

[0055] Table 1 Segmentation results on the validation set and test set of the VOC dataset

[0056] In the table, NSROM: Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation, non-salient region object mining; R101: ResNet 101, residual network 101 layer; RCA: Regional semantic contrast and aggregation for weakly supervised semantic segmentation, regional semantic contrast and aggregation for weakly supervised semantic segmentation; ESOL: Expansion and shrinkage of localization for weakly-supervised semantic segmentation, expansion and shrinkage of localization for weakly supervised semantic segmentation; L2G: A simple local to global knowledge transfer framework for weakly supervised semantic segmentation, a simple local to global knowledge transfer framework for weakly supervised semantic segmentation; IRN: Weakly supervised learning of instancesegmentation with inter-pixel relations, instance segmentation weakly supervised learning based on pixel-to-pixel relations; R50: ResNet 50, residual network 50 layers; AdvCAM: Anti-adversarially manipulated attributions for weakly and semi-supervised semantic Segmentation, attribution against adversarial manipulation for weakly supervised and semi-supervised semantic segmentation; SIPE: Self-supervised image-speciffc prototype exploration for weakly supervised semantic segmentation, self-supervised image-specific prototype exploration for weakly supervised semantic segmentation; OOD+Adv: Weakly supervised semantic segmentation using out-of-distribution data, weakly supervised semantic segmentation using out-of-distribution data;MCTfomer: Multi-class token transformer for weakly supervised semantic segmentation; WR-38: Weakly Supervised Semantic Segmentation with Relations and Multi-Scale Features-38; BECO: Boundary-enhanced Co-training for Weakly Supervised Semantic Segmentation; MIT-B2: MIT Boundary-aware Bilateral Network; Mat-Label: Treating pseudo-labels generation as image matting for weakly supervised semantic segmentation; ACR: Weakly supervised semantic segmentation via adversarial learning of classifier and reconstructor; ToCo: Token contrast for weakly-supervised semantic segmentation. segmentation, token comparison for weakly supervised semantic segmentation; VIT-B: VisionTransformer-Base, basic version of visual transformer; DUPL: Dual student with trustworthyprogressive learning for robust weakly supervisedsemantic segmentation, dual student model with trustworthy progressive learning for robust weakly supervised semantic segmentation; CLIMS: Cross Language ImageMatching for Weakly Supervised Semantic Segmentation, weakly supervised semantic segmentation with cross-language image matching;CLIP-es: CLIP is Also an Efficient Segmenter: A Text-Driven Approachfor Weakly Supervised Semantic Segmentation, CLIP is also an efficient segmenter: a text-driven approach for weakly supervised semantic segmentation; POLE: Prompting classes: exploring the power of prompt class learning in weakly supervised semantic segmentation, prompting categories: exploring the power of prompting category learning in weakly supervised semantic segmentation. ;

[0057] Table 2 Segmentation results on the COCO dataset validation set

[0058] In the table, EPS: Railroad is not a train: Saliency as pseudo-pixel supervision for weakly supervised semantic segmentation, Railway is not a train: Saliency is used as pseudo-pixel supervision for weakly supervised semantic segmentation.

[0059] It should be noted that the VOC dataset and COCO dataset (specifically the Microsoft CommonObjects in Context 2014 dataset, which is not officially available in Chinese) are used here. The PASCAL VOC 2012 dataset contains 21 categories and one background category. During the training process, an extended dataset containing 10,582 images was used. The MS COCO 2014 dataset contains 80 target categories and one background category. Among them, 82,081 images are used for training and 40,137 images are used for verification.

[0060] from Figure 4 It can be seen that the comparison of the CAM visualization results of the above method and the CLIP-es method, the first column represents the original image, the second column is the visualization result of the CAM generated by the CLIP-es method, the third column is the visualization result of the CAM generated by the above method, and the fourth column is the true label for comparison. From the first and third rows of figures, it can be seen that the activation of the two backgrounds, the rails and the ground, is significantly weakened. In the second row of figures, it can be seen that the above method strengthens the activation of the target pixels and weakens the activation of non-target pixels.

[0061] Figure 5 Schematic diagram of an embodiment of the weakly supervised semantic segmentation device disclosed in the present invention. Figure 5 The embodiment is a virtual device that can be loaded and executed by an image semantic segmentation terminal, including an acquisition module and a semantic segmentation module.

[0062] The acquisition module of the embodiment is configured to acquire image A; wherein image A is an image to be semantically segmented.

[0063] The semantic segmentation module of the embodiment is configured to perform semantic segmentation on image A using a trained semantic segmentation model; wherein the semantic segmentation model is trained using semantic segmentation pseudo labels generated by a visual language pre-training model.

[0064] In the semantic segmentation module, the process of generating semantic segmentation pseudo labels using the visual language pre-training model is as follows: using the similar word list of the foreground category name of image X and the preset context, constructing foreground auxiliary text, filtering similar words according to the similarity between the foreground auxiliary text feature and the image X feature, constructing foreground text prompts using the preset context, foreground category name and the filtered similar words, filtering common background categories according to the similarity between the common background category name feature of image X and the image X feature, constructing background text prompts using the preset context and the filtered common background category names, and obtaining pseudo labels for semantic segmentation according to the foreground text prompts and the background text prompts; wherein, image X is the image for generating semantic segmentation pseudo labels.

[0065] It should be noted that in the visual language pre-training model of the semantic segmentation module, similar words are screened according to the similarity between the foreground auxiliary text features and the image X features, and the foreground text prompt is constructed by using the preset context, the foreground category name and the screened similar words, including: using a text encoder to extract the features of the foreground auxiliary text, and using an image encoder to extract the features of the image X; calculating the similarity between the foreground auxiliary text features and the image X features; and constructing the foreground text prompt according to the preset context, the foreground category name and the similar words corresponding to the highest similarity.

[0066] It should be noted that, in the visual language pre-training model of the semantic segmentation module, common background categories are screened according to the similarity between the common background category name features of image X and the features of image X, and background text prompts are constructed using the preset context and the screened common background category names, including: extracting the features of the common background category names of image X using a text encoder, extracting the features of image X using an image encoder; calculating the similarity between the common background category name features and the features of image X; and constructing the background text prompt according to the preset context and the common background category names corresponding to the similarity not less than a threshold.

[0067] Similar to the above method, when generating a semantic segmentation pseudo label, the above device selects similar words for the foreground category name according to the similarity between the foreground auxiliary text feature and the image X feature, constructs a foreground text prompt, helps the visual language pre-training model to activate pixels, and solves the problem of insufficient activation of target pixels. According to the similarity between the common background category name feature of image X and the image X feature, the common background category name is selected, and the background text prompt is constructed, so that the visual language pre-training model is more sensitive to the co-occurring background, and the problem of co-occurring background is solved.

[0068] Based on the same technical solution, the present disclosure also relates to a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device performs a weakly supervised semantic segmentation method.

[0069] Based on the same technical solution, the present disclosure also relates to a computer device, comprising one or more processors and one or more memories, one or more programs stored in the one or more memories and configured to be executed by the one or more processors, and the one or more programs include instructions for executing a weakly supervised semantic segmentation method.

[0070] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A weakly supervised semantic segmentation method, characterized in that: include: Obtain image A; wherein image A is the image to be semantically segmented; Using the trained semantic segmentation model, perform semantic segmentation on image A; wherein the semantic segmentation model is trained using the semantic segmentation pseudo labels generated by the visual language pre-training model; The process of generating semantic segmentation pseudo labels using the visual language pre-training model is as follows: A foreground auxiliary text is constructed using a similar word list and a preset context of the foreground category name of image X, where image X is an image for generating semantic segmentation pseudo labels; According to the similarity between the foreground auxiliary text feature and the image X feature, similar words are filtered, and the foreground text prompt is constructed using the preset context, the foreground category name and the filtered similar words; According to the similarity between the common background category name features of image X and the features of image X, common background categories are screened, and background text prompts are constructed using the preset context and the screened common background category names; According to the foreground text cues and background text cues, pseudo labels for semantic segmentation are obtained.

2. The weakly supervised semantic segmentation method according to claim 1, characterized in that: According to the similarity between the foreground auxiliary text features and the image X features, similar words are filtered, and the foreground text prompt is constructed using the preset context, foreground category name and the filtered similar words, including: A text encoder is used to extract the features of the foreground auxiliary text, and an image encoder is used to extract the features of image X; Calculate the similarity between the foreground auxiliary text features and the image X features; Construct a foreground text hint based on the preset context, the foreground category name, and the similar words corresponding to the highest similarity.

3. The weakly supervised semantic segmentation method according to claim 1, characterized in that: According to the similarity between the common background category name features of image X and the features of image X, common background categories are screened, and background text prompts are constructed using the preset context and the screened common background category names, including: A text encoder is used to extract features of common background category names of image X, and an image encoder is used to extract features of image X; Calculate the similarity between the common background category name features and the image X features; Construct background text prompts based on common background category names corresponding to preset contexts and similarities not less than a threshold.

4. The weakly supervised semantic segmentation method according to claim 1, characterized in that: According to the foreground text prompt and background text prompt, the pseudo label of semantic segmentation is obtained, including: According to the foreground text prompt and the background text prompt, the gradient-based class activation mapping is used to obtain the initial class activation map; The initial class activation map is improved by using category-aware attention affinity; According to the improved class activation map, the pseudo label of semantic segmentation is obtained.

5. The weakly supervised semantic segmentation method according to claim 4, characterized in that: When using gradient-based class activation mapping, softmax is introduced into the class activation map to calculate the class probability using the softmax score Y The formula is: ； In the formula, F T To use the text prompt features extracted by the text encoder, F I is the image X feature extracted by the image encoder, is the temperature factor, and cos represents the cosine similarity.

6. A weakly supervised semantic segmentation device, characterized in that: include: An acquisition module acquires image A, wherein image A is an image to be semantically segmented; The semantic segmentation module uses a trained semantic segmentation model to perform semantic segmentation on image A; wherein the semantic segmentation model is trained using semantic segmentation pseudo labels generated by a visual language pre-training model; The process of generating semantic segmentation pseudo labels using the visual language pre-training model is as follows: A foreground auxiliary text is constructed using a similar word list and a preset context of the foreground category name of image X, where image X is an image for generating semantic segmentation pseudo labels; According to the similarity between the foreground auxiliary text feature and the image X feature, similar words are filtered, and the foreground text prompt is constructed using the preset context, the foreground category name and the filtered similar words; According to the similarity between the common background category name features of image X and the features of image X, common background categories are screened, and background text prompts are constructed using the preset context and the screened common background category names; According to the foreground text cues and background text cues, pseudo labels for semantic segmentation are obtained.

7. The weakly supervised semantic segmentation device according to claim 6, characterized in that: In the visual language pre-training model of the semantic segmentation module, similar words are filtered according to the similarity between the foreground auxiliary text features and the image X features, and the foreground text prompt is constructed using the preset context, foreground category name and the filtered similar words, including: A text encoder is used to extract the features of the foreground auxiliary text, and an image encoder is used to extract the features of image X; Calculate the similarity between the foreground auxiliary text features and the image X features; Construct a foreground text hint based on the preset context, the foreground category name, and the similar words corresponding to the highest similarity.

8. The weakly supervised semantic segmentation device according to claim 6, characterized in that: In the visual language pre-training model of the semantic segmentation module, common background categories are screened based on the common background category name features of image X and the similarity of the features of image X. The preset context and the screened common background category names are used to construct background text prompts, including: A text encoder is used to extract features of common background category names of image X, and an image encoder is used to extract features of image X; Calculate the similarity between the common background category name features and the image X features; Construct background text prompts based on common background category names corresponding to preset contexts and similarities not less than a threshold.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes any one of the methods of claims 1 to 5.

10. A computer device, characterized in that: include: One or more processors and one or more memories, one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods described in claims 1 to 5.

Images