Weak supervision semantic segmentation method based on large model auxiliary supervision

Abstract

The invention discloses a weak supervision semantic segmentation method based on large model auxiliary supervision, and belongs to the technical field of image segmentation in computer vision, and the method comprises the following steps: (1) initializing a visual Transform and a large model CLIP, and respectively extracting visual features; and (2) respectively generating class activation mapping based on the features of the visual Transform and the CLIP, and obtaining a classification result of the visual Transform. And (3) semantic affinity information is extracted based on multi-head self-attention of CLIP, an auxiliary mask is generated, segmentation prediction is acquired from a visual Transform, and auxiliary loss is calculated with the mask. And (4) performing noise detection on the auxiliary mask by using a Gaussian mixture model, and measuring distribution difference to filter noise pixels. And (5) fusing CLIP and visual Transform class activation mapping, refining the class activation mapping by using a post-processing technology to obtain a pseudo tag, and training a semantic segmentation model by using the pseudo tag. According to the method, the CLIP is introduced as auxiliary supervision information, a more reliable pseudo tag is generated, and the semantic segmentation performance is effectively improved.

Description

Technical Field

[0001] The invention discloses a weakly supervised semantic segmentation method based on large model auxiliary supervision, belonging to the technical field of image segmentation in computer vision. Background Art

[0002] Semantic segmentation is a crucial computer vision task that has been widely used in fields such as autonomous driving, medical image analysis, video surveillance, and robot vision. With the rapid development of deep neural networks and multimodal large models, many important breakthroughs have been made in the field of semantic segmentation. Traditional semantic segmentation requires manual pixel-level annotation to predict the category of each pixel in the image. Due to the time-consuming and labor-intensive pixel-level annotation, researchers have proposed weakly supervised semantic segmentation methods, which use sparse annotations such as scribbles, bounding boxes, points, and image-level labels for segmentation training, which greatly reduces the cost of pixel-level annotation. Among them, image-level labels only need to provide the category information of the image and have been widely studied because they are the easiest to obtain. The weakly supervised semantic segmentation task based on image-level labels usually follows the following three-step process: (1) First, a classification network is trained using image-level labels to generate class activation maps with location and category information; (2) Then, the class activation maps are refined into pixel-level segmentation pseudo-labels using techniques such as conditional random fields and semantic affinity; (3) Finally, the generated segmentation pseudo-labels are used as pixel-level supervision to train the semantic segmentation network.

[0003] However, there is a huge gap between the image-level supervision information provided by the class activation map and the pixel-level supervision information required for segmentation training. The main problem of weakly supervised semantic segmentation following the above process is that the class activation map tends to focus on the most discriminative areas and incorrectly activate the target category area, which leads to inaccurate segmentation pseudo-labels and provides insufficient supervision information for segmentation training. To alleviate this problem, some methods study the semantic relationship between pixels, use similarity matrices to learn the semantic association between pixels, and apply methods such as random walks to diffuse semantic information; other methods explore the erasure or mining of semantic information, using regional erasing or mining techniques to remove or mine the most discriminative areas of the class activation map to promote the activation of other target areas. Such methods generally have the problems of over-erasing and over-mining, which will disrupt the classification network's prediction of the target category. Other more direct methods use additional resources to enhance supervision information. These methods focus on effectively using additional easily accessible resources, such as saliency maps, out-of-distribution data, and cross-images to shorten the supervision gap. In recent years, large multimodal models such as CLIP and SAM have developed rapidly. Their powerful zero-shot generalization capabilities can adapt well to downstream tasks such as semantic segmentation and improve the performance of semantic segmentation. The use of large models for weakly supervised semantic segmentation has been widely studied and is challenging. Summary of the invention

[0004] In response to the above technical problems, the present invention uses the contrastive image text pair pre-training model (CLIP) as additional supervisory information, which is simple and effective. The CLIP model contains image and text encoders, which are trained on a large dataset with 400 million image-text pairs. It uses contrastive learning to align images and text in feature space, can be applied to a variety of visual tasks, and exhibits powerful zero-shot classification capabilities. CLIP provides richer visual and text information, alleviates the problem of insufficient class activation mapping supervisory information, and improves the reliability of the generated pseudo-labels.

[0005] Specifically, the present invention provides a weakly supervised semantic segmentation method assisted by a large model, comprising the following steps:

[0006] S1, initialize the visual Transformer and the large model CLIP to extract visual features respectively;

[0007] S2, generate class activation maps based on the features of visual Transformer and CLIP, and obtain the classification results of visual Transformer;

[0008] S3, CLIP-based multi-head self-attention extracts semantic affinity information, generates auxiliary masks, obtains segmentation predictions from visual Transformer, and calculates auxiliary losses with masks;

[0009] S4, using Gaussian mixture model to detect noise on auxiliary mask, measuring distribution difference to filter noise pixels;

[0010] S5. Fuse the class activation maps of CLIP and visual Transformer, use post-processing technology to refine the class activation maps, obtain pseudo labels, and use pseudo labels to train the semantic segmentation model.

[0011] Furthermore, in step S1, for the visual Transformer, the ImageNet pre-trained model is used to initialize the network parameters; for CLIP, the ViT-B pre-trained model is used. H×W×3 and image-level labels y∈R 1×C Input into the visual Transformer encoder and CLIP model at the same time, extract the category label T c ∈R C×D , patch mark T p ∈R N×D and CLIP visual features F c ∈R H×W×D , label the patch T p Rearrange and convolve to get feature F v ∈R (1+C)×H×W, the process is as follows:

[0012] F v =Conv(reshape(T p ∈R N×D ))

[0013] Fv can be used as segmentation prediction, including foreground features F fg ∈R C×H×W and background features F bg ∈R 1×H×W H and W are the height and width of the image, respectively, N = H × W, C is the number of categories, and D is the feature dimension.

[0014] Furthermore, in step S2, for the visual Transformer, the foreground feature F extracted in step S1 is fg After the ReLU function, the initial activation map M is obtained v :

[0015] M v =ReLU(F v [1:,:,:])

[0016] Use CLIP as an auxiliary network to extract the feature F of the last layer c ∈R H×W×D , multiply the weight of category c on the kth channel of the feature by the feature and then pass it through the ReLU function to get the initial class activation map M c :

[0017]

[0018] Compute the classification results of category labels and patch labels in the visual Transformer. For the category label T c , perform mean processing to obtain the predicted probability p of category c c ; For patch mark T p , use global average pooling to get the category prediction probability p p , then p c and p p Calculate the multi-label classification loss and get L cls_1 and L cls_2 , the calculation formula is as follows, where σ(·) is the sigmoid function:

[0019]

[0020]

[0021] Furthermore, in step S3, the attention weight W∈R is obtained based on the multi-head attention mechanism in CLIP. N×N, regularize W to get the matrix R, which represents the optimal mapping relationship of semantic information, and linearly add the matrix R to get the symmetric semantic affinity matrix A. T(·) represents the transposition operation. Then the matrix A is randomly walked to propagate the semantic affinity information to improve the M generated in step S2. c get right The maximum activation value of each pixel in is negated and raised to the power to obtain the background vector M bg ,and Splice and then use the argmax operation to get the category index of each pixel, convert the index into the category label to get the final auxiliary mask Y aux , the specific process is as follows:

[0022]

[0023]

[0024] The feature F containing background information v Used as segmentation prediction, the auxiliary mask Y is obtained using cross entropy loss aux and segmentation prediction F v The auxiliary loss, as additional supervision information, is formulated as follows:

[0025]

[0026] Further, in step S4, a Gaussian mixture model is used to detect noise on the auxiliary mask, and the distribution difference is measured to filter the noise pixels. A Gaussian mixture model containing two Gaussian distributions is established to fit the pixels with too high auxiliary loss values, which is implemented using a Gaussian probability density function, and the formula is as follows:

[0027]

[0028] where σ 2 is the variance, μ is the mean value, and the weight coefficient α of Gaussian mixture distribution is used k Combining the two distributions, the formula is as follows:

[0029]

[0030] Initialize the parameters of the Gaussian mixture model and use the expectation maximization algorithm to estimate the distribution. When the mean difference between the two distributions is greater than the set threshold, the pixel is considered to have noise. In the expectation step, the probability γ that the pixel belongs to noise is calculated. k (x), the pixels with high noise probability are recorded as 255, indicating that this noise pixel is ignored and does not participate in the subsequent training process. Then use γ in the maximization step k (x) to update the parameters μ of all Gaussian distributionsk , and α k , the process is as follows:

[0031]

[0032]

[0033] By repeating the expectation step and the maximization step, the modeling is terminated when the parameter changes of the two distributions are less than the preset value or the maximum number of iterations is reached.

[0034] Further, in step S5, the improved class activation map generated by CLIP is used As guidance information, the class activation map M generated by the visual Transformer v Add the weights, using β as the weight coefficient, to get the final class activation map M final , the process is as follows:

[0035]

[0036] The semantic information is propagated through post-processing technology to obtain the final segmentation pseudo-label, and the pseudo-label is used to train the segmentation network to complete the semantic segmentation of the image.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention uses a large model to provide auxiliary supervision, effectively solving the problem of supervision differences in weakly supervised semantic segmentation. The additional information generated by the frozen CLIP model can help the backbone network obtain high-quality pixel-level pseudo-labels and provide more accurate supervision for segmentation training. Secondly, the class activation map fusion and mask denoising are designed to improve the quality of class activation maps, further enhance auxiliary supervision, and significantly improve segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a weakly supervised semantic segmentation method based on large model assisted supervision according to the present invention;

[0040] Figure 2 Schematic diagram of a weakly supervised semantic segmentation method based on large model assisted supervision in an embodiment of the present invention;

[0041] Figure 3 It is the average intersection-over-union result of the class activation map and pseudo label on the PASCAL VOC 2012 training set of the present invention;

[0042] Figure 4 This is the visualization result of the pseudo-label generated by the present invention on the PASCAL VOC 2012 training set. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this specification clearer, the technical solutions in the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Figure 1 A flowchart of a weakly supervised semantic segmentation method based on large model assisted supervision provided by the present invention, Figure 2 The schematic diagram is a structural diagram of a weakly supervised semantic segmentation method based on large model auxiliary supervision in an embodiment of the present invention, which mainly includes auxiliary mask generation, auxiliary mask denoising and class activation map fusion, and specifically includes the following steps:

[0045] S1. Initialize the visual Transformer and the large model CLIP to extract visual features respectively.

[0046] The present invention includes two networks, Visual Transformer and CLIP. Visual Transformer uses the ImageNet pre-trained model to initialize network parameters, and CLIP uses the ViT-B pre-trained model. First, data enhancement is performed on the image, and the image is uniformly cropped to a size of 448×448. Then, the image I∈R H×W×3 and image-level labels y∈R 1×C The two models are input into the visual Transformer encoder and the frozen CLIP model for parallel training, respectively extracting the visual features F of CLIP. c ∈R H ×W×D , category label T of visual Transformer c ∈R C×D and patch marker T p ∈R N×D (N = H × W), H and W are the height and width of the image, respectively, and D is the feature dimension. The patch labels are rearranged and convolved to obtain the feature F v ∈R (1+C)×H×W , the formula is as follows:

[0047] F v =Conv(reshape(T p ∈R N×D ))

[0048] F v Contains foreground features F fg ∈R C×H×Wand background features F bg ∈R 1×H×W ,Since Fv contains the background, it also represents the segmentation prediction for subsequent additional supervision.

[0049] S2. Generate class activation maps based on the features of visual Transformer and CLIP, and obtain the classification results of visual Transformer.

[0050] The visual Transformer is used as the main network to provide classification supervision, and the CLIP is used as an auxiliary network to provide additional supervision for segmentation training. The class activation map is obtained using the features of the visual Transformer, and the F v The foreground features are passed through the ReLU function to obtain the initial activation map M v :

[0051] M v =ReLU(F v [1:,:,:])

[0052] Extract the last layer feature F from the CLIP network c ∈R H×W×D , multiply the weight of category c on the kth channel of the feature by the feature and then pass it through the ReLU function to get the initial class activation map M c :

[0053]

[0054] In order to obtain the classification score of the network, the category label T c Perform mean processing to obtain the predicted probability p of category c c ; Mark the patch with T p Perform global average pooling to obtain the category prediction probability p p , then p c and p p Calculate the multi-label classification loss and get L cls_1 and L cls_2 , the calculation formula is as follows, where σ(·) is the sigmoid function:

[0055]

[0056]

[0057] S3. CLIP-based multi-head self-attention extracts semantic affinity information, generates auxiliary masks, obtains segmentation predictions from the visual Transformer, and calculates auxiliary losses with the masks.

[0058] Based on the class activation map M generated by CLIP in step S2c , M c It inherits the advantages of CLIP, has strong positioning capabilities, and can generate effective auxiliary supervision. To generate auxiliary supervision information, namely auxiliary masks, the attention weight W∈R is obtained from the multi-head attention mechanism in CLIP. N×N , regularize W to get the matrix R, which represents the optimal mapping relationship of semantic information. Linear addition is performed to get the symmetric semantic affinity matrix A:

[0059]

[0060] The above formula T(·) represents the transposition operation, which performs random walks on the matrix A to propagate semantic affinity information to improve the class activation map M generated by CLIP in the above c , get the improved class activation map Such a semantic propagation process effectively suppresses the wrongly activated areas and makes the class activation map better aligned with the semantic boundary. The maximum activation value of each pixel in is negated and then the power is calculated to obtain the background vector M bg ,and After splicing, use the argmax operation to get the category index of each pixel, and convert the index into the category label to get the final auxiliary mask Y aux , the specific process is as follows:

[0061]

[0062]

[0063] Since the incomplete pseudo-labels come from the initial class activation map, it is essentially an inherent defect of using image-level labels to train pixel-level segmentation networks. The auxiliary mask generated by CLIP contains more accurate pixel-level information, suppresses the background area well, and can provide additional information. Based on this, the feature F containing background information is v Used as segmentation prediction, using the auxiliary mask Y generated above aux Calculate the cross entropy loss as the auxiliary loss, the formula is as follows:

[0064]

[0065] The loss function is used to update the weights during the entire training process, and the three losses L cls_1 , L cls_2 and L aux The weights are added to complete the network update, where the auxiliary loss L aux The strong pixel-level supervision information provided enhances the network's ability to locate target category areas.

[0066] S4. Use Gaussian mixture model to detect noise on auxiliary mask and measure distribution difference to filter noise pixels.

[0067] The auxiliary mask generated in step S3 inevitably contains noise information, which will reduce the supervision ability of the auxiliary information. The auxiliary loss corresponding to the auxiliary mask with noise is relatively high. For this reason, the auxiliary loss L can be directly used. aux As a noise detection condition, it is assumed that the auxiliary loss satisfies the Gaussian distribution. The specific steps of S4 are as follows:

[0068] A1. First, use the Gaussian probability density function to establish a Gaussian mixture model containing two Gaussian distributions, fit the pixels with too high auxiliary loss values, and then use the weight coefficient α of the Gaussian mixture distribution k Combining two distributions, the specific process is as follows, where σ 2 is the variance and μ is the mean value:

[0069]

[0070]

[0071] A2. Then initialize the parameters of the Gaussian mixture model and use the expectation maximization algorithm to estimate the distribution. When the mean difference between the two distributions is greater than the set threshold, the pixel is considered to have noise. In the expectation step, the probability γ that the pixel belongs to noise is calculated. k (x), the pixels with high noise probability are recorded as 255, indicating that this noise pixel is ignored and does not participate in the subsequent training process. Then use γ in the maximization step k (x) to update the parameters μ of all Gaussian distributions k , and α k , the process is as follows:

[0072]

[0073]

[0074] A3. Finally, by repeating the expectation step and the maximization step, the modeling is terminated when the parameter change of the two distributions is less than the preset value or the maximum number of iterations is reached. By reducing the noise information, the network training can be smoother, which effectively enhances the auxiliary supervision information provided by CLIP.

[0075] S5. Fuse the class activation maps of CLIP and visual Transformer, use post-processing technology to refine the class activation maps, obtain pseudo labels, and use pseudo labels to train the semantic segmentation model.

[0076] CLIP can not only provide high-quality auxiliary masks, but also the generated class activation maps have certain guiding significance due to CLIP's remarkable positioning ability for target category areas. However, due to the freezing of the CLIP model, directly using the generated class activation maps to participate in subsequent segmentation tasks cannot adapt well to the dynamically updated segmentation network. To this end, the present invention uses CLIP's improved class activation map as guidance information, and weights it with the class activation map generated by the visual Transformer main network, using CLIP's positioning knowledge as a guide to improve the positioning ability of the main network. A weight coefficient β is introduced to improve the positioning ability of the main network. and the class activation map M of the main network v Add together to get the final class activation map M used to generate pseudo labels final , the process is as follows:

[0077]

[0078] The semantic information is propagated through post-processing technology to obtain the final segmentation pseudo-label, and the pseudo-label is used to train the segmentation network to complete the semantic segmentation of the image.

[0079] Figure 3 This is the mean intersection over union (mIoU) result of the class activation map and class activation map seeds on the PASCAL VOC 2012 training set. Compared with other methods, the class activation map generated by the present invention has a more complete and correct target category area, reduces erroneous activations, and generates more complete pseudo labels, thereby improving the performance of weakly supervised semantic segmentation.

[0080] Figure 4 This is the visualization result of the pseudo-labels generated by the present invention on the PASCAL VOC 2012 training set and another weakly supervised semantic segmentation method. From the visualization diagram, it can be seen that the pseudo-labels generated by the present invention have a more complete area and a clearer boundary. Intuitively, the outline of the target category is clearer.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit them. The present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A weakly supervised semantic segmentation method based on large model assisted supervision, characterized in that: The following steps are involved: S1, initialize the visual Transformer and the large model CLIP to extract visual features respectively; S2, generating class activation maps based on the features of the visual Transformer and CLIP, and obtaining the classification results of the visual Transformer; S3, CLIP-based multi-head self-attention extracts semantic affinity information, generates auxiliary masks, obtains segmentation predictions from visual Transformer, and calculates auxiliary losses with masks; S4, using Gaussian mixture model to detect noise on auxiliary mask, measuring distribution difference to filter noise pixels; S5. Fuse the class activation maps of CLIP and visual Transformer, use post-processing technology to refine the class activation maps, obtain pseudo labels, and use pseudo labels to train the semantic segmentation model.

2. According to claim 1, a weakly supervised semantic segmentation method based on large model assisted supervision is characterized in that: In step S1, for the visual Transformer, the ImageNet pre-trained model is used to initialize the network parameters; for CLIP, the ViT-B pre-trained model is used to transform the image and image-level labels y∈R 1 × C Input into the visual Transformer encoder and CLIP model at the same time, extract the category label T c ∈R C×D , patch mark T p ∈R N×D and CLIP visual features F c ∈R H×W×D , label the patch T p Rearrange and convolve to get feature F v ∈R (1+C)×H×W , the process is as follows: F v =Conv(reshape(T p ∈R N×D )) Fv can be used as segmentation prediction, including foreground features F fg ∈R C×H×W and background features F bg ∈R 1×H×W , H and W are the height and width of the image respectively, N = H × W, C is the number of categories, and D is the feature dimension.

3. According to claim 2, a weakly supervised semantic segmentation method based on large model auxiliary supervision is characterized in that: In step S2, for the visual Transformer, the foreground feature F extracted in step S1 is fg After the ReLU function, the initial activation map M is obtained v : M v =ReLU(F v [1:,:,:]) Use CLIP as an auxiliary network to extract the feature F of the last layer c ∈R H×W×D , multiply the weight of category c on the kth channel of the feature by the feature and then pass it through the ReLU function to get the initial class activation map M c : Compute the classification results of category labels and patch labels in the visual Transformer. For the category label T c , perform mean processing to obtain the predicted probability p of category c c ; For patch mark T p , use global average pooling to get the category prediction probability p p , then p c and p p Calculate the multi-label classification loss and get L cls_1 and L cls_2 , the calculation formula is as follows, where σ(·) is the sigmoid function:

4. According to claim 3, a weakly supervised semantic segmentation method based on large model assisted supervision is characterized in that: In step S3, the attention weight W∈R is obtained based on the multi-head attention mechanism in CLIP N × N , regularize W to get the matrix R, which represents the optimal mapping relationship of semantic information, and linearly add the matrix R to get the symmetric semantic affinity matrix A. T(·) represents the transposition operation, and then the matrix A is randomly walked to propagate the semantic affinity information to improve the M generated in step S2 c get right The maximum activation value of each pixel in is negated and raised to the power to obtain the background vector M bg ,and Splice and then use the argmax operation to get the category index of each pixel, convert the index into the category label to get the final auxiliary mask Y aux , the specific process is as follows: The feature F containing background information v Used as segmentation prediction, the auxiliary mask Y is obtained using cross entropy loss aux and segmentation prediction F v The auxiliary loss, as additional supervision information, is formulated as follows:

5. According to claim 4, a weakly supervised semantic segmentation method based on large model assisted supervision is characterized in that: In step S4, a Gaussian mixture model is used to detect noise on the auxiliary mask, measure the distribution difference to filter the noise pixels, and establish a Gaussian mixture model containing two Gaussian distributions to fit the pixels with too high auxiliary loss values. This is achieved using a Gaussian probability density function, and the formula is as follows: where σ 2 is the variance, μ is the mean value, and the weight coefficient α of Gaussian mixture distribution is used k Combining the two distributions, the specific formula is as follows: Initialize the parameters of the Gaussian mixture model and use the expectation maximization algorithm to estimate the distribution. When the mean difference between the two distributions is greater than the set threshold, the pixel is considered to have noise. In the expectation step, the probability γ that the pixel belongs to noise is calculated. k (x), the pixels with high noise probability are recorded as 255, indicating that this noise pixel is ignored and does not participate in the subsequent training process, and then γ is used in the maximization step k (x) to update the parameters μ of all Gaussian distributions k , σ k 2 and α k , the process is as follows: By repeating the expectation step and the maximization step, the modeling is terminated when the parameter changes of the two distributions are less than the preset value or the maximum number of iterations is reached.

6. The weakly supervised semantic segmentation method based on large model assisted supervision according to claim 5 is characterized in that: In step S5, the improved class activation map generated by CLIP is used As guidance information, the class activation map M generated by the visual Transformer v Add the weights, using β as the weight coefficient, to get the final class activation map M final , the process is as follows: The final class activation map is refined through post-processing technology to propagate semantic information, and the final segmentation pseudo-label is obtained. The pseudo-label is used to train the segmentation network to complete the semantic segmentation of the image.

Images