A general and efficient semantic segmentation method without training based on iterative optimization

Abstract

The present invention relates to a general and efficient semantic segmentation method without training based on iterative optimization. The pre-trained diffusion model is improved. After extracting and selecting the cross attention map and the self attention map, a cross attention map A of the same scale is obtained. ca and the self-attention map A sa , the self-attention map A sa It is sent to the entropy reduction self-attention module to remove irrelevant global information. The method is as follows: calculate the entropy gradient of the self-attention map, calculate the self-attention entropy by considering the distribution of the self-attention map as the probability mass function of a discrete random variable, and then calculate the entropy gradient; update the self-attention: calculate the self-attention map A after entropy reduction by updating the selected self-attention map along the negative gradient direction c e a nt ; Self-attention map A after entropy reduction s e a nt and the cross attention map A ca Perform matrix multiplication, iteratively optimize the cross attention map, and obtain the final cross attention A ca Restore the image size to its original size.

Description

Technical Field

[0001] The present invention relates to a computer vision semantic segmentation method based on a Transformer architecture in an intelligent system (such as intelligent driving, intelligent monitoring, intelligent interaction, etc.), and in particular to a method for semantic segmentation using prior knowledge and self-attention maps in a diffusion model. Background Art

[0002] Semantic segmentation aims to distinguish pixels into different semantic groups. It is a basic task in the field of computer vision and has a wide range of application scenarios. For example, in the scenario of autonomous driving, the car uses a camera to capture the surrounding scene in real time, and uses semantic segmentation technology to determine the predicted category of each pixel, so as to avoid other cars and pedestrians around, or identify lane lines to determine the driving direction; for example, semantic segmentation of different organs of the human body can assist doctors in determining possible lesions in medical images.

[0003] In the past decade, fully supervised semantic segmentation has made significant progress. Among them, FCN, as a pioneering method, introduced convolutional neural networks with cross-layer connections. Based on this, researchers designed many variants such as multi-layer feature fusion [1], spatial pyramid structure [2], and non-local enhancement [3] to innovate. On the other hand, with the success of VIT, researchers began to explore VIT-based methods. For example, SegFormer [4] focuses on leveraging the powerful representation ability of Transformer as the backbone. In addition, Mask2Former regards semantic segmentation as a set of set prediction tasks and uses Transformer as a decoder.

[0004] Due to the fact that fully supervised semantic segmentation requires training on large-scale semantic segmentation datasets that are difficult to obtain, and this fully supervised setting hinders the zero-shot segmentation capability of unseen categories, some works have explored using the powerful zero-shot capability of CLIP for semantic segmentation. For example, OVSeg[6] directly uses the text-image mapping learned in the pre-trained CLIP, uses a mask generator to extract some category-independent masks, and then classifies its cropped and masked images to achieve open vocabulary semantic segmentation. Other works such as GroupViT[7] introduced a hierarchical grouping visual transformer with text supervision, which can group different semantic regions without pixel-level annotation. These methods improve the generalization ability of semantic segmentation and reduce the dependence on pixel-level annotation to a certain extent.

[0005] Recently, researchers have begun to adopt diffusion models for segmentation. Diffusion models were originally designed to perform high-quality image synthesis. Therefore, researchers believe that diffusion models contain rich semantic clues and have proposed various diffusion-based segmentation methods. For example, VPD[8] uses stable diffusion as a feature extractor and retrains it on a segmentation dataset. Other methods directly explore the untrained framework by leveraging self-attention maps and cross-attention maps in stable diffusion. DiffSeg[9] iteratively merges self-attention maps at different locations and uses non-maximum suppression to assign pixels to different targets. DiffuMask

[10] uses cross-attention maps to generate initial object masks for synthesized images. DiffSegmenter

[11] uses self-attention maps to refine cross-attention maps to improve segmentation. These methods are preliminary attempts to use self-attention maps and cross-attention maps for untrained segmentation tasks and have achieved satisfactory results.

[0006] References:

[0007] [1]Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. SegNet: Adeep convolutional encoder-decoder architecture for image segmentation. IEEETransactions on PatternAnalysis and Machine Intelligence, 39:2481–2495, 2017.

[0008] [2] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, and etal. DeepLab: Semantic image segmentation with deep convolutional nets, atrousconvolution, and fully connected CRFs. IEEE Transactions on PatternAnalysis andMachine Intelligence, 40:834–848, 2018.

[0009] [3]Yuhui Yuan,Lang Huang,Jianyuan Guo,and et al.OCNet:Object contextfor semantic segmentation.International Journal of Computer Vision,129:2375–2398,2021.

[0010] [4]Enze Xie,Wenhai Wang,Zhiding Yu,and et al.SegFormer:Simple andefficient design for semantic segmentation with transformers.In Conference onNeural Information Processing Systems,pages 12077–12090,2021.

[0011] [5]Bowen Cheng,Ishan Misra,Alexander G Schwing,and et al.Masked-attention mask transformer for universal image segmentation.In IEEE / CVFConference on ComputerVision and Pattern Recognition,pages 1290–1299,2022.

[0012] [6]Feng Liang,Bichen Wu,Xiaoliang Dai,and et al.Open-vocabularysemantic segmentation with mask adapted CLIP.In IEEE / CVF Conference onComputer Vision and Pattern Recognition,pages 7061–7070,2023.

[0013] [7]Jiarui Xu,Shalini De Mello,Sifei Liu,and et al.GroupViT:Semanticsegmentation emerges from text supervision.In IEEE / CVF Conference on ComputerVision and Pattern Recognition,pages 18134–18144,2022.

[0014] [8]Wenliang Zhao,Yongming Rao,Zuyan Liu,and et al.Unleashing text-to-image diffusion models for visual perception.In IEEE / CVF InternationalConference on ComputerVision,pages 5706–5716,2023.

[0015] [9]Junjiao Tian,LavishaAggarwal,Andrea Colaco,and et al.Diffuse,attend,and segment:Unsupervised zero-shot segmentation usingstablediffusion.arXiv preprint arXiv:2308.12469,2023.

[0016]

[10] Weijia Wu,Yuzhong Zhao,Mike Zheng Shou,and et al.DiffuMask:Synthesizing images with pixel-level annotations for semantic segmentationusing diffusion models.In IEEE / CVF International Conference on ComputerVision,pages1206–1217,2023.

[0017]

[11] Jinglong Wang,Xiawei Li,Jing Zhang,and et al.Diffusion model issecretly a training-free open vocabulary semantic segmenter.arXiv preprintarXiv:2309.02773,2023. Summary of the invention

[0018] The present invention provides a method for semantic segmentation by directly using a pre-trained diffusion model. The semantic categories of an input image can be segmented through interactive methods such as text. The present invention can perform segmentation in any natural image without training and has certain openness and generalization capabilities. The technical solution is as follows:

[0019] A general and efficient semantic segmentation method without training based on iterative optimization is proposed to improve the pre-trained diffusion model. The method is characterized in that after extracting and selecting the cross-attention map and the self-attention map, a cross-attention map A of the same scale is obtained. ca and the self-attention map A sa , the self-attention map A sa It is sent to the entropy reduction self-attention module to remove irrelevant global information. The method is:

[0020] (1) Calculate the entropy gradient of the self-attention map. The self-attention entropy is calculated by considering the distribution of the self-attention map as the probability mass function of a discrete random variable, and then the entropy gradient is calculated.

[0021] (2) Update self-attention: Calculate the self-attention map with reduced entropy by updating the selected self-attention map along the negative gradient direction

[0022] (3) Self-attention map after entropy reduction and the cross attention map A ca Perform matrix multiplication to optimize the criss-cross attention map until the number of optimizations reaches a predefined N times:

[0023] (4) The final cross attention A ca Restore the image size to its original size.

[0024] Furthermore, the method of step (1) is as follows:

[0025] The calculation method of the self-attention entropy is: H and W are the height and width of the self-attention map, respectively. For all elements, When the self-attention entropy A sa The value of is the largest, and when A saIts value is minimum when there is a response only at the true label, for a given element in the self-attention map Calculate the entropy gradient:

[0026] Furthermore, in step (2), the calculation formula for calculating the self-attention map after entropy reduction by updating the selected self-attention map along the negative gradient direction is: Where λ represents the update factor, which is a hyperparameter. The self-attention graph after entropy reduction is expressed as

[0027] Furthermore, the method of step (3) is as follows:

[0028] (a) Put A ca Expand to HW×1, through matrix multiplication Optimize

[0029] (b) New cross-attention map obtained by normalization optimization Make its range between [0,1];

[0030] (c) Loop through (a) and (b) until N times are reached, then exit the loop.

[0031] Since the present invention directly uses the pre-trained diffusion model for semantic segmentation, the proposed network does not have any parameter learning and therefore does not require training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Overall framework

[0033] Figure 2 Detailed process of the proposed method

[0034] Figure 3 Schematic diagram of segmentation results DETAILED DESCRIPTION

[0035] The present invention is described below in conjunction with the accompanying drawings and embodiments.

[0036] The overall framework process of the present invention is as follows Figure 1 As shown in the flow chart, Figure 2 shown.

[0037] The cross attention in the diffusion model can often reflect the approximate location of the image that matches the text. Therefore, optimizing the cross attention map to make it more accurate is a feasible segmentation method. However, the existing method only uses the self-attention map to optimize the cross attention map. Figure 1 This is because direct multiple optimization will cause drastic changes in the cross-attention map, such as Figure 3As shown, the attention of the segmented bird object has changed (0 represents the initial cross-attention; 1, 2, 5, 10 represent the results after the corresponding number of optimizations). The present invention proposes that a more accurate cross-attention map can be obtained through iterative optimization, and the entropy reduction self-attention module is used to prevent the cross-attention map from changing dramatically during the optimization process. Figure 2 A brief description of the specific implementation of the method proposed in the present invention is given, and the specific steps are as follows:

[0038] Step 1: Set relevant hyperparameters including update factor λ and number of iterations N; load the pre-trained diffusion model, which consists of three parts: VAE encoder, CLIP text encoder and denoising network; select appropriate prompt methods, including points, lines, boxes and text.

[0039] Step 2: Read an image and normalize its size to 512×512, and its value is normalized to [-1,1]. The normalized image is sent to the VAE encoder to obtain an image feature z with a size of 64×64.

[0040] Step 3: For text prompts, the text matching the image (such as: "a photo of cat.") is fed into the CLIP text encoder; for dotted lines, the empty text (such as: "") is fed into the CLIP text encoder, and the CLIP text encoder outputs the text feature ε.

[0041] Step 4: Feed the image feature z and text feature ε into the denoising network and extract the attention map.

[0042] a) Extract the attention maps of all layers. The denoising network in the diffusion model is a U-net network that incorporates the Transformer structure. The U-net network has a four-layer structure with a total of four sizes: 8×8, 16×16, 32×32, and 64×64. Each layer of the structure embeds the Transformer structure, so the extracted cross-attention map and self-attention map also have the characteristics of these four sizes.

[0043] b) Selecting the attention map. In the denoising network, semantic information is mainly present in the two scales of 16×16 and 32×32. Therefore, the present invention selects the cross attention maps of these two scales, and upsamples the cross attention maps of these two layers to 64×64 and averages them into a cross attention map A. ca For self-attention, the larger the scale of the self-attention, the higher the accuracy. Therefore, the present invention selects the 64×64 self-attention map A. sa .

[0044] c) For the dotted line frame prompt, the spatial position map corresponding to the dotted line frame is directly used as the cross attention map to replace the cross attention map extracted from the denoising network. This is because when the dotted line frame prompt is used, the input text is empty text, and the extracted cross attention has no meaning. However, the dotted line frame prompt has the characteristics of the cross attention map, which roughly indicates the location of the object to be segmented, so it can be directly replaced.

[0045] Step 5: Send the self-attention map into the entropy reduction self-attention module to remove irrelevant global information.

[0046] a) Calculate the entropy gradient of the self-attention map. The self-attention entropy is calculated by considering the distribution of the self-attention map as the probability mass function of a discrete random variable, which can be written as H and W are the height and width of the self-attention map respectively. For all elements, When the self-attention entropy A sa The value of is the largest, and when A sa Its value is minimum when there is response only at the true label. This means that the higher the self-attention entropy, the more global information there is, and therefore the greater the possibility of irrelevant information. In order to reduce irrelevant information, the self-attention entropy needs to be reduced. For a given element in the self-attention map Its entropy gradient can be calculated as

[0047] b) Update self-attention. The self-attention map with reduced entropy can be calculated by updating the original self-attention map along the negative gradient direction: Where λ represents the update factor, which is a hyperparameter and can be freely specified. The final self-attention map is expressed as The entropy reduction operation makes The value close to 1 becomes larger, making The value close to 0 becomes smaller. Therefore, the entropy reduction operation can suppress the weak response corresponding to irrelevant global information in the self-attention map while maintaining the strong response.

[0048] Step 6: The self-attention map and the cross attention map A ca Perform matrix multiplication to optimize the cross attention map until the number of optimizations reaches a predefined N times.

[0049] a) The size of is HW×HW, which represents the similarity of each feature pixel to other pixels. ca The size of A is H×W, which represents the approximate location where the segmentation is required. ca Expand to HW×1, through matrix multiplication Make optimizations.

[0050] b) New cross attention map obtained by normalization optimization Make its range between [0,1].

[0051] c) Loop through (a) and (b) until N times are reached, then exit the loop.

[0052] Step 7: The final cross attention A ca Restore the image size to its original size. Figure 3 This is a schematic diagram of the segmentation result.

[0053] Step 8: Result comparison. The present invention achieves better results by making full use of self-attention to optimize cross-attention for segmentation, such as Figure 3 shown.

Claims

1. A general and efficient semantic segmentation method without training based on iterative optimization, which improves the pre-trained diffusion model, characterized in that: After extracting and selecting the cross-attention map and the self-attention map, we obtain the cross-attention map A of the same scale. ca and self-attention map A sa , the self-attention map A sa It is sent to the entropy reduction self-attention module to remove irrelevant global information. The method is: (1) Calculate the entropy gradient of the self-attention map, calculate the self-attention entropy by considering the distribution of the self-attention map as a probability mass function of a discrete random variable, and then calculate the entropy gradient; the calculation method of the self-attention entropy is: H and W are the height and width of the self-attention map, respectively. For all elements, When the self-attention entropy A sa The value of is the largest, and when A sa Its value is minimum when there is a response only at the true label, for a given element in the self-attention map Calculate the entropy gradient: (2) Update self-attention: Calculate the self-attention map with reduced entropy by updating the selected self-attention map along the negative gradient direction (3) Self-attention map after entropy reduction and the cross attention map A ca Perform matrix multiplication to optimize the criss-cross attention map until the number of optimizations reaches a predefined N times, as follows: (a) Put A ca Expand to HW×1, through matrix multiplication Optimize (b) New cross-attention map obtained by normalization optimization Make its range between [0,1]; (c) Loop (a) and (b) until N times are reached, then exit the loop; (4) The final cross attention A ca Restore the image size to its original size.

2. The non-training general and efficient semantic segmentation method according to claim 1, characterized in that: In step (2), the formula for calculating the self-attention map after entropy reduction by updating the selected self-attention map along the negative gradient direction is: Where λ represents the update factor, which is a hyperparameter. The self-attention graph after entropy reduction is expressed as

Images