Open vocabulary semantic segmentation method based on image block sensitive agent attention network

Abstract

The invention provides an open vocabulary semantic segmentation method based on an image block sensitive proxy attention network, which comprises the following steps of: constructing the image block sensitive proxy attention network which comprises a text encoder and two image encoders, the two image encoders are respectively a pre-trained CLIP image encoder and a visual basic model VFM encoder; for a given image, semantic annotation information corresponding to the image is given, the semantic annotation information comprises a specific category to which an object belongs and a Ground Truth graph of an object mask, and image features are respectively extracted by using a CLIP image encoder and a visual basic model VFM encoder of a backbone network; image block sensitive agent attention processing is carried out; carrying out adaptive normalization processing on the image blocks; performing mask processing to obtain an attention map; and segmenting the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, such as autonomous driving, intelligent robots, and the like, and in particular to a method for open vocabulary semantic segmentation based on deep learning. BACKGROUND

[0002] Semantic segmentation is a fundamental task in computer vision, which aims to divide pixels in an image into different categories according to their semantic content. In the early stage, researchers assumed that the semantic categories in the training and inference stages were fixed [1], which greatly limited the application of this technology in real-world scenarios. To solve this limitation, the research community began to focus on open vocabulary semantic segmentation [2, 3], which refers to the ability of a model to handle classes that have not been seen in training. The open vocabulary semantic segmentation task aims to perform pixel-level segmentation on any semantic category that appears in the inference process. It is a novel segmentation task that refers to the ability of a model to segment object categories that have not been seen in the training set.

[0003] In recent years, open vocabulary semantic segmentation has been widely applied in many computer vision fields, such as autonomous driving and traffic scene understanding, intelligent robots and human-computer interaction, image retrieval, medical image analysis, and other computer vision fields. Through deep learning-based open vocabulary semantic segmentation technology, it can understand the vast number of objects in its environment, and these objects are not all pre-recorded in its pre-training system. Open vocabulary segmentation technology can locate previously unseen objects. Figure 1 Figures 1(a) and 1(b) show examples of open semantic segmentation used for autonomous driving and animal detection, respectively. As shown in Figure 1 (a), through deep learning-based open vocabulary semantic segmentation technology, the image detected by the dashcam is segmented to achieve accurate target detection and recognition, enabling automatic avoidance of people and vehicles [4]. As shown in Figure 1 (b), previously unseen animals can be effectively segmented, which can be used for animal reconnaissance in nature reserves to find wild animals, or for detection in military areas to avoid collisions with animals that can cause danger [5, 6].

[0004] Traditional semantic segmentation has made great progress in the past decade. Initially, researchers began to explore semantic segmentation using convolutional neural networks (CNNs) as CNNs achieved great success in image classification tasks [1]. For example, DeepLab [7] uses cross-connection fusion or spatial pyramid structures to extract contextual information to generate high-quality segmentation maps. Other works focus on designing lightweight structures to achieve efficient segmentation [8, 9], and some studies use inter-image enhancement strategies

[10] to improve segmentation performance. With the advent of Vision Transformers (ViTs)

[11] , researchers have turned to exploring Transformer architectures [12, 13]. One direct approach is to replace CNNs with ViTs as backbone networks [14, 15]. At the same time, inspired by the set prediction idea in Transformers, some methods treat semantic segmentation as a semantic set prediction task [16, 17], which usually uses a set of learnable queries to represent different classes or objects and predicts masks corresponding to these queries.

[0005] In the field of open vocabulary semantic segmentation, the success of the Contrastive Language-Image Pre-training (CLIP) model

[18] has played a key role in promoting. CLIP aligns features between an image encoder and a text encoder, showing strong generalization ability on the zero-shot image classification task. Inspired by this, researchers have begun to explore how to adapt CLIP to open vocabulary semantic segmentation, mainly divided into training-based [19, 20] and training-free [3, 21] methods. Training-free methods have attracted much attention due to their simplicity. The core idea of this class of methods is to improve the quality of spatial representations of the original attention map in the last layer of CLIP. For example, SCLIP [3] and ClearCLIP

[21] replace the original query-to-key attention map with a query-to-query or key-to-key self-attention map.

[0006] References:

[0007] [1] Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation [C] / / Conference on Computer Vision and Pattern Recognition. 2015: 3431-3440.

[0008] [2] Tu Z, Ding Z, Wang J. Open-vocabulary universal image segmentation with maskclip [C] / / International Conference on Machine Learning. 2023.

[0009] [3] Wang F, Mei J, Yuille A. Sclip: Rethinking self-attention for dense vision-language inference [C] / / European Conference on Computer Vision. 2024: 315-332.

[0010] [4] Cordts M, Omran M, Ramos S, et al. The cityscapes dataset for semantic urban scene understanding [C] / / Conference on Computer Vision and Pattern Recognition. 2016: 3213-3223.

[0011] [5] Mottaghi R, Chen X, Liu X, et al. The role of context for object detection and semantic segmentation in the wild [C] / / Conference on Computer Vision and Pattern Recognition. 2014: 891-898.

[0012] [6] Lin T Y, Maire M, Belongie S, et al. Microsoft coco: Common objects in context [C] / / European Conference on Computer Vision. 2014: 740-755.

[0013] [7] Chen L C, Papandreou G, Schroff F, et al. Rethinking atrous convolution for semantic image segmentation [J]. arXiv preprint arXiv:1706.05587, 2017.

[0014] [8] Fan M, Lai S, Huang J, et al. Rethinking bisenet for real-time semantic segmentation [C] / / Conference on Computer Vision and Pattern Recognition. 2021:9716-9725.

[0015] [9] Xu J, Xiong Z, Bhattacharyya S P. PIDNet: A real-time semantic segmentation network inspired by PID controllers [C] / / Conference on Computer Vision and Pattern Recognition. 2023:19529-19539.

[0016]

[10] Wang W, Zhou T, Yu F, et al. Exploring cross-image pixel contrast for semantic segmentation [C] / / Conference on Computer Vision and Pattern Recognition. 2021:7303-7313.

[0017]

[11] Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [J]. arXiv preprint arXiv:2010.11929, 2020.

[0018]

[12] Cheng B, Misra I, Schwing A G, et al. Masked-attention mask transformer for universal image segmentation [C] / / Conference on Computer Vision and Pattern Recognition. 2022:1290-1299.

[0019]

[13] Lu C, de Geus D, Dubbelman G. Content-aware token sharing for efficient semantic segmentation with vision transformers [C] / / Conference on Computer Vision and Pattern Recognition. 2023:23631-23640.

[0020]

[14] Xie E, Wang W, Yu Z, et al. SegFormer: Simple and efficient design for semantic segmentation with transformers [J]. Advances in neural information processing systems. 2021, 34:12077-12090.

[0021]

[15] Gu J, Kwon H, Wang D, et al. Multi-scale high-resolution vision transformer for semantic segmentation [C] / / Conference on Computer Vision and Pattern Recognition. 2022:12094-12103.

[0022]

[16] Zheng S,Lu J,Zhao H,et al.Rethinking semantic segmentation from asequence-to-sequence perspective with transformers[C] / / Conference on ComputerVision and Pattern Recognition.2021:6881-6890.

[0023]

[17] Cheng B,Schwing A,Kirillov A.Per-pixel classification is not allyou need for semanticsegmentation[J].Advances in neural informationprocessing systems.2021,34:17864-17875.

[0024]

[18] Radford A,Kim J W,Hallacy C,et al.Learning transferable visualmodels from naturallanguage supervision[C] / / International Conference onMachine Learning.2021:8748-8763.

[0025]

[19] Xu M,Zhang Z,Wei F,et al.Side adapter network for open-vocabularysemanticsegmentation[C] / / Conference on Computer Vision and PatternRecognition.2023:2945-2954.

[0026]

[20] Cho S,Shin H,Hong S,et al.Cat-seg:Cost aggregation for open-vocabulary semanticsegmentation[C] / / Conference on Computer Vision and PatternRecognition.2024:4113-4123.

[0027]

[21] Lan M, Chen C, Ke Y, et al. Clearclip: Decomposing clip representations for dense vision-language inference [C] / / European Conference on Computer Vision. 2024: 143-160. SUMMARY

[0028] The present application aims to solve the problem that the existing open vocabulary semantic segmentation method fails to fully utilize the spatial representation of each image block, and proposes an open vocabulary semantic segmentation method based on image block sensitive agent attention network to realize better open vocabulary semantic segmentation function. The method fully utilizes the features of each image block itself to adaptively filter image blocks with more similarity, and then completes the open vocabulary semantic segmentation task.

[0029] The technical scheme of the present application is as follows:

[0030] An open vocabulary semantic segmentation method based on image block sensitive agent attention network, comprising the following steps:

[0031] (1) Constructing an image block sensitive agent attention network based on text encoder, two image encoders, wherein the two image encoders are a pre-trained CLIP image encoder and a visual foundation model VFM encoder respectively;

[0032] (2) For a given image I∈R H×W×3 , the semantic annotation information of the corresponding image is given, including the specific category of the object and the Ground Truth graph of the object mask, and the CLIP image encoder and the visual foundation model VFM encoder of the backbone network are used to extract image features F clip and F vfm ;

[0033] (3) Image block sensitive agent attention processing:

[0034] According to the extracted image features F clip and F vfm , the similarity graph

[0035] Calculate the average value μ i and the standard deviation σ i of each row of the similarity graph, that is, the average value and the standard deviation of each block of the image feature, input the similarity graph, the average value and the standard deviation into the image block adaptive normalization module;

[0036] (4) Image block adaptive normalization processing:

[0037] For the image block of the i-th row of the similarity graph, a threshold value θ is set i = μ i + β·σ i , β is a normalization coefficient, used to adjust the size of the threshold value;

[0038] Each row of the similarity graph is normalized by subtracting the corresponding threshold value, and scaled by dividing by the product of the standard deviation and the scaling coefficient, to obtain the normalized similarity graph after normalization operation;

[0039] (5) Mask processing to obtain attention graph A: the normalized similarity graph is subjected to mask operation, if the corresponding value is greater than 0, it is retained, if it is less than 0, it is discarded; after activation, the attention graph A is obtained;

[0040] (6) Image segmentation: multiply the obtained attention graph A and the image feature F clip extracted by CLIP, and then perform projection processing to obtain the final image feature F img , multiply the image feature F img and the category feature F cls obtained by the CLIP text encoder to obtain the segmented result image.

[0041] Further, the VFM encoder used is DINO, MAE or SAM.

[0042] Further, through the ablation experiment of parameters β and r, β = 0.6 and r = 3.0 are determined.

[0043] Further, in step (4), the formula for normalizing the similarity graph is as follows:

[0044]

[0045] In the formula, r is a scaling coefficient greater than 0, used to adjust the scaling multiple.

[0046] Compared with other semantic segmentation methods, the method proposed by the present application has the following advantages: (a) the design paradigm of the image block sensitive proxy attention module realizes the full use of various information in the different level feature maps extracted by the original backbone network, uses the text-image alignment ability of CLIP and the ability of VFM to learn local visual representation with strong spatial consistency, and can also adaptively select the image block most relevant to each image block. (b) The method can be applied to various scenes and has strong flexibility, and the size of CLIP can be changed, and different VFM models can be used to achieve good segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Application examples of open vocabulary semantic segmentation

[0048] Figure 2 The basic architecture of the image patch-sensitive proxy attention module segmentation network

[0049] Figure 3 Detailed process of the image block adaptive normalization module

[0050] Figure 4 Specific implementation method of the method proposed in the present invention DETAILED DESCRIPTION

[0051] The present invention will be described below with reference to the accompanying drawings and embodiments.

[0052] We first introduce the patch-sensitive proxy attention module proposed in this paper, and then show how to use this method for open-word semantic segmentation. Figure 2 The overall process architecture of the present invention is given.

[0053] (1) Patch-sensitive proxy attention module

[0054] Figure 2 The basic architecture of the open vocabulary segmentation network is given, which consists of a text encoder, two image encoders, an image block-sensitive proxy attention module, and a projection module. The two image encoders are the CLIP image encoder and the Vision Foundation Model (VFM) encoder. There are many types of VFM, including DINO, SAM, MAE, etc. The present invention is applicable to various VFM models. In this embodiment, the DINO encoder is used. For a given image I∈R H×W×3 First, the backbone network CLIP and VFM models are used to extract image features F clip and F vfm , both can extract image patch features separately, input them into the image patch sensitive proxy attention module, and then calculate the similarity graph Then calculate the average value μ of each row of the similarity graph i and standard deviation σ i , that is, for each block of image features, the mean and standard deviation are input into the image block adaptive normalization module. The process of the image block adaptive normalization module is as follows: Figure 3 As shown on the right, first pass the normalization and scaling module to set a threshold value θ i =μ i +β·σ i, where b is a normalization coefficient used to adjust the size of the threshold value, the threshold value of each row of the similarity graph is different, then each row of the similarity graph is subtracted by the corresponding threshold value for normalization, and divided by the standard deviation for scaling, where r is a scaling coefficient greater than 0 used to adjust the scaling multiple, which is expressed as follows:

[0055]

[0056] Through multiple ablation experiments on parameters b and r, in this experiment, b = 0.6, r = 3.0, the segmentation performance is the best. Then through the mask module, the normalized similarity graph is subjected to mask operation, if the corresponding value is greater than 0, its value is retained, if less than 0, it is assigned as negative infinity, and it is considered that the value less than 0 corresponds to a block that is not similar to the block, which is expressed as follows:

[0057]

[0058] Then through the activation module, which is expressed as follows:

[0059]

[0060] Get the attention graph A, then multiply the image feature F clip extracted by CLIP, and then project to get the final image feature F img , then multiply the image feature F img and the category feature F cls obtained by the CLIP text encoder to get the segmented result graph.

[0061] Compared with other semantic segmentation methods, the method proposed in the present application has the following advantages: (a) the design paradigm of the image block sensitive proxy attention module realizes the full use of various information in the different level feature maps extracted by the original backbone network, uses the text image alignment ability of CLIP and the ability of VFM to learn local visual representation with strong spatial consistency, and can also adaptively select the image block most relevant to each image block. (b) The method can be applied to various scenes and has strong flexibility, can change the size of CLIP, and can also use different VFM models, achieving good segmentation performance. Using ViT-Huge as the backbone network and DINO as the VFM, the segmentation performance of 84.1% mIoU is achieved on the VOC20 dataset, the segmentation performance of 65.9% mIoU is achieved on the VOC21 dataset, the segmentation performance of 42.0% mIoU is achieved on the CityScapes dataset, and the segmentation performance of 40.4% mIoU is achieved on the Context59 dataset.

[0062] ​(2) Apply the proposed image block sensitive proxy attention module to open vocabulary semantic segmentation

[0063] To apply the proposed image block sensitive proxy attention module to open vocabulary semantic segmentation, we need to go through an inference process without going through a training process. It completely does not involve any retraining, fine-tuning or gradient update of the neural network model parameters when performing the open vocabulary segmentation task, and their weights are frozen throughout the process, only using the proposed method to extract features.

[0064] As Figure 4 shown below, the inference process of the proposed method is introduced.

[0065] Step 1: Prepare a test image or dataset and give the semantic annotation information corresponding to the image, including the specific category of the object and the Ground Truth graph of the object mask.

[0066] Step 2: Select the backbone network for feature extraction, including CLIP and visual base model VFM, initialize with pre-trained weights, keep the network weights fixed, freeze the parameters, and do not train. Set the related hyperparameters of the test process, such as the parameter β of the image block adaptive normalization module when normalizing, the parameter γ of the image block adaptive normalization module when scaling, and the step size and window cropping size when sliding window inference.

[0067] Step 3: Use the image features obtained after the image passes through the CLIP and visual base model pre-trained model, input the text query word (corresponding to the class name) into CLIP, and extract the text features.

[0068] Step 4: Through the image block adaptive normalization module, cross-modal matching is performed between the image block level features and the text features, the similarity score is calculated, the segmentation graph corresponding to each category is generated, and the final semantic segmentation mask is generated.

Claims

1. A method for open-word semantic segmentation based on image patch-sensitive proxy attention network, comprising the following steps: (1) Construct an image block-sensitive proxy attention network, including a text encoder and two image encoders, where The two image encoders are the pre-trained CLIP image encoder and the visual basis model VFM encoder; (2) For a given image I∈R H×W×3 , gives the semantic annotation information of the corresponding image, including the specific category of the object and the ground truth map of the object mask, and uses the CLIP image encoder and the visual basic model VFM encoder of the backbone network to extract the image features F clip and F vfm ; (3) Image patch-sensitive proxy attention processing: According to the extracted image features F clip and F vfm , calculate the similarity graph Calculate the average value μ of each row of the similarity graph i and standard deviation σ i , that is, for each block of image features, the mean and standard deviation are input into the image block adaptive normalization module together with the similarity map, mean and standard deviation; (4) Image block adaptive normalization processing: For the image block in the i-th row of the similarity graph, set a threshold value θ i =μ i +β·σ i , β is the normalization coefficient, which is used to adjust the threshold value; Each row of the similarity graph is normalized by subtracting the corresponding threshold value, and then divided by the product of the standard deviation and the scaling factor to obtain the normalized similarity graph; (5) Masking to obtain attention map A: The similarity map after normalization operation is masked. If If the corresponding value is greater than 0, it is retained, and if it is less than 0, it is discarded; after activation, the attention map A is obtained; (6) Image segmentation: The obtained attention map A and the image features F extracted by CLIP are combined clip Multiply and project to get the final image feature F img , the image feature F img And the category feature F obtained by CLIP text encoder cls Multiply to get the segmentation result map.

2. The open vocabulary semantic segmentation method according to claim 1, characterized in that The VFM encoder used is DINO, MAE or SAM.

3. The open vocabulary semantic segmentation method according to claim 1, characterized in that Through the ablation experiment on the parameters β and r, β = 0.6 and r = 3.0 are determined.

4. The open vocabulary semantic segmentation method according to claim 1, characterized in that In step (4), the formula for normalizing the similarity graph is as follows: Where r is a scaling factor greater than 0, which is used to adjust the scaling factor.

Images