A zero-shot semantic segmentation method based on a heat map and a self-attention module

Abstract

A zero-shot semantic segmentation method based on heat map and self-attention module, comprising: for each training sample of a training set, generating a heat map of a category on an image according to a pre-trained visual-linguistic model CLIP; connecting a feature fusion module, a self-attention module and a mask prediction head in series to build a deep neural network; in each training batch, sampling multiple groups of "category-image-corresponding object mask" and corresponding heat maps, inputting the heat maps and the images into the network to obtain the predicted mask, and training the network; given any category not appearing in the training set and a picture to be segmented, generating a heat map of the category in the picture to be segmented, inputting the heat map and the picture to be segmented into the trained network, and segmenting the objects of the category in the picture to be segmented. The method can have the ability to segment category objects without training pictures at all.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and image processing, and in particular to a zero-shot semantic segmentation method based on a heat map and a self-attention module. BACKGROUND

[0002] Semantic segmentation is one of the most important and challenging problems in the field of computer vision. Given an input image of any size, a semantic segmentation model will output the class to which each pixel in the image belongs. Semantic segmentation has very wide applications in real-world scenarios, such as industrial quality inspection, autonomous driving, medical image processing, etc. Therefore, semantic segmentation has always been a research problem that is of great concern to both academia and industry.

[0003] However, the training of a semantic segmentation model requires a large amount of labeled data. Since the training images required by a semantic segmentation model need to be labeled pixel by pixel, it takes a lot of time and manpower to label enough training data. On the other hand, for some rare classes, it is almost impossible to collect enough training samples. Therefore, these shortcomings greatly increase the application cost of semantic segmentation models and limit the application scenarios of semantic segmentation models.

[0004] Zero-shot semantic segmentation aims to eliminate the dependence of semantic segmentation models on training samples. For a given one or more classes, a zero-shot semantic segmentation model does not need to learn segmentation ability from a large number of training images like a conventional semantic segmentation model; instead, a zero-shot semantic segmentation model can learn segmentation ability for the corresponding classes without any training images based on prior knowledge (such as attributes, inter-class relationships, etc.). Generally, the prior knowledge is provided by the word vector of the class to be segmented. The feature similarity between word vectors implicitly reflects the relationship between the corresponding classes, providing additional knowledge for the model, thus playing the role of training images.

[0005] Zero-shot semantic segmentation models generally use meta-learning to train. In the meta-training phase, the model will construct a zero-shot semantic segmentation scenario, segment objects of a specific class (base class) in the training image according to the word vector, and supervise according to the pixel-level label. In the meta-testing phase, when facing a class (new class) not contained in the training set, the segmentation ability for objects of the new class can be obtained without the need for training images of the new class, relying on the class word vector of the new class.

[0006] Existing zero-shot semantic segmentation algorithms can be mainly divided into two categories. The first category is generative algorithms. In the meta-training stage, the algorithm first trains a conventional semantic segmentation model on the base class training data to extract the image features of the base class objects; then a generative model is trained to have the ability to convert the word vectors of the base class into the image features of the base class objects. In the meta-testing stage, when facing new classes not contained in the training set, the generative model can generate the features of the new class objects through the word vectors of the new class, and fine-tune or retrain the semantic segmentation model through the related features. The training process of this kind of algorithm is complex, and the virtual image features of the new class objects generated are often quite different from the distribution of the real features, so the performance of the related model still needs to be improved. The other category of algorithms is discriminative algorithms. In the meta-training stage, the algorithm directly trains a model that converts word vectors into segmentation masks on the base class data, and directly acts on the word vectors of the new class in the meta-testing stage. In recent years, with the continuous improvement of the quality of word vectors, this kind of algorithm has gradually become the mainstream method of zero-shot semantic segmentation due to its simplicity and efficiency.

[0007] It should be noted that the information disclosed in the above background section is only for understanding the background of the present application, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0008] The main purpose of the present application is to provide a zero-shot semantic segmentation method based on heat maps and self-attention modules, which is different from the existing zero-shot semantic segmentation model, and can effectively solve the problem that the model training is highly dependent on training data.

[0009] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0010] A zero-shot semantic segmentation method based on heat maps and self-attention modules, comprising the following steps:

[0011] First, generate a heat map: for each set of training samples of “class-image-corresponding object mask” in the training set, generate a heat map of the class on the image according to the pre-trained visual-linguistic model CLIP;

[0012] Second, network building: the feature fusion module, the self-attention module and the mask prediction head are connected in series to build a deep neural network;

[0013] Third, network training: in each training round, in each training batch, sample multiple sets of “class-image-corresponding object mask” and the corresponding heat map generated in the first step as training data, input the heat map and the image into the network to obtain the predicted mask, calculate the loss function between the predicted mask and the real mask, and train the network by minimizing the loss function;

[0014] Fourth step, zero-shot semantic segmentation: Given any class that does not appear in the training set and the picture to be segmented, first generate the heat map of the class in the picture to be segmented in the first step, and then input the heat map and the picture to be segmented into the trained network to segment the objects of the class in the picture to be segmented.

[0015] Further:

[0016] In the first step, given a pre-trained visual-linguistic model CLIP, for each group of "class-image-corresponding object mask" in the training set, first input the class text into the text encoder of CLIP to obtain the text feature; then construct a single classification model composed of the image encoder of CLIP and a fully connected layer, where the weight of the fully connected layer is the value of the text feature; after resizing the image, input it into the classification model, and calculate the heat map of the output class on the last layer of the convolution feature map in the CLIP image encoder using the Grad CAM algorithm, and fix the resolution by bilinear interpolation.

[0017] In the first step, the text feature f∈R 1024 , the dimension of the fully connected layer is 1024*1; the image size is resized to 400*400, and the resolution is fixed to 1*50*50.

[0018] In the second step, the network includes a feature fusion module, a self-attention module and a mask prediction head; the image and the heat map are input into the feature fusion module to obtain the fusion feature map; the fusion feature map is input into the self-attention module to obtain the enhanced feature map; the enhanced feature map and the heat map are input into the mask prediction head to obtain the final output mask.

[0019] In the second step, the feature fusion module includes a feature extraction network, a multi-scale feature fusion module and a feature-heat map fusion module; the feature extraction network ensures that the input image size is uniformly scaled in training and testing, the multi-scale feature fusion module first extracts the convolution feature map of the feature extraction network, and then uniformly scales the spatial resolution of the feature map by bilinear interpolation, and then adds the convolution feature maps to fuse feature maps of different resolutions; the feature-heat map module uses the addition result of the feature map to reduce the dimension of the feature map, and connects the heat map and the reduced feature map in the channel to obtain the fusion feature map.

[0020] In the second step, the feature extraction network is VGG16, the input image size is uniformly scaled to 400*400, the multi-scale feature fusion module extracts the last 7 layers of convolution feature maps of VGG16 network, that is, all convolution feature maps with a feature dimension of 512d, and uniformly scales the spatial resolution of the feature maps to 50*50 by bilinear interpolation, and then adds the 7 layers of feature maps; the feature-heat map module uses a 1*1 convolution to reduce the dimension of the addition result of the 7 layers of feature maps from 512*50*50 to 127*50*50, and connects the heat map in the channel to obtain a fusion feature map with a dimension of 128*50*50.

[0021] In the second step, the self-attention module is stacked by multiple self-attention layers, and the self-attention layer is defined as follows: let the input feature Fi have a dimension of C*H*W, first stretch it to a feature Fi' with a dimension of C*HW; then input Fi' into three parallel fully connected layers WQ, WK, and Wv to obtain features Q, K, and V; weight and sum V according to the similarity of Q and K to obtain output feature A; after A passes through a fully connected layer and a drop out module, it is added to Q to obtain the enhanced output feature.

[0022] In the second step, the mask prediction head includes an upper branch and a lower branch, the upper branch is composed of two 3*3 convolution layers, and the initial prediction result of the segmentation mask is obtained according to the enhanced feature map, and the lower branch is composed of two 1*1 convolution layers, which are used to weight and fuse the initial prediction result with the heat map, so that the heat map can directly reach the output end.

[0023] A zero-shot semantic segmentation method based on a heat map and a self-attention module, comprising the following steps:

[0024] First, generate a class heat map based on a self-supervised pre-trained visual-linguistic model;

[0025] Second, feature fusion: first extract multi-scale image features, then weight and sum the features of different resolutions, and then connect the fusion result with the heat map to obtain a fusion feature map;

[0026] Third, input the fusion feature map into a stacked self-attention layer, so that the model captures the spatial correlation under the constraint of the heat map prior, and obtains an enhanced feature map;

[0027] Fourth, input the enhanced feature map and the heat map into a mask prediction head, the mask prediction head includes an upper branch and a lower branch, the upper branch is composed of stacked convolution layers, and the initial prediction result of the segmentation mask is obtained according to the enhanced feature map, and the lower branch weights and fuses the initial prediction result with the heat map, so that the heat map can directly reach the output end;

[0028] In the fifth step, the trained network is used for zero-shot semantic segmentation, that is, for a new class not contained in the training set, the semantic segmentation of the objects of the class can be performed only by the class text of the class.

[0029] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the zero-shot semantic segmentation method based on the heat map and the self-attention module.

[0030] The beneficial effects of the present application compared with the prior art include:

[0031] The present application provides a zero-shot semantic segmentation method based on a heat map and a self-attention module. Compared with other traditional fully supervised semantic segmentation models, the present application can learn the ability to segment objects of a specific class without collecting and labeling training pictures corresponding to the class when the specific class is given. Unlike existing zero-shot semantic segmentation models, the present application incorporates the knowledge contained in the class text into the image features of the image to be segmented through a heat map, and realizes the interaction and enhancement of the information in the image features and the information in the heat map through a self-attention module, thereby segmenting the picture only relying on the class text, so that the model can effectively eliminate the dependence on training pictures. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of the zero-shot semantic segmentation method based on the heat map and the self-attention module of the present application.

[0033] Figure 2 is a model structure diagram when VGG16 is used in the zero-shot semantic segmentation method based on the heat map and the self-attention module of an embodiment of the present application.

[0034] Figure 3 is a feature fusion module diagram in the zero-shot semantic segmentation method based on the heat map and the self-attention module of an embodiment of the present application.

[0035] Figure 4 is a mask prediction head diagram in the zero-shot semantic segmentation method based on the heat map and the self-attention module of an embodiment of the present application. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described in detail below. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present application and its applications.

[0037] Referring to Figure 1 The embodiment of the present application provides a zero-shot semantic segmentation method based on a heat map and a self-attention module, including the following steps:

[0038] First step, generate heat map: for each training sample of "category-image-corresponding object mask" in the training set, generate the heat map of the category in the image according to the pre-trained visual-linguistic model CLIP;

[0039] Second step, network building: connect the feature fusion module, self-attention module and mask prediction head in series to build a deep neural network;

[0040] Third step, network training: in each training round, sample multiple groups of "category-image-corresponding object mask" and the corresponding heat map generated in the first step as training data in each training batch. Input the heat map and the image into the network to get the predicted mask, calculate the loss function between the predicted mask and the real mask, and train the network by minimizing the loss function.

[0041] Fourth step, zero-shot semantic segmentation: given any category not appearing in the training set and the picture to be segmented, first generate the heat map of the category in the picture to be segmented in the first step, then input the heat map and the picture to be segmented into the trained network, which can segment the category objects in the picture to be segmented.

[0042] In some other embodiments, a zero-shot semantic segmentation method based on heat map and self-attention module includes the following steps:

[0043] First step, generate category heat map based on self-supervised pre-trained visual-linguistic model;

[0044] Second step, feature fusion, that is, first extract multi-scale image features, fuse feature maps of different resolutions in a weighted sum manner, and then connect the fusion result with the heat map to get the final fusion feature map;

[0045] Third step, input the fusion feature map into the stacked self-attention layer, so that the model captures the spatial correlation under the constraint of the heat map prior, and gets the enhanced feature map;

[0046] Fourth step, input the enhanced feature map and the heat map into the mask prediction head, which includes

[0047] The upper branch is composed of stacked convolutional layers, which obtains the initial prediction result of the segmentation mask according to the enhanced feature map, and the lower branch weights and fuses the initial prediction result with the heat map to ensure that the heat map can reach the output end and avoid the loss of prior information in the deep network;

[0048] Fifth step, use the trained network for zero-shot semantic segmentation, that is, for new categories not included in the training set, the model can only perform semantic segmentation on the category objects through the category text of the category.

[0049] The method of the present application can generate word vectors using a pre-trained CLIP text encoder, and embed the prior knowledge contained in the word vectors into the segmentation model through a heat map. Compared with traditional fully supervised semantic segmentation methods, the present application can have the ability to segment objects of a given category without the need to train pictures at all.

[0050] The specific embodiments of the present application are further described below.

[0051] A zero-shot semantic segmentation method based on a heat map and a self-attention module, comprising the following steps:

[0052] First, generate a heat map: for each set of "category-image-corresponding object mask" in the training set, generate a heat map of the category in the image according to the pre-trained visual-linguistic model CLIP.

[0053] The first step specifically includes: given a pre-trained visual-linguistic model CLIP, for each set of "category-image-corresponding object mask" in the training set, first input the category text into the text encoder of CLIP to obtain the feature f∈R1024; then construct a single classification model composed of the image encoder of CLIP and a fully connected layer, where the dimension of the fully connected layer is 1024*1, and the weight is the value of f; resize the image size to 400*400, input it into the classification model, and calculate the heat map of the output category on the last layer of the convolution feature map of the CLIP image encoder using the Grad CAM algorithm, and fix the resolution to 1*50*50 by bilinear interpolation.

[0054] Second, network building: connect the feature fusion module, the self-attention module and the mask prediction head in series to build a deep neural network.

[0055] In the second step, the network built includes a feature fusion module, a self-attention module and a mask prediction head. As shown in Figure 2 The image and the heat map are input into the feature fusion module at the same time to obtain a fused feature map. The fused feature map is input into the self-attention module to obtain an enhanced feature map. The enhanced feature map and the heat map are input into the mask prediction head at the same time to obtain the final output mask.

[0056] The second step, the feature fusion module includes three parts, respectively, feature extraction network, multi-scale feature fusion module and feature-heat map fusion module. The feature extraction network is the commonly used VGG16, and at the same time, the input image size is uniformly scaled to 400*400 in training and testing. The multi-scale feature fusion module first extracts the last 7 layers of the VGG16 network (i.e. all convolutional feature maps with a feature dimension of 512d), and uniformly scales the spatial resolution of the feature map to 50*50 by bilinear interpolation, and then adds the 7 layers of feature maps to fuse the feature maps of different resolutions. The feature-heat map module first uses a 1*1 convolution to reduce the dimension of the added result of the 7 layers of feature maps from 512*50*50 to 127*50*50 feature maps, and then connects the heat map and the reduced feature map in the channel to obtain a fusion feature map with a dimension of 128*50*50. The specific structure of the feature fusion module is shown in Figure 3

[0057] The second step, the self-attention module is stacked by multiple self-attention layers. The definition of the self-attention layer is as follows: let the input feature Fi have a dimension of C*H*W, first stretch it to a feature Fi' with a dimension of C*HW; then input Fi' into three parallel fully connected layers WQ, WK, and Wv to obtain features Q, K, and V; weight and sum V according to the similarity of Q and K to obtain output feature A; after A passes through a fully connected layer and a dropout module, it is added to Q to obtain the enhanced output feature.

[0058] The second step, the mask prediction head includes an upper and lower branch. The upper branch is composed of two 3*3 convolutional layers, which obtains the initial prediction result of the segmentation mask according to the enhanced feature map, and the lower branch is composed of two 1*1 convolutional layers, which are used to weight and fuse the initial prediction result with the heat map, so that the heat map can directly reach the output end, avoiding the loss of prior information in the deep network, as shown in Figure 4

[0059] The third step, network training: in each training round, in each training batch, sample multiple groups of "category-image-corresponding object mask" and the corresponding heat map generated in the first step as training data. Input the heat map and the image into the network to obtain the predicted mask, calculate the loss function between the predicted mask and the real mask, and train the network by minimizing the loss function.

[0060] The fourth step, zero-shot semantic segmentation: given any category not appearing in the training set and the picture to be segmented, first generate the heat map of the category in the picture to be segmented in the first step, and then input the heat map and the picture to be segmented into the trained network, which can segment the objects of the category in the picture to be segmented.

[0061] ​​Further details are described below.

[0062] Generate heat maps:

[0063] Suppose there are C classes in total in the training set, and each class contains Nc images, then a total of heat maps should be generated. For each "class-image" pair, first insert the class name into the template "A photo of [class name]", and then input the text encoder of CLIP to obtain a 1024-dimensional text feature. Then construct a single-classification model composed of the image encoder of CLIP (improved ResNet) and a fully connected layer classifier in series, where the dimension of the fully connected layer is 1024*1, and the weight is the value of the text feature. Resize the image size to 400*400, input the classification model, and calculate the heat map of the output class on the last layer of the convolution feature map of the CLIP image encoder using the Grad CAM algorithm. In this way, a "class-image-heat map" combination can be generated for each group of "class-image".

[0064] Network building:

[0065] The built network includes a feature fusion module, a self-attention module, and a mask prediction head, as shown in Figure 2 .

[0066] The feature fusion module includes three parts, namely the feature extraction network, the multi-scale feature fusion module, and the feature-heat map fusion module. The feature extraction network is the commonly used VGG16, and its weight is the ImageNet pre-training weight. The multi-scale feature fusion module first extracts the last 7 layers of convolution feature maps of the VGG16 network (i.e. all convolution feature maps with a feature dimension of 512d), and then uniformly resizes the spatial resolution of the feature maps to 50*50 using bilinear interpolation. Then, the 7-layer feature maps are added to fuse feature maps of different resolutions. The feature-heat map module first uses a 1*1 convolution to reduce the dimension of the addition result of the 7-layer feature maps from 512*50*50 to 127*50*50 feature maps, and then connects the heat map and the reduced dimension feature map in the channel to obtain a fusion feature map with a dimension of 128*50*50.

[0067] The self-attention module is stacked by multiple self-attention layers. The fusion feature map with a dimension of 128*50*50 is stretched to a feature map with a dimension of 128*2500. Then, using a fully connected layer WQ, WK, WV with a dimension of 128*128, the input feature map with a dimension of 128*2500 is converted into features Q, K, V, each with a dimension of 128*2500. The transpose matrix of Q is calculated and divided by The similarity matrix (dimension 2500*2500) is obtained. The softmax operation is performed on the second dimension of the similarity matrix, and then the logarithm is taken for all values to obtain the attention matrix Attn (dimension 2500*2500). The product of the feature V and the Attn is calculated as the output matrix O1. Similarly, another output matrix O2 is obtained by the same operation but with different WQ, WK, and WV matrices. The O1 and O2 matrices are connected to obtain the output feature O with a dimension of 256*2500. The feature O is processed by a fully connected layer with a dimension of 128*256 (containing a dropout operation, in which the neuron inactivation probability is 0.5), and is added to the original feature, that is, the enhanced feature (dimension 128*2500) output by the first self-attention layer. Three self-attention layers are connected in series, and the output of the last self-attention layer is the final enhanced feature. The algorithm of each self-attention layer is specifically shown in Algorithm 1.

[0068]

[0069]

[0070] The mask prediction head includes an upper branch and a lower branch. The upper branch is composed of two 3*3 convolutional layers to obtain the initial prediction result of the segmentation mask according to the enhanced feature map, and the lower branch is composed of two 1*1 convolutional layers to perform weighted fusion of the initial prediction result and the heat map, so that the heat map can directly reach the output end and avoid loss of prior information in the deep network, as shown in Figure 4

[0071] Network training:

[0072] In the training set, the “category-image-corresponding object mask” is sampled and the heat map generated in the first step. According to the data flow order shown in Figure 2 , the mask prediction result is obtained. The model is trained by minimizing the cross-entropy loss function.

[0073] Zero-shot semantic segmentation:

[0074] Given any category not appearing in the training set and the picture to be segmented, first generate the heat map of the category in the picture to be segmented in the first step, and then input the heat map and the picture to be segmented into the trained network, so as to segment the category objects in the picture to be segmented.

[0075] According to the experimental settings in [1], the present application performs comparative experiments on Pascal-5 i and COCO-20 i . The Pascal-5 i ​Each sub-dataset contains 5 classes, and the classes are different among sub-datasets. When one sub-dataset is the new class dataset, the other three sub-datasets will be the base class dataset. COCO-20 i Similarly, each sub-dataset contains 20 classes. The test metric is mIoU.

[0076] The comparison experiments of the proposed method and other zero-shot semantic segmentation methods are as follows:

[0077] 1) Pascal-5 i

[0078]

[0079] 2) COCO-20 i

[0080]

[0081] The comparison experiment results show that the proposed method can use a lighter feature extraction network (ResNet50 vs ResNet101) to achieve higher performance.

[0082] Reference:

[0083] [1] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In ICLR, 2022. [2] Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, and Zeynep Akata. Semantic projection network for zero-and few-label semantic segmentation. In CVPR, 2019

[0084] [3] Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Perez. Zero-shot semantic segmentation. NeurIPS, 2019

[0085] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0086] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0087] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0089] The background section of the present application can include information about background issues or context of the present application, and does not necessarily constitute prior art. Therefore, the contents included in the background section are not admissions of prior art by the applicant.

[0090] The above further describes the present application in conjunction with specific / preferred embodiments, and cannot be deemed to limit the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, they can make several substitutions or variations to the described embodiments, and these substitutions or variations shall be deemed to fall within the protection scope of the present application. In the description of the present application, the description of the terms "an embodiment", "some embodiments", "a preferred embodiment", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are contained in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In the case of no mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples. Although the embodiments of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A zero-shot semantic segmentation method based on a heat map and a self-attention module, characterized in that, The method comprises the following steps: The first step is to generate a heat map: for each training sample of "category-image-corresponding object mask" in the training set, a heat map of the category in the image is generated according to a pre-trained visual-linguistic model CLIP; The second step is to build a network: a deep neural network is built by connecting a feature fusion module, a self-attention module and a mask prediction head; the built network comprises the feature fusion module, the self-attention module and the mask prediction head; the image and the heat map are input into the feature fusion module at the same time to obtain a fusion feature map; the fusion feature map is input into the self-attention module to obtain an enhanced feature map; the enhanced feature map and the heat map are input into the mask prediction head at the same time to obtain a final output mask; wherein the feature fusion module comprises a feature extraction network, a multi-scale feature fusion module and a feature-heat map fusion module; the feature extraction network ensures that the input image size is uniformly scaled in training and testing; the multi-scale feature fusion module first extracts the convolution feature map of the feature extraction network, and uniformly scales the spatial resolution of the convolution feature map; then the convolution feature maps are added to fuse feature maps of different resolutions; the feature-heat map module uses the addition result of the feature maps to reduce the dimension of the feature maps, and connects the heat map with the reduced feature maps in the channel to obtain a fusion feature map; wherein the mask prediction head comprises an upper branch and a lower branch; the upper branch obtains an initial prediction result of the segmentation mask according to the enhanced feature map; the lower branch is used for weighted fusion of the initial prediction result and the heat map, so that the heat map can directly reach the output end; the self-attention module is stacked by a plurality of self-attention layers; the definition of the self-attention layer is as follows: let the input feature Fi have a dimension of C*H*W; first stretch it into a feature Fi' with a dimension of C*HW; then input Fi' into three parallel fully connected layers WQ, WK and Wv to obtain features Q, K and V; according to the similarity of Q and K, the weighted sum of V is obtained; after the enhanced feature A passes through a fully connected layer and a drop out module, it is added to Q to obtain the output enhanced feature; The third step is network training: in each training round, in each training batch, a plurality of "category-image-corresponding object mask" and the corresponding heat map generated in the first step are sampled as training data; the heat map and the image are input into the network to obtain a predicted mask; the loss function between the predicted mask and the real mask is calculated, and the network is trained by minimizing the loss function; The fourth step is zero-shot semantic segmentation: given any category not appearing in the training set and a picture to be segmented, first generate a heat map of the category in the picture to be segmented in the first step, and then input the heat map and the picture to be segmented into the trained network to segment the objects of the category in the picture to be segmented.

2. The zero-shot semantic segmentation method of claim 1, wherein, In the first step, given a pre-trained visual-linguistic model CLIP, for each "category-image-corresponding object mask" in the training set, first input the category text into the text encoder of CLIP to obtain a text feature; Then a single classification model is constructed, which is composed of the image encoder of CLIP and a fully connected layer, and the weight of the fully connected layer is the value of the text feature; After the image size is scaled, it is input into the classification model, and the Grad CAM algorithm is used to calculate the heat map of the output category on the last layer of the CLIP image encoder, and the resolution is fixed by bilinear interpolation.

3. The zero-shot semantic segmentation method of claim 2, wherein, In the first step, the text feature f e R 1024 The dimension of the full connection layer is 1024*1; the image size is scaled to 400*400, and the resolution is fixed to 1*50*50.

4. The zero-shot semantic segmentation method of any one of claims 1 to 3, wherein, In the second step, the multi-scale feature fusion module uniformly scales the spatial resolution of the convolutional feature map by bilinear interpolation.

5. The zero-shot semantic segmentation method of any one of claims 1 to 3, wherein, In the second step, the feature extraction network is VGG16, the input image size is uniformly scaled to 400*400, the multi-scale feature fusion module extracts the last 7 layers of convolutional feature maps of VGG16 network, that is, all convolutional feature maps with a feature dimension of 512d, and uniformly scales the spatial resolution of the feature map to 50*50 by bilinear interpolation, then adds the 7 layer feature maps; The feature-heat map module uses 1*1 convolution to reduce the dimension of the addition result of the 7 layer feature maps from 512*50*50 to 127*50*50, and connects the heat map in the channel to obtain a fusion feature map with a dimension of 128*50*50.

6. The zero-shot semantic segmentation method of any one of claims 1 to 3, wherein, In the second step, the upper branch is composed of two 3*3 convolutional layers, and the lower branch is composed of two 1*1 convolutional layers.

7. A zero-shot semantic segmentation method based on a heat map and a self-attention module, characterized in that, The method comprises the following steps: First step, generate a category heat map based on a self-supervised pre-trained visual-linguistic model; Second step, feature fusion: first extract multi-scale image features, fuse features with different resolutions by weighted summation, then connect the fusion result with the heat map to obtain a fusion feature map; Third step, input the fusion feature map into a stacked self-attention layer, so that the model captures the spatial dimension correlation under the constraint of the heat map prior, and obtains an enhanced feature map; Fourth step, input the enhanced feature map and the heat map into a mask prediction head, the mask prediction head includes an upper branch and a lower branch, the upper branch is composed of stacked convolutional layers, and the initial prediction result of the segmentation mask is obtained according to the enhanced feature map, and the lower branch fuses the initial prediction result and the heat map by weighting, so that the heat map can directly reach the output end; The second step to the fourth step uses a deep neural network built in series by a feature fusion module, a self-attention module and a mask prediction head; the network built contains the feature fusion module, the self-attention module and the mask prediction head; the image and the heat map are input into the feature fusion module at the same time to obtain a fusion feature map; the fusion feature map is input into the self-attention module to obtain an enhanced feature map; the enhanced feature map and the heat map are input into the mask prediction head at the same time to obtain a final output mask; wherein the feature fusion module contains a feature extraction network, a multi-scale feature fusion module and a feature-heat map fusion module; the feature extraction network ensures that the input image size is uniformly scaled in training and testing, the multi-scale feature fusion module first extracts the convolution feature map of the feature extraction network, and uniformly scales the spatial resolution of the feature map, then adds the convolution feature map to fuse feature maps of different resolutions; the feature-heat map module uses the addition result of the feature map to reduce the dimension of the feature map, and connects the heat map and the reduced feature map in the channel to obtain a fusion feature map; wherein the mask prediction head contains two branches, the upper branch obtains an initial prediction result of the segmentation mask according to the enhanced feature map, and the lower branch is used for weighted fusion of the initial prediction result and the heat map, so that the heat map can directly reach the output end; the self-attention module is stacked by a plurality of self-attention layers, and the definition of the self-attention layer is as follows; let the input feature Fi have a dimension of C*H*W, first stretch it into a feature Fi' with a dimension of C*HW; then input Fi' into three parallel fully connected layers WQ, WK and Wv to obtain features Q, K and V; according to the similarity of Q and K, the weighted sum of V is obtained, and the output feature A is obtained; after A passes through a fully connected layer and a drop out module, it is added to Q to obtain the output enhanced feature; In the fifth step, the trained network is used for zero-shot semantic segmentation, that is, for new categories not contained in the training set, the category objects can be segmented only by the category text of the category.

8. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, implements the zero-shot semantic segmentation method based on the heat map and the self-attention module according to any one of claims 1 to 7. The computer program, when executed by the processor, implements the zero-shot semantic segmentation method based on the heat map and the self-attention module according to any one of claims 1 to 7.

Images