A zero-shot semantic segmentation method based on semantic attention mechanism

Abstract

The application discloses a zero sample semantic segmentation method based on a semantic attention mechanism. The method comprises the following steps: extracting a background mask through a background extraction module, inputting the background mask and initial background features into a background updating module, and outputting learned background semantic features. A semantic attention map and a visual feature map are obtained through a pixel feature extraction module, a semantic feature suitable for the current picture is obtained by using the semantic feature map and the semantic attention map, and finally the cosine similarity of the semantic feature and the visual feature map is calculated, and semantic segmentation is realized according to the maximum similarity. Through the learnable and updatable background and semantic features, different semantic features are learned for different pictures, and the obtained semantic features help the zero sample semantic segmentation model to better perform semantic segmentation on each pixel point, thereby improving the generalization and universality of the model.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a zero-shot semantic segmentation method based on a semantic attention mechanism. BACKGROUND

[0002] The rapid development of deep learning technology has promoted the rapid development of the field of computer vision. Among them, the effect of computer vision tasks such as low-level denoising, deblurring and de-pseudo and high-level object classification, object detection and object semantic segmentation has been improved most obviously. Among them, semantic segmentation, as a basic task in computer vision tasks with wide application, has been widely valued by the scientific research and industrial circles. Although the semantic segmentation method based on deep learning has good effect on various data sets, it has never been seen before. However, these traditional semantic segmentation methods need a large amount of pixel-by-pixel labeled labeled data to support such results. At the same time, these methods can only perform semantic segmentation on the classes provided in the training phase, and have no ability to handle unseen classes, lacking generalization ability and universality.

[0003] In order to solve the above problems of traditional semantic segmentation based on deep learning, a zero-shot semantic segmentation task is proposed. The zero-shot semantic segmentation task aims to realize semantic segmentation of more classes of pictures through a small amount of labeled data of classes. This task not only solves the need to reduce the labeled training data, but also improves the generalization ability of the semantic segmentation model, enabling it to have the ability to segment unseen classes.

[0004] The basic solution to the current zero-shot semantic segmentation task is to use fixed semantic features to calculate the similarity of the picture feature maps extracted by the traditional semantic segmentation model for pixel classification. The characteristics of zero-shot semantic features are not fully utilized, and variable semantic features are used for semantic segmentation for different pixels. This method is based on this problem, and proposes a learnable background class semantic feature. The background semantic feature is updated according to the background of the picture, and the semantic feature corresponding to the pixel point is obtained according to the attention of the pixel point to different classes. Each pixel point is classified after calculating the similarity according to the adjusted semantic feature. SUMMARY

[0005] (I) Technical problems to be solved

[0006] The current zero-shot semantic segmentation task basically solves the problem that the fixed semantic features are used to calculate the similarity of the picture feature maps extracted by the traditional semantic segmentation model for pixel classification, and the characteristics of zero-shot semantic features are not fully utilized, and variable semantic features are used for semantic segmentation for different pixels. A zero-shot semantic segmentation method based on a semantic attention mechanism is provided.

[0007] (II) Technical Solution

[0008] The purpose of the present application is achieved by the following technical solution: a zero-shot semantic segmentation method based on semantic attention mechanism, comprising the following steps:

[0009] Step one: construct a zero-shot semantic segmentation dataset;

[0010] Step two: input the training picture of step one dataset into the background extraction module, and output the mask picture of the background area of the training picture;

[0011] Step three: use the mask picture obtained in step two to cover the foreground part of the training picture, and obtain the background area of the training picture;

[0012] Step four: calculate the semantic segmentation loss of the background of the training picture by using the background annotation of the training picture;

[0013] Step five: input the background area obtained in step three into the background feature updating module, and input the initial background feature into the background module; the background feature updating module outputs the background feature suitable for the semantic segmentation of the current picture;

[0014] Step six: input the training picture into the pixel feature extraction module, and the pixel feature extraction module outputs two pictures, one is the feature map of the pixel-level visual feature of the picture, and the other is the semantic attention map;

[0015] Step seven: apply the semantic attention map obtained in step six to the semantic feature to obtain a semantic feature vector for semantic segmentation of each pixel point;

[0016] Step eight: calculate the similarity between the semantic feature vector obtained in step seven for each pixel point and the background feature obtained in step five and each pixel point of the feature map output in step six as the classification confidence of each pixel point, and calculate the semantic segmentation loss of the whole picture through the loss function;

[0017] Step nine: calculate the total loss according to the background and the semantic segmentation loss of the whole picture calculated in steps four and eight, and train the background extraction module, the background feature updating module and the pixel feature extraction module with the total loss;

[0018] Step ten: when testing, first extract the background area mask of the test picture using the background extraction module in step three, then get the background area of the test picture using the mask, input the background area and the initial background feature into the background feature update module in step five to get the background feature of the current test picture; at the same time, input the test picture into the pixel feature extraction model in step six, get the test semantic feature using the semantic attention map output by the pixel feature extraction model; finally, calculate the classification confidence of each pixel point using the method in step eight, and take the maximum value as the classification result of each pixel point.

[0019] As a preferred technical solution, in step one, the Visual Object Classes Challenge 2012 (VOC2012) dataset is used, there are 8256 training pictures and 1449 test pictures, there are 15 visible classes and 5 unseen classes.

[0020] As a preferred technical solution, in step two, the background extraction module uses the MaskFormer model, which is mainly used for supervised semantic segmentation tasks and can output the background area of the picture after inputting the picture.

[0021] As a preferred technical solution, in step four, the semantic segmentation loss of the training picture background is calculated by the binary cross-entropy loss function.

[0022] As a preferred technical solution, in step five, the initial background feature is the word vector feature of the word "background".

[0023] As a preferred technical solution, in step five, the background feature update module is composed of 2 layers of 3x3 convolution layers and ReLU activation layers, and finally gets the background feature through global average pooling, then splices the initial background feature, and then gets the background feature suitable for the semantic segmentation of the current picture through 1 layer of fully connected layer and ReLU activation layer in step five.

[0024] As a preferred technical solution, in step six, the skeleton model used by the pixel feature extraction module is the deeplabv3 model.

[0025] As a preferred technical solution, in step six, the pixel feature map output by deeplabv3 is multiplied by the semantic feature of the visible class to get a similarity map, and then a semantic attention map is output after inputting the similarity map into 3 layers of 3x3 convolution layers and ReLU layers.

[0026] As a preferred technical solution, the specific operation of applying the semantic attention map to the semantic feature in step seven is to first copy the semantic feature of the visible class number on each pixel point to obtain a semantic feature map, and then multiply the semantic attention map obtained in step six and the copied semantic feature map to obtain the semantic attention semantic feature map after using the semantic attention.

[0027] As a preferred technical solution, the similarity between the feature vector and the semantic feature vector is calculated by cosine similarity in step eight, and the semantic segmentation loss in step eight is calculated by a binary cross-entropy loss function.

[0028] (Three) beneficial effects

[0029] The beneficial effects of the present application are: the present method learns variable semantic features for semantic segmentation of each pixel point in the way of semantic attention for zero-shot semantic segmentation task. Specifically, the present method obtains a background region by extracting the background region of a picture, thereby updating the semantic features of the background semantic class to obtain background semantic features suitable for the current picture. Meanwhile, the present method extracts a semantic attention map through a pixel feature extraction module to help each pixel point obtain semantic features suitable for it. Learning variable semantic features from two aspects helps the zero-shot semantic segmentation model to improve performance. BRIEF DESCRIPTION OF DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0031] Figure 1 is the overall model structure;

[0032] Figure 2 is the background feature updating module; DETAILED DESCRIPTION

[0033] The present application will be further described in detail below in combination with the drawings and specific embodiments.

[0034] As Figure 1 shown, the main steps of the technical solution adopted by the present application are as follows:

[0035] Step one: use 15 classes of 8256 pictures in the Visual Object Classes Challenge 2012 (VOC2012) dataset as the training data set of zero-shot semantic segmentation.

[0036] Step two: useFigure 1 The background extraction module in the background extraction module extracts the background region of the training picture. We use MaskFormer as the background extraction module.

[0037] Step three: use the mask obtained in step two to mask the foreground part of the training picture to obtain the background mask of the training picture.

[0038] Step four: use the background annotation of the training picture to calculate the semantic segmentation loss of the background of the training picture by using the binary cross-entropy loss function.

[0039] Step five: input the background region obtained in step three into the background feature update module, and input the initial background feature into the background module. The background feature update module outputs the background feature suitable for the semantic segmentation of the current picture. The background update module structure is shown in Figure 2 The background update module structure is shown in

[0040] Step six: input the training picture into the pixel feature extraction module. The pixel feature extraction module outputs two graphs, one is the pixel-level visual feature graph of the picture, and the other is the semantic attention graph. The basic model of the pixel feature extraction module is deeplabv3. The similarity graph obtained by multiplying the feature graph output by deeplabv3 with the semantic feature of the visible class is input into 3 layers of convolution layer and ReLU activation layer to obtain the semantic attention graph.

[0041] Step seven: apply the semantic attention graph obtained in step six to the semantic feature to obtain a semantic feature vector suitable for semantic segmentation of each pixel.

[0042] Step eight: use the semantic feature vector suitable for each pixel obtained in step seven and the background feature obtained in step five for semantic segmentation of the feature graph output in step six to obtain the classification confidence of each pixel. The semantic segmentation loss of the whole picture is calculated by using the binary cross-entropy according to the semantic segmentation annotation of the picture.

[0043] Step nine: calculate the total loss according to the background and whole picture semantic segmentation loss calculated in steps four and eight. Use the total loss to train the background extraction module, background feature update module and pixel feature extraction module.

[0044] Step ten: when testing, first use the background extraction module in step three to extract the background area mask of the test picture, then use the mask to obtain the background area of the test picture, input the background area and the initial background feature into the background feature updating module in step five to obtain the background feature of the current test picture. At the same time, input the test picture into the pixel feature extraction model in step six, use the semantic attention map output by the pixel feature extraction model to obtain the test semantic feature. Finally, calculate the classification confidence of each pixel point by the method in step eight, and take the maximum value as the classification result of each pixel point.

[0045] It should be noted that: the present application builds a zero-shot semantic segmentation model that can learn semantic features, uses the updateable background semantic features and the semantic features obtained by the semantic attention mechanism to help the model better use semantic features for picture semantic segmentation.

[0046] The above embodiments only describe the preferred embodiments of the present application, and do not limit the concept and scope of the present application. Without departing from the design concept of the present application, various modifications and improvements of the technical solutions of the present application made by those skilled in the art shall fall within the protection scope of the present application. The technical content claimed by the present application has been fully recorded in the claims.

Claims

1. A zero-shot semantic segmentation method based on semantic attention mechanism, characterized in that, The method comprises the following steps: Step one: constructing a zero-shot semantic segmentation dataset; Step two: inputting the training picture of the dataset of step one into a background extraction module to output a mask picture of the background region of the training picture; Step three: using the mask picture obtained in step two to mask the foreground part of the training picture to obtain the background region of the training picture; Step four: calculating the semantic segmentation loss of the background of the training picture by using the background label of the training picture; Step five: inputting the background region obtained in step three into a background feature updating module, and inputting an initial background feature into the background module; The background feature updating module outputs a background feature suitable for the semantic segmentation of the current picture; Step six: inputting the training picture into a pixel feature extraction module, and the pixel feature extraction module outputs two pictures, one is a feature picture of the pixel-level visual feature of the picture, and the other is a semantic attention picture; Step seven: using the semantic attention picture obtained in step six to act on the semantic feature to obtain a semantic feature vector for the semantic segmentation of each pixel point; Step eight: calculating the similarity between the semantic feature vector of each pixel point obtained in step seven and the background feature obtained in step five and each pixel point of the feature picture output in step six as the classification confidence of each pixel point, and calculating the semantic segmentation loss of the whole picture by using a loss function; Step nine: calculating the total loss according to the background semantic segmentation loss calculated in step four and the semantic segmentation loss of the whole picture calculated in step eight, and training the background extraction module, the background feature updating module and the pixel feature extraction module by using the total loss; Step ten: when testing, first, the background region mask of the test picture is extracted by using the background extraction module in step three, then the background region of the test picture is obtained by using the mask, the background region and the initial background feature are input into the background feature updating module in step five to obtain the background feature of the current test picture; at the same time, the test picture is input into the pixel feature extraction model in step six, and the test semantic feature is obtained by using the semantic attention picture output by the pixel feature extraction model; finally, the classification confidence of each pixel point is calculated by using the method in step eight, and the maximum value is taken as the classification result of each pixel point.

2. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, In step one, the VisualObject Classes Challenge 2012 (VOC2012) dataset is used, the training picture has 8256 pictures, the test picture has 1449 pictures, the visible class has 15 classes, and the unseen class has 5 classes.

3. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, In step two, the background extraction module uses the MaskFormer model, which is mainly used for supervised semantic segmentation tasks and can output the background region of the picture after inputting the picture.

4. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The semantic segmentation loss of the background of the training picture in step four is calculated by using a binary cross-entropy loss function.

5. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The initial background feature in step five is the word vector feature of the word "background".

6. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The background feature updating module in the step five is composed of a 2-layer 3*3 convolution layer and a ReLU activation layer, and finally obtains the background feature through global average pooling, and then splices the initial background feature, and obtains the background feature suitable for the semantic segmentation of the current picture in the step five after passing through a 1-layer full connection layer and a ReLU activation layer.

7. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The skeleton model adopted by the pixel feature extraction module in the step six is a deeplabv3 model.

8. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The pixel feature map output by the deeplabv3 in the step six is multiplied with the semantic feature of the visible class to obtain a similarity map, and then the similarity map is input into a 3-layer 3*3 convolution layer and a ReLU layer to output a semantic attention map.

9. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The specific operation of the semantic attention map on the semantic feature in the step seven is that the semantic feature of the visible class number is copied on each pixel point to obtain a semantic feature map, and then the semantic attention map obtained in the step six is multiplied with the copied semantic feature map to obtain a semantic attention semantic feature map after the semantic attention is utilized.

10. The zero-shot semantic segmentation method based on semantic attention mechanism according to claim 1, characterized in that, The similarity between the feature vector and the semantic feature vector is calculated by the cosine similarity in the step eight, and the semantic segmentation loss in the step eight is calculated by a binary cross-entropy loss function.

Images