Zero-shot semantic segmentation method, system, device and medium based on knowledge distillation

Abstract

The application belongs to the technical field of computer vision, and discloses a zero-shot semantic segmentation method, system, device and medium based on knowledge distillation, comprising the following steps: using a pre-trained picture encoder to extract the region picture features of each class in a training picture, taking the picture features as visual supervision of a picture feature map extracted by a semantic segmentation model, and applying visual knowledge distillation in the picture encoder to the training of the semantic segmentation model; using a pre-trained text encoder to extract the text features of a class name converted into text, taking the text features as classification basis of the feature map of the semantic segmentation model; and classifying each pixel point of the picture according to the score of each class by the semantic segmentation model. The pre-trained picture encoder and text encoder obtained by the CLIP model under a large amount of text picture pairing data help the semantic segmentation model to have the ability to identify unseen classes, and expand the recognition range of the semantic segmentation model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a zero-sample semantic segmentation method, system, device and medium based on knowledge distillation. Background Technology

[0002] Existing semantic segmentation models can only perform semantic segmentation on images containing labeled categories in the semantic segmentation dataset. To enable these models to recognize more categories, it's necessary to collect images of those categories, semantically annotate them, and then retrain the model. During data collection, there may be issues with collecting sufficient datasets for certain categories. This hinders the widespread adoption and application of semantic segmentation models in reality. However, the internet is teeming with massive amounts of paired text and image data. Researchers have used this vast amount of data (over 400 million text-image pairs) to train the CLIP (Contrastive Language-Image Pre-training) model (Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021). This model includes a text encoder and an image encoder. The most important capability of this method is its ability to classify any category appearing in an image using the text encoder. While this method is very successful in extracting image features, applying this model to semantic segmentation remains a challenging task.

[0003] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0004] (1) Most current semantic segmentation models have difficulty expanding the number of recognized classes and can only improve the generalization ability of the model by collecting more labeled data of classes.

[0005] (2) Most current semantic segmentation models can only perform downstream work on models pre-trained on semantic segmentation datasets and cannot utilize knowledge from lower-level datasets, such as image classification datasets. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a zero-sample semantic segmentation method, system, device, and medium based on knowledge distillation.

[0007] This invention is implemented as follows: a zero-shot semantic segmentation method based on knowledge distillation, wherein the zero-shot semantic segmentation method based on knowledge distillation includes:

[0008] A pre-trained image encoder is used to extract region image features as visual supervision for the semantic segmentation model; a pre-trained text encoder is used to extract text features that are converted into category names of text, which are used as the classification basis of the feature map of the semantic segmentation model, and then the semantic segmentation model is trained to achieve image classification.

[0009] Furthermore, the specific process of the zero-shot semantic segmentation method based on knowledge distillation is as follows:

[0010] A pre-trained image encoder and a text encoder are built using text and image data. Training and testing datasets with pixel-level annotations are collected. A semantic segmentation model is constructed to extract feature maps from the training dataset. The image encoder extracts region image features of different objects in the training dataset, which are used as supervision for image patches in the feature maps. Information from the region image features is distilled, and loss 1 is calculated. The labels in the training dataset are converted into text, and the text encoder is used to obtain the text features of the corresponding labels in the training dataset, which are used as classification weights for image patches in the feature maps, and loss 2 is calculated. The overall loss is calculated based on loss 1 and loss 2, backpropagated, and the semantic segmentation model is updated and trained until the model weights stabilize. The model is then tested.

[0011] Furthermore, the image encoder is a visual converter of the ViT-B / 16 model in the CLIP model, which has 12 layers of neural network. The input is a 224×224 image and the output is 512-dimensional image features.

[0012] The text encoder is a text converter of the ViT-B / 16 model in the CLIP model. It has 12 layers of neural network, with text as input and 512-dimensional text features as output.

[0013] Furthermore, the training dataset and the test dataset are the VOC2012 dataset, with the training dataset containing 10,582 images and the test dataset containing 1,449 images.

[0014] Furthermore, the semantic segmentation model uses the DeepLab V3 model as the backbone network.

[0015] Furthermore, before extracting region image features, the image encoder uses the labeled category mask of the training image to obtain a region image of a certain class in the training image with a black background, and then extracts the region image features.

[0016] Furthermore, the text form is represented as: an image with a category name.

[0017] Furthermore, the loss 1 is calculated using the squared difference loss function, expressed as:

[0018]

[0019] In the formula, f i.j It trains the features of image patches belonging to class c in the image, f c It is a region map feature that belongs to class c, where C is the total number of classes;

[0020] The loss 2 is calculated by obtaining the confidence scores of various types of the image blocks, taking the highest confidence score as the classification result of each block, and calculating it using the cross-entropy loss function based on the actual semantic segmentation results.

[0021] The confidence level is obtained by calculating the cosine similarity between image patch features and text features of various types.

[0022] Furthermore, the model testing includes:

[0023] The test image is input into the trained semantic segmentation model to obtain the feature map of the test image; the category names of the visible and unseen classes in the test phase are converted into text, and the corresponding text features are obtained using the text encoder; after calculating the image features and text features of the image blocks in the feature map of the test image, the maximum value of each category score is used as the classification result of the image block, and the classification results of all blocks are integrated to obtain the predicted semantic segmentation result of the test image.

[0024] Another object of the present invention is to provide a zero-shot semantic segmentation system based on knowledge distillation, the zero-shot semantic segmentation system based on knowledge distillation comprising:

[0025] The image encoding module is used to extract regional image features of different objects in an image;

[0026] The text encoding module is used to obtain the text features of the corresponding labels for the images;

[0027] The semantic segmentation module is used to obtain pixel-level classification results for images.

[0028] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the knowledge distillation-based zero-sample semantic segmentation method.

[0029] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the knowledge distillation-based zero-sample semantic segmentation method.

[0030] Another objective of this invention is to provide an information data processing terminal for implementing the knowledge distillation-based zero-sample semantic segmentation system.

[0031] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0032] First, addressing the technical problems existing in the prior art and the difficulty in solving them, this paper closely analyzes, in conjunction with the technical solution to be protected by this invention and the results and data obtained during the research and development process, how the technical solution of this invention solves the technical problems, and the inventive technical effects brought about by solving these problems. The specific description is as follows:

[0033] (1) This invention utilizes a pre-trained image encoder to extract regional image features of various types in training images, and uses them as visual supervision for the feature maps extracted by semantic segmentation. The knowledge learned by the image encoder during pre-training is distilled from the image encoder to help the semantic segmentation model extract features of the training images better.

[0034] (2) This invention utilizes a pre-trained text encoder to extract the features of the category names converted into text, and uses them as the pixel classification weights of the feature map extracted by semantic segmentation.

[0035] Second, considering the technical solution as a whole or from a product perspective, the technical effects and advantages of the technical solution to be protected by this invention are specifically described as follows:

[0036] This invention utilizes pre-trained image encoders and text encoders obtained from a large amount of text-image pairing data using the CLIP model to help semantic segmentation models have the ability to recognize unseen classes, thus expanding the recognition range of semantic segmentation models.

[0037] Third, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0038] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0039] This invention borrows knowledge learned from the CLIP model and applies it to the invention, enabling it to perform semantic segmentation on images containing unseen classes, which has great commercial value in the field of semantic segmentation.

[0040] (2) The technical solution of this invention fills a technical gap in the industry both domestically and internationally:

[0041] To enable semantic segmentation models to recognize more categories, images of these categories need to be collected, semantically labeled, and then the model retrained. This invention utilizes the visual feature encoder and text encoder obtained from the CLIP method to help semantic segmentation models achieve semantic segmentation of images containing unseen categories, filling this technological gap.

[0042] (3) Whether the technical solution of the present invention solves the technical problem that people have long wanted to solve but have never been able to solve successfully:

[0043] Most current semantic segmentation models struggle to expand the number of classes they can recognize, and they must collect more labeled data for each class to improve their generalization ability. This invention provides a zero-shot semantic segmentation method based on knowledge distillation, which uses a visual feature encoder and a text encoder obtained from the CLIP method to help the semantic segmentation model achieve semantic segmentation of images containing unseen classes.

[0044] (4) Does the technical solution of the present invention overcome technical bias?

[0045] In the field of semantic segmentation, improving the generalization ability of a model requires collecting more labeled data for various classes. For some categories, there may be a problem of not being able to collect enough datasets. This invention addresses this by leveraging the knowledge learned during pre-training of a pre-trained image encoder to help the semantic segmentation model better extract visual features from images. Furthermore, it utilizes the ability of a pre-trained text encoder to convert categories into text, thus enabling semantic segmentation to perform semantic segmentation of unseen classes. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the training phase provided in an embodiment of the present invention;

[0047] Figure 2 This is a flowchart illustrating the testing phase provided in an embodiment of the present invention;

[0048] Figure 3 These are the original image, semantic segmentation annotation image, and semantic segmentation prediction image provided in the embodiments of the present invention: (a) computer monitor, (b) sofa and chair, and (c) bus. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0050] To enable those skilled in the art to fully understand how the present invention is specifically implemented, this section provides an explanatory description of the embodiments that expand upon the technical solutions of the claims.

[0051] like Figure 1 and Figure 2 As shown, the zero-shot semantic segmentation method based on knowledge distillation provided in this embodiment of the invention includes:

[0052] Step 1: Fix the parameters of the text encoder and image encoder trained by the CLIP model using 400 million pairs of text and image data. This paper uses the text encoder and image encoder from its ViT-B / 16 model.

[0053] Step 2: Use the VOC2012 dataset as the experimental dataset for this method, with 10,582 images in the training dataset and 1,449 images in the test dataset.

[0054] Step 3: Use the semantic annotations of the training images to obtain the image region with a black background after masking.

[0055] Step 4: Use deeplabv3 as the backbone network of this method to extract feature maps from the training images.

[0056] Step 5: Use the image encoder in Step 1 to extract the visual features of the region map obtained in Step 3, and use these visual features to supervise the feature map extracted in Step 4. Calculate the loss using the squared difference loss.

[0057] Step 6: Convert the category labels of the training images into text in the form of "an image with a category name", and then use the text encoder in Step 1 to extract text features.

[0058] Step 7: Use the text features from Step 6 as the classification basis for the pixels of the feature map in Step 4, calculate the cosine similarity, obtain the prediction result of each pixel based on the highest score, and then use cross-entropy to calculate the loss of each pixel, resulting in a loss of 2.

[0059] Step 8: Calculate the total loss function based on loss 1 and loss 2 from steps 5 and 7, then backpropagate to update the DeepLab V3 model in step 4. After multiple iterations, a stable model weight is obtained.

[0060] Step Nine: Testing Phase, such as Figure 2 As shown, the test image is input into the DeepLab V3 semantic segmentation model obtained after training in step eight to obtain the feature map of the test image.

[0061] Step 10: Convert the category names of the visible and unseen classes in the testing phase into text, and use the text encoder in Step 1 to obtain the corresponding text features.

[0062] Step 11: Calculate the cosine similarity between the feature maps of the test image obtained in Step 9 and the text features obtained in Step 10. Then, select the class with the highest class score as the classification for each feature block. Integrate the classification results of all blocks and compare them with the actual semantic annotations of the test image. The final semantic segmentation result is shown in the image below. Figure 3 As shown in the image, the first row is the original image, the second row is the semantic segmentation annotation image, and the third row is the prediction result image of this method. The results show that the results obtained by this method are very similar to the annotation results, and it can basically identify the shape of the object. The computer monitor in (a) and the sofa in (b) are visible classes, and this method can perform semantic segmentation well. For unseen classes (such as the chair on the right in (b) and the bus in (c), this method can also perform approximate semantic segmentation through the semantic category relationships carried in CLIP.

[0063] like Figure 1 As shown, S1 and S2 represent the visible text features obtained by the text encoder.

[0064] N1 and N2 represent the unseen text features obtained by the text encoder, and I1 and I2 represent the visual features of the region map obtained by the image encoder.

[0065] The embodiments of this invention provide a text encoder and an image encoder using the CLIP model. The image encoder is a visual transducer of the ViT-B / 16 model in the CLIP model, which has 12 layers. The input is a 224×224 image, and the output is 512-dimensional image features. The text encoder is a text transducer of the ViT-B / 16 model in the CLIP model, which has 12 layers. The input is text, and the output is 512-dimensional text features.

[0066] The calculation method for loss 1 provided in this embodiment of the invention is as follows:

[0067]

[0068] Where f i,j It trains the features of a specific image patch belonging to class c in the image, f c It is a region map feature that belongs to class c in step four.

[0069] This invention utilizes pre-trained image and text encoders obtained from a large amount of text-image pairing data using the CLIP model to enhance the semantic segmentation model's ability to identify unseen classes, thus expanding the model's recognition range. Specifically, on one hand, the pre-trained image encoder leverages the knowledge learned during pre-training to help the semantic segmentation model better extract visual features from images. On the other hand, the pre-trained text encoder's ability to convert categories into text endows the semantic segmentation model with the capability to perform semantic segmentation of unseen classes. An example of this segmentation is shown below. Figure 3 As shown in (c).

[0070] The zero-shot semantic segmentation system based on knowledge distillation provided in this invention specifically includes:

[0071] Image encoding module: used to extract regional image features of different objects in an image;

[0072] Text encoding module: used to obtain the text features of the labels corresponding to the images;

[0073] Semantic segmentation module: Used to obtain pixel-level classification results for images.

[0074] To demonstrate the inventiveness and technical value of the technical solution of this invention, this section provides specific product or related technology application examples of the technical solution claimed.

[0075] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or using software executed by various types of processors, or using a combination of the above-described hardware circuitry and software, such as firmware.

[0076] The embodiments of the present invention have achieved some positive results during the research and development or use process, and have indeed great advantages compared with the prior art. The following content describes the experimental process with data, charts and other information.

[0077] As shown in Table 1, the evaluation metrics of the method of this invention on the VOC2012 test set are as follows: the average intersection-union ratio (IU) for visible classes is 49.0%, the average IU for unseen classes is 30.2%, and the coordinated IU is 37.4%, which are slightly higher than the three metrics obtained by the similar ZS3Net method (Bucher, Maxime, et al. "Zero-shot semantic segmentation." Advances in Neural Information Processing Systems 32 (2019)). The method utilizes the pre-trained image encoder and text encoder obtained from a large amount of text-image pairing data using the CLIP model to help the semantic segmentation model recognize unseen classes, thus expanding the recognition range of the semantic segmentation model. Specifically, on the one hand, the knowledge learned by the pre-trained image encoder during pre-training helps the semantic segmentation model better extract the visual features of images. On the other hand, the ability of the pre-trained text encoder to convert categories into text is used to endow the semantic segmentation model with the ability to perform semantic segmentation of unseen classes.

[0078] Table 1 Evaluation Results

[0079] VOC2012 Visible class average intersection-union ratio No class average intersection ratio Coordinated intersection and merging ratio ZS3Net 47.7％ 25.2％ 33.0％ This method 49.0％ 30.2％ 37.4％

[0080] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A zero-shot semantic segmentation method based on knowledge distillation, characterized in that, The zero-shot semantic segmentation method based on knowledge distillation includes: A pre-trained image encoder is used to extract region image features as visual supervision for the semantic segmentation model; a pre-trained text encoder is used to extract text features that are converted into category names of text as the classification basis of the feature map of the semantic segmentation model, and then the semantic segmentation model is trained to achieve image classification. The test image is input into the trained semantic segmentation model to obtain the feature map of the test image; the category names of the visible and unseen classes in the test phase are converted into text, and the corresponding text features are obtained using the text encoder; after calculating the image features and text features of the image blocks in the feature map of the test image, the maximum value of each category score is used as the classification result of the image block, and the classification results of all blocks are integrated to obtain the predicted semantic segmentation result of the test image.

2. The zero-sample semantic segmentation method based on knowledge distillation as described in claim 1, characterized in that, The specific process of the zero-shot semantic segmentation method based on knowledge distillation is as follows: Pre-trained image encoder and text encoder are used with text and image data; training dataset and test dataset with pixel-level annotation are collected; a semantic segmentation model is constructed to extract feature maps from the training dataset; the image encoder is used to extract regional image features of different objects in the training dataset, and these features are used as supervision for image blocks in the feature maps; information in the regional image features is distilled out, and loss 1 is calculated. The labels in the training dataset are converted into text form. The text encoder is used to obtain the text features of the corresponding labels in the training dataset, which are used as the classification weights of the image blocks in the feature map. Loss 2 is calculated. Loss 1 and loss 2 are summed to obtain the overall loss. This is backpropagated to update and train the semantic segmentation model until the model weights are stable. The model is then tested.

3. The zero-sample semantic segmentation method based on knowledge distillation as described in claim 2, characterized in that, The image encoder is a visual converter of the ViT-B / 16 model in the CLIP model. It has 12 layers of neural network. The input is a 224×224 image and the output is 512-dimensional image features. The text encoder is a text converter of the ViT-B / 16 model in the CLIP model, which has 12 layers of neural network. The input is text and the output is 512-dimensional text features. The semantic segmentation model uses the DeepLab V3 model as its backbone network.

4. The zero-sample semantic segmentation method based on knowledge distillation as described in claim 2, characterized in that, Before extracting region image features, the image encoder uses the labeled category mask of the training image to obtain a region image of a certain class in the training image with a black background, and then extracts the region image features.

5. The zero-sample semantic segmentation method based on knowledge distillation as described in claim 2, characterized in that, The text format refers to converting the category name of the label into text that represents an image of the category name.

6. The zero-shot semantic segmentation method based on knowledge distillation as described in claim 2, characterized in that, The loss 1 is calculated using the squared difference loss function, expressed as: In the formula, f i.j It trains the features of image patches belonging to class c in the image. f c It is a region map feature whose feature map belongs to class c. C Total number of categories; The loss 2 is calculated by obtaining the confidence scores of various types of the image blocks, taking the highest confidence score as the classification result of each block, and calculating it using the cross-entropy loss function based on the actual semantic segmentation results. The confidence level is obtained by calculating the cosine similarity between image patch features and text features of various types.

7. A zero-sample semantic segmentation system based on knowledge distillation as described in any one of claims 1-6, characterized in that, The zero-sample semantic segmentation system based on knowledge distillation includes: Image encoding module: used to extract regional image features of different objects in an image; Text encoding module: used to obtain the text features of the labels corresponding to the images; Semantic segmentation module: Used to obtain pixel-level classification results for images.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the zero-sample semantic segmentation method based on knowledge distillation as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of the knowledge distillation-based zero-sample semantic segmentation method as described in any one of claims 1-6.

Images