Method, system, device, and medium for zero-shot semantic segmentation based on knowledge distillation

Abstract

Method of training a zero-shot image classifier, comprising: utilizing a pre-trained image encoder to extract regional image features of different objects in an image I1, I2; utilizing a pre-trained text encoder to extract text features of class names S1, S2; and utilizing a semantic segmentation model to derive a feature map wherein the feature map uses the image features for visual supervision and the text features for classification. Training the classifier may further comprise: using information from the regional image features to calculate a first loss using a mean squared error loss function; using the text features as classification weights for image blocks within the feature maps, and calculating a second loss using a cross-entropy loss function based on the segmentation results and a cosine similarity confidence level between image features and text features; summing the first loss and second loss to obtain a total loss; and performing a backpropagation to train the semantic segmentation model. The image encoder and text encoder may be from a CLIP model. The semantic segmentation model may be a DeepLab V3 model. Model testing may comprise using applying the classifier to previously visible classes and previously unseen classes N1, N2.

Description

CROSS-REFERENCE TO RELATED APPLCATIONS This application claims priority under 35 U.S.C. § 119(a) to Chinese Patent Application No. 2022114722385, filed on November 22, 2022, which is hereby incorporated by reference herein in its entirety. TECHNICAL FIELD The present disclosure relates to the field of computer vision, more particularly to a method, a system, a device, and a medium for zero-shot semantic segmentation based on knowledge distillation. BACKGROUND Existing semantic segmentation models can only perform semantic segmentation on images that include classes with labels in the semantic segmentation dataset. If it is desired for the semantic segmentation model to have the ability to recognize more classes, it is necessary to collect images of these classes and annotate the collected images semantically, followed by retraining the model. In the process of collecting data, for some classes, there may also be the problem of being unable to collect a sufficiently large dataset. This also hinders the promotion and application of semantic segmentation models in practice. On the other hand, the Internet is flooded with massive paired text and image data. Now people have used this massive data (more than 400 million pairs of text and images) to train the CLIP (Contrastive Language-Image Pre-training) model (Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on Machine Learning. PMLR, 2021), which includes a text encoder and an image encoder. The most important ability of this method is to classify any class that appears in the image using the text encoder. Although this method is very successful in extracting image features, 25 07 24 how to apply this model to semantic segmentation is still a challenging task. From the above analysis, the problems and shortcomings of the existing technologies are as follows: (1) most of the existing semantic segmentation models find it difficult to extend the number of recognizable classes and must improve the model's generalization capability by collecting more labeled data for additional classes; and (2) most of the existing semantic segmentation models can only perform downstream tasks on models pre-trained on semantic segmentation datasets and cannot leverage knowledge on lower-level datasets, such as image classification datasets. SUMMARY To address the problems in the existing technology, the present disclosure provides a zero-shot semantic segmentation method, system, device, and medium based on knowledge distillation. The disclosure is implemented as follows: a zero-shot semantic segmentation method based on knowledge distillation, wherein the zero-shot semantic segmentation method based on knowledge distillation includes: utilizing a pre-trained image encoder to extract regional image features, serving as visual supervision for a semantic segmentation model; utilizing a pre-trained text encoder to extract text features of class names converted into text, serving as a classification basis for a feature map of the semantic segmentation model, thereby training the semantic segmentation model to achieve image classification. 25 07 24 Further, the specific process of the zero-shot semantic segmentation method based on knowledge distillation are: pre-training the image encoder and the text encoder using both text and image data; collecting a training dataset and test dataset with pixellevel annotations; constructing a semantic segmentation model to extract feature maps of the training dataset; utilizing the image encoder to extract regional image features of different objects within the training dataset, and using the extracted regional image features as supervision for image blocks within the feature map, distilling information from the regional image features to calculate a loss 1; converting labels in the training dataset into a text form, utilizing the text encoder to obtain text features corresponding to the labels of the training dataset, using the obtained text features as classification weights for image blocks within the feature map, and calculating a loss 2; summing the loss 1 and the loss 2 to obtain the total loss, performing a backpropagation, updating and training the semantic segmentation model until model weights stabilize, and performing a model testing. Further, the image encoder is a visual transformer of a ViT-B / 16 model in a CLIP model, comprising 12 layers of neural networks, accepting 224x224 size images as input, and outputting 512-dimensional image features; the text encoder is a text transformer of the ViT-B / 16 model in the CLIP model, comprising 12 layers of neural networks, accepting text as input, and outputting 512-dimensional text features. Further, the training dataset and test dataset are the VOC2012 dataset, wherein the training dataset has 10582 images and the test dataset has 1449 images. Further, the semantic segmentation model adopts a DeepLab V3 model as its backbone network. Further, prior to extracting regional image features, the image encoder utilizes annotation class masks of training images to obtain a region of a certain class within 25 07 24 the training images that has a black background, and then extracts the regional image features. Further, the text form is represented as: an image of the class names. Further, the loss 1 is calculated using a mean squared error loss function, expressed as: ceC where is the feature of the image block belonging to class c within the training images, fc is the regional feature in the feature map belonging to class c, and C is the total number of classes; the loss 2 is calculated by determining the confidence level for each class of the image blocks, selecting the highest confidence level as the classification result for each block, and calculating using a cross-entropy loss function based on the actual semantic segmentation results; the confidence level is obtained by calculating a cosine similarity between the features of the image blocks and the text features for each class. Further, the model testing comprises: inputting the test images into the trained semantic segmentation model to obtain feature maps of the test images; converting the class names of visible and unseen classes during the testing phase into text, and obtaining corresponding text features using the text encoder; after calculating the image features of image blocks in the feature map of the test images and the text features, selecting a maximum value among scores for each class as the classification result for each image block, and integrating all block classification results to obtain the predicted semantic segmentation result for the test images. Another object of the present disclosure is to provide a zero-shot semantic segmentation system based on knowledge distillation, which comprises: an image encoding module for extracting regional image features of different 25 07 24 objects within images; a text encoding module for obtaining text features corresponding to the labels of the images; and a semantic segmentation module for obtaining pixel-level classification results of the images. Another object of the present disclosure is to provide a computing device, the computing device comprises a memory and a processor, wherein the memory stores a computer program which, when executed by the processor, causes the processor to carry out the steps of the zero-shot semantic segmentation method based on knowledge distillation. Another object of the present disclosure is to provide a computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the zero-shot semantic segmentation method based on knowledge distillation. Another object of the present disclosure is to provide an information data processing terminal, the information data processing terminal being utilized for implementing the zero-shot semantic segmentation system based on knowledge distillation. In conjunction with the above technical solution and the technical problems being addressed, the advantages and positive effects possessed by the technical solution sought to be protected by the present disclosure are as follows: Firstly, considering the technical problems existing in the prior art and the difficulty in solving them, the disclosure closely integrates the technical solution sought to be protected, and the outcomes and data obtained during the research and development process, to provide a detailed and profound analysis of how the technical solution of 25 07 24 the disclosure solves these technical problems and the innovative technical effects it achieves. Specific descriptions are as follows: 1) the disclosure uses a pre-trained image encoder to extract regional image features for various classes from the training images, using these as visual supervision for the feature maps obtained through semantic segmentation. The knowledge learned during pre-training is distilled from the image encoder to aid the semantic segmentation model in better feature extraction from the training images; and 2) the disclosure uses a pre-trained text encoder to extract features from class names converted to text, using these as pixel classification weights for the feature maps obtained through semantic segmentation. Secondly, when looking at the technical solution as a whole or from the product perspective, the technical effects and advantages possessed by the technical solution sought to be protected by the present disclosure are as follows: the disclosure employs pre-trained image and text encoders obtained from the CLIP model, trained on a large corpus of text-image paired data, to endow the semantic segmentation model with the capability to recognize unseen classes, thereby expanding the recognition range of the semantic segmentation model. Thirdly, as corroborative evidence of the inventiveness of the claims of this disclosure, additional important aspects are manifested as follows: 1) the expected benefits and commercial value of the technical solution of the disclosure are: by borrowing knowledge learned in the CLIP model and applying it in this disclosure, the disclosure can perform semantic segmentation on images containing unseen classes, offering tremendous commercial value in the field of semantic segmentation. 25 07 24 2) the technical solution of the disclosure fills a technological gap in the Chinese and international industry: if one wishes to endow the semantic segmentation model with the ability to recognize additional classes, it is necessary to collect images of these classes and annotate them semantically for further model training. The disclosure leverages the regional image feature encoder and text encoder obtained through the CLIP method, enabling the semantic segmentation model to perform semantic segmentation on images containing unseen classes, thus filling this technological gap. 3) the technical solution of the disclosure solves a longstanding technical problem that people have desired to solve but have been unable to: most existing semantic segmentation models find it challenging to extend the number of classes they can recognize and must collect more annotated data for these classes to improve the model's generalization capabilities. The disclosure provides a zero-shot semantic segmentation method based on knowledge distillation, utilizing regional image feature and text encoders obtained from the CLIP method to enable the semantic segmentation model to perform semantic segmentation on images containing unseen classes. 4) the technical solution of the disclosure overcomes technical prejudices: in the field of semantic segmentation, improving the model's generalization capability requires collecting more annotated data for additional classes. For some classes, one may even encounter the problem of being unable to collect enough data. The disclosure overcomes this issue by leveraging the knowledge learned by the pre-trained image encoder during its pre-training to better extract regional image features from images. On the other hand, it utilizes the pre-trained text encoder's ability to convert classes into text, endowing the semantic segmentation model with the capability to perform semantic segmentation on unseen classes. 25 07 24 BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram illustrating a training phase according to an embodiment of the present disclosure; FIG. 2 is a schematic diagram illustrating a testing phase according to an embodiment of the present disclosure; and FIG. 3 shows an original image, a semantic segmentation annotated image, and a semantic segmentation prediction image according to an embodiment of the present disclosure, with (a) representing a computer monitor, (b) representing a sofa and chair, and (c) representing a bus. DETAILED DESCRIPTION To make the objectives, technical solutions, and advantages of the present disclosure more clearly understood, a further detailed description of the disclosure is provided below in conjunction with embodiments. It should be understood that the specific embodiments described herein are only intended to explain the disclosure and should not be construed as limiting the disclosure. For those skilled in the art to fully understand how the disclosure is specifically implemented, this section provides an illustrative embodiment that elaborates on the technical solution claimed in the claims. As shown in FIG. 1 and FIG. 2, the zero-shot semantic segmentation method based on knowledge distillation provided by an embodiment of the present disclosure includes the following steps: Step 1: fixing the parameters of the text encoder and image encoder trained on 400 million text-image pairs by the CLIP model. The ViT-B / 16 model is used for both the 25 07 24 text and image encoders in this embodiment. Step 2: using the VOC2012 dataset as the experimental dataset, including 10,582 training images and 1,449 testing images. Step 3: obtaining black-masked image regions by semantic annotation of the training images. Step 4: employing DeepLab V3 as the backbone networkto extract feature maps from the training images. Step 5: utilizing the image encoder from the step 1 to extract regional image features from the region maps obtained in the step 3, using these regional image features to supervise the feature maps extracted in the step 4, and a loss 1 is calculated using a mean square error. Step 6: converting the class labels of the training images into a text form resembling "an image of the class name," and then extracting text features using the text encoder from the step 1. Step 7: utilizing the text features obtained in the step 6 as a basis for pixel classification in the feature maps of the step 4, and calculating a cosine similarity and predicting the class of each pixel based on the highest score, followed by the calculation of pixelwise loss using a cross-entropy to obtain a loss 2. Step 8: calculating the overall loss function based on the loss 1 and the loss 2 from the steps 5 and 7, performing a backpropagation to update the DeepLab V3 model in the step 4, and stable model weights are obtained after several iterations. 25 07 24 Step 9: during the testing phase, as shown in FIG. 2, inputting the test images into the DeepLab V3 semantic segmentation model trained in the step 8 to obtain feature maps of the test images. Step 10: converting the class names of visible and unseen classes during the testing phase into text and obtain corresponding text features using the text encoder from the step 1. Step 11: calculating a cosine similarity between the feature maps obtained in the step 9 and the text features obtained in the step 10. The class with the highest score is taken as the classification for each feature block. Integrate the classification results for all blocks and compare them with the actual semantic annotations of the test images. The final semantic segmentation result is shown in FIG. 3. The first row shows the original images, the second row shows the annotated semantic segmentation, and the third row shows the predicted results using this method. It can be seen that the results obtained by this method are very similar to those obtained by annotations, indicating that the shapes of the objects can be generally recognized. Visible classes, such as the computer monitor in (a) and the sofa in (b), are well segmented. For unseen classes, such as the chair on the right side of (b) and the bus in (c), this method can also perform approximate semantic segmentation using the semantic class relationships present in CLIP. As shown in FIG. 1, Si and S2 represent the visible class text features obtained through the text encoder, N1 and N2 represent the unseen class text features obtained through the text encoder, and h and I2 represent the regional image features of the regional images obtained through the image encoder. In one embodiment of the present disclosure, the text and image encoders from the CLIP model are employed. Specifically, the image encoder is the visual transformer in 25 07 24 the ViT-B / 16 model of the CLIP framework. It comprises 12 layers, takes a 224x224 image as input, and outputs 512-dimensional image features. The text encoder is the textual transformer in the ViT-B / 16 model of the CLIP framework. It also comprises 12 layers, takes text as input, and outputs 512-dimensional text features. The calculation method of the loss 1 provided in this embodiment of the disclosure is as follows: 1,=3½ - / / ceC where ftj represents the features of a specific image block belonging to class c in the training images, and fc represents the features of the regional image in the step 4 that belong to class c. This embodiment of the disclosure leverages pre-trained image and text encoders from the CLIP model, obtained from a large dataset of text-image pairs, to help semantic segmentation models recognize unseen classes, thereby expanding the recognition range of semantic segmentation models. Specifically, on one hand, the pre-trained image encoder's knowledge obtained during pre-training aids the semantic segmentation model in better extracting regional image features from images. On the other hand, the ability of the pre-trained text encoder to convert classes into text grants the semantic segmentation model the capability to perform unseen class semantic segmentation, as exemplified in part (c) of FIG. 3. The zero-shot semantic segmentation system provided by this embodiment of the disclosure specifically includes: an image encoding module, which is used for extracting regional image features of different objects in the images; 25 07 24 a text encoding module, which is used for obtaining text features corresponding to the labels of the images; and a semantic segmentation module, which is used for obtaining pixel-level classification results of the images. To demonstrate the inventive and technical value of this disclosure, this section provides specific embodiments applied to products or relevant technologies based on the claimed technical solutions. It should be noted that embodiments of the present disclosure can be implemented through hardware, software, or a combination of both. The hardware portion can be realized through dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or purpose-designed hardware. It will be understood by those skilled in the art that the aforementioned devices and methods can be implemented using computer-executable instructions and / or processor control code provided on a carrier medium, such as a disk, CD, or DVD-ROM, programmable memory such as Read-Only Memory (firmware), or data carriers like optical or electronic signal carriers. The devices and modules of the present disclosure can be realized by hardware circuits of programmable hardware devices, such as Very Large Scale Integration circuits or gate arrays, semiconductor logic chips, transistors, etc., or can be implemented through software executed by various types of processors, or can be realized through a combination of such hardware circuits and software, such as firmware. The embodiment of the present disclosure has achieved some positive effects in the course of its development or use, demonstrating significant advantages compared to existing technologies. The following content describes these using data and charts from experimental processes. After testing, as shown in Table 1, the evaluation metrics on the VOC2012 test set for the method of the present disclosure show a visible class average Intersection over Union (loU) of 49.0%, an unseen class average loU of 30.2%, and a harmonized loU of 37.4%. These figures are slightly higher than the three metrics obtained by the similar ZS3Net method (Bucher, Maxime, et al. "Zero-shot semantic segmentation." Advances in Neural Information Processing Systems 32 (2019)). The method expands the recognition scope of the semantic segmentation model by leveraging pre-trained image and text encoders from the CLIP model, obtained from a large dataset of textimage pairs. Specifically, it uses the knowledge learned by the pre-trained image encoder during pre-training to help the semantic segmentation model better extract regional image features from images. On the other hand, it uses the ability of the pretrained text encoder to convert classes into text, granting the semantic segmentation model the capability to perform unseen class semantic segmentation. 25 07 24 Table 1 Evaluation Results VOC2012 Visible Class Average loU Unseen Class Average loU Harmonized loU ZS3Net 47.7% 25.2% 33.0% Present Method 49.0% 30.2% 37.4% The foregoing is merely specific embodiments of the present disclosure, and it should be understood that the scope of protection for this disclosure is not limited thereto. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for image semantic segmentation, comprising:5              utilizing a pre-trained image encoder to extract regional image features, serving asvisual supervision for a semantic segmentation model; andutilizing a pre-trained text encoder to extract text features of class names converted into text, serving as a classification basis for a feature map of the semantic segmentation10 model, thereby training the semantic segmentation model to achieve image semantic segmentation,wherein pre-training the image encoder and the text encoder using text-image paired data;15collecting a training dataset with pixel-level annotations and a test dataset with pixellevel annotations;constructing a semantic segmentation model to extract feature maps of the training20 dataset;utilizing the image encoder to extract regional image features of different objects within the training dataset, and using the regional image features as supervision for image blocks within the feature maps, distilling information from the regional image25           features, and calculating a loss 1;converting labels in the training dataset into a text form, utilizing the text encoder to obtain text features corresponding to the labels of the training dataset, using the text features as supervision for image blocks within the feature maps, and calculating a loss30          2; andsumming the loss 1 and the loss 2 to obtain a total loss, performing a backpropagation, updating and training the semantic segmentation model until model weights stabilize, and performing a model testing;wherein the model testing comprises:25 07 24inputting the test images into the trained semantic segmentation model to obtain feature maps of the test images;converting the class names of visible and unseen classes during the testing phase 5 into text, and obtaining corresponding text features using the text encoder; andafter calculating the image features of image blocks in the feature map of the test images and text features, selecting a maximum value among scores for each class as the classification result for each image block, and integrating all block classification10 results to obtain the predicted semantic segmentation result for the test images.

2. The method according to claim 1, wherein the image encoder is a visual transformer of a ViT-B / 16 model in a CLIP model, comprising 12 layers of neural networks, 15          accepting 224x224 size images as input, and outputting 512-dimensional imagefeatures;the text encoder is a text transformer of the ViT-B / 16 model in the CLIP model, comprising 12 layers of neural networks, accepting text as input, and outputting 512-20 dimensional text features; andthe semantic segmentation model adopts a DeepLab V3 model as its backbone network.25             3. The method according to claim 1, wherein prior to extracting regional imagefeatures, the image encoder utilizes annotation class masks of training images to obtain a region of a certain class within the training images that has a black background, and then extracts the regional image features.30             4. The method according to claim 1, wherein the text form consists of converting theclass names of the labels into a text of an image of the class names.

5. The method according to claim 1, wherein the loss 1 is calculated using a mean squared error loss function, expressed as:25 07 24i.=D4- / Ji2ceCwhere is the feature of the image block belonging to class c within the training images, fc is the regional feature in the feature map belonging to class c, and C is the5 total number of classes;the loss 2 is calculated by determining a confidence level for each class of the image blocks, selecting the highest confidence level as the classification result for each block, and calculating using a cross-entropy loss function based on the actual semantic10 segmentation results; andthe confidence level is obtained by calculating a cosine similarity between the features of the image blocks and the text features for each class.

156. An image classification system for implementing the method according to claim1, wherein the image classification system comprises:an image encoding module, which is used for extracting regional image features of 20 different objects within images;a text encoding module, which is used for obtaining text features corresponding to labels of the images; and25             a semantic segmentation module, which is used for obtaining pixel-levelclassification results of the images.

Images