New category discovery method in visual open world based on large model

Abstract

The invention relates to a new category discovery method in a visual open world based on a large model. The method comprises the following steps: dividing a target data set into a labeled data set and a label-free data set; then, a trained CLIP image encoder and a trained CLIP text encoder are finely adjusted by adopting a data set with a label; then extracting image features of unknown class images by using the fine-tuned CLIP image encoder; based on the fine-tuned CLIP text encoder, text features of each unknown class image are encoded; fusing and splicing the image feature and the text feature of each unknown class image to obtain a feature code of each unknown class image in the label-free data set; and finally, performing iterative clustering on the label-free data set based on the feature code of each unknown class image by using a semi-supervised K-means clustering algorithm to obtain a class division result of all images in the label-free data set. According to the method, the text features and the image features of unknown class images are fused to enhance the class discrimination, the feature expression robustness is improved, the long tail deviation problem is relieved, and the clustering accuracy is effectively enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image classification, in particular to a new class discovery method in a visual open world based on a large model. BACKGROUND

[0002] With the development of deep learning technology, image classification tasks have made significant progress. Most existing image classification models assume that all classes encountered during testing have appeared in the training set, which is not true in a real open world. For example, e-commerce platforms need to continuously launch new products, and content review systems need to identify new types of violations. The image classification model obtained under the above assumption is not accurate for unknown class division in a real open environment.

[0003] To solve the problem of inaccurate unknown class division, a new class discovery method is proposed. The goal of the new class discovery method is to simultaneously complete accurate classification of known classes and discovery and division of unknown classes in a mixed image set containing known classes and unknown classes. The new class discovery method has been widely used in fields such as automated product launch, industrial visual inspection of known and unknown defects, and network content review.

[0004] However, existing new class discovery methods rely on a feature extractor pre-trained on a known class dataset. This feature extractor has good discrimination for known classes, but its feature expression is often not robust and generalizable for semantically different unknown classes, resulting in low clustering accuracy. In addition, there is a long-tail bias problem, i.e., the features of new classes with fewer quantities in real data are easily overwhelmed by known classes with more quantities, leading to missed detection of new classes. SUMMARY

[0005] Therefore, it is necessary to provide a new class discovery method in a visual open world based on a large model to address the above technical problems. The text features and image features of unknown class images are fused to enhance class discrimination, improve feature expression robustness, and alleviate the long-tail bias problem, effectively enhancing clustering accuracy.

[0006] The present application provides a new class discovery method in a visual open world based on a large model, comprising the following steps: Divide the target dataset into a labeled dataset and an unlabeled dataset, the unlabeled dataset containing known class images and unknown class images; Fine-tune the trained CLIP image encoder and CLIP text encoder using the labeled dataset; Extract the image features of each unknown class image in the unlabeled dataset using the fine-tuned CLIP image encoder; The text features of each unknown class image in the unlabeled dataset are determined based on the fine-tuned CLIP text encoder. The feature encoding of each unknown class image in the unlabeled dataset is obtained by fusing the image features and text features of each unknown class image in the unlabeled dataset; The semi-supervised K-means clustering algorithm is used to iteratively cluster the unlabeled dataset based on the feature encoding of each unknown class image in the unlabeled dataset, so as to obtain the class classification results of all images in the unlabeled dataset.

[0007] In one embodiment, determining the text features of each unknown class image in the unlabeled dataset based on a fine-tuned CLIP text encoder includes the following steps: For each image of an unknown class in the unlabeled dataset, retrieve K relevant text descriptions from a text corpus; K relevant text descriptions for each unknown class image in the unlabeled dataset are input into the fine-tuned CLIP text encoder to extract K text description features for each image. Average pooling is performed on the K text description features of each unknown class image in the unlabeled dataset to obtain the text features of each unknown class image in the unlabeled dataset.

[0008] In one embodiment, retrieving K relevant text descriptions from a text corpus for each unknown class image in the unlabeled dataset includes the following steps: The CLIP model was used to calculate the similarity between each unknown class image in the unlabeled dataset and all text descriptions in the text corpus. We selected the K text descriptions with the highest similarity to each unknown class image in the unlabeled dataset as relevant text descriptions.

[0009] In one embodiment, a semi-supervised K-means clustering algorithm is used to iteratively cluster the unlabeled dataset based on the feature encoding of each unknown class image in the unlabeled dataset to obtain the class classification results of all images in the unlabeled dataset, including the following steps: Label assignments for known class images in a fixed unlabeled dataset; The feature encodings of all unknown class images in the unlabeled dataset are input into a semi-supervised K-means clustering algorithm. The cluster centers are continuously optimized until the semi-supervised K-means clustering algorithm converges to obtain the clustering results of the unknown class images in the unlabeled dataset. The clustering results of known class images in the unlabeled dataset are obtained by assigning them to unknown class images in the unlabeled dataset according to their labels.

[0010] In one embodiment, 1≤K≤8.

[0011] The beneficial effects of this invention are: (1) The image-text multimodal framework based on CLIP in this invention enhances class discriminability by combining image and text features. By adding the image and text features of CLIP to generate a joint representation, the fine-grained visual information and semantic constraints of the text are preserved. In the new class discovery stage, it can improve the feature discrimination power of the semi-supervised K-means clustering algorithm for unknown classes and enhance the clustering accuracy. (2) The present invention uses the text features of each unknown class image in the unlabeled dataset to enhance the image features, which effectively alleviates the bias problem caused by the long tail distribution in the real data and improves the ability to discover new and rare categories; (3) This invention only requires pre-training and fine-tuning of the CLIP model, which is lightweight. Attached Figure Description

[0012] Figure 1 This is one of the flowcharts of the new category discovery method based on a large model in a visual open world provided in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the image and text feature extraction process provided in one embodiment of the present invention. Figure 3 This invention provides a demonstration of t-SNE results using only image features on the CIFAR-10 dataset, as provided in this embodiment. Figure 4 This invention provides a demonstration of t-SNE results using both image and text features on the CIFAR-10 dataset, as provided in this embodiment. Figure 5 This invention provides the accuracy of clustering for retrieving different numbers of text descriptions from a text corpus, as provided in the embodiments of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0014] In one embodiment, such as Figure 1 As shown, Figure 1 This is one of the flowcharts of a new category discovery method based on a large model in a visual open world provided by an embodiment of the present invention. When applied to a computer device, this method includes the following steps: S1. Divide the target dataset into a labeled dataset and an unlabeled dataset, wherein the unlabeled dataset contains images of known classes and images of unknown classes.

[0015] It should be noted that the labeled dataset includes a plurality of images and their corresponding classification labels, and the known class images of the unlabeled dataset are a plurality, and their corresponding classification labels are also known. The unknown class is not visible in the training process, and its classification label is unknown.

[0016] S2, fine-tuning the trained CLIP image encoder and CLIP text encoder using the labeled dataset.

[0017] CLIP (Contrastive Language-Image Pre-training) is a kind of multimodal model pre-trained by a large amount of image-text data, which can realize the semantic alignment of image and text. Its core idea is to supervise the training of the visual model through natural language, so as to learn a general model that can connect vision and language.

[0018] Specifically, the related parameters of the over-layer training fine-tuning are as follows: the batch size is 128, the gradient update block starts from the 11th block, the training period is 200 epochs, the supervised contrast loss weight is 0.35, and the learning rate is 0.1.

[0019] S3, using the fine-tuned CLIP image encoder to extract image features of each unknown class image in the unlabeled dataset. The image feature extraction and text feature extraction are as shown in Figure 2 .

[0020] S4, determining the text features of each unknown class image in the unlabeled dataset based on the fine-tuned CLIP text encoder. The CLIP text encoder is used to retrieve text descriptions with context as supplementary information for clustering of the unlabeled data.

[0021] Specifically, determining the text features of each unknown class image in the unlabeled dataset based on the fine-tuned CLIP text encoder includes the following steps: S41, retrieving K relevant text descriptions from a text corpus for each unknown class image in the unlabeled dataset.

[0022] The text corpus can be Conceptual Captions (3M), Conceptual Captions (12M), COCO, and LION, etc.

[0023] Among them, retrieving K relevant text descriptions from a text corpus for each unknown class image in the unlabeled dataset includes the following steps: S411, calculating the similarity between each unknown class image in the unlabeled dataset and all text descriptions in the text corpus using the CLIP model; S412, select K text descriptions with the highest similarity to each unknown class image in the unlabeled data set as relevant text descriptions. Specifically, 1≤K≤8.

[0024] S42, input the K relevant text descriptions of each unknown class image in the unlabeled data set into the fine-tuned CLIP text encoder to extract the K text description features of each image.

[0025] S43, average pool the K text description features of each unknown class image in the unlabeled data set to obtain the text feature of each unknown class image in the unlabeled data set.

[0026] S5, fuse and concatenate the image features and text features of each unknown class image in the unlabeled data set to obtain the feature encoding of each unknown class image in the unlabeled data set.

[0027] S6, use a semi-supervised K-means clustering algorithm to iteratively cluster the unlabeled data set based on the feature encoding of each unknown class image in the unlabeled data set to obtain the class division result of all images in the unlabeled data set.

[0028] Specifically, the semi-supervised K-means clustering algorithm is used to iteratively cluster the unlabeled data set based on the feature encoding of each unknown class image in the unlabeled data set to obtain the class division result of all images in the unlabeled data set, including the following steps: S61, fix the label assignment of the known class images in the unlabeled data set; S62, input the feature encoding of all unknown class images in the unlabeled data set into the semi-supervised K-means clustering algorithm, and constantly optimize the clustering center until the semi-supervised K-means clustering algorithm converges to obtain the clustering result of the unknown class images in the unlabeled data set.

[0029] In the semi-supervised K-means clustering algorithm, the known class images calculate the classification loss, the unknown class images calculate the clustering loss, and the sum of the classification loss and the clustering loss is the total loss of the semi-supervised K-means clustering algorithm. The total loss is used to optimize the clustering center.

[0030] S63, assign the known class images in the unlabeled data set to the clustering result of the unknown class images in the unlabeled data set according to the label to obtain the class division result of all images in the unlabeled data set.

[0031] The new class discovery method in the open world based on a large model of the embodiment fuses the image features and text features of each unknown class image in the spliced unlabeled data set to generate joint feature encoding, while retaining the visual fine-grained information and text semantic constraints, so as to improve the feature discrimination of the semi-supervised K-means clustering algorithm for unknown classes in the new class discovery stage and enhance the clustering effect. And it can effectively alleviate the deviation problem caused by the long-tail distribution in real data.

[0032] In a specific embodiment, in order to verify the method of the application, the embodiment uses a general image classification benchmark test on the ImageNet-100 and CIFAR-100 target data sets, takes K equal to 4, and compares the method of the application with the existing most advanced RankStats+, UNO+ and XCon methods. RankStats+ and UNO+ are two competitive baseline methods for NCD (new class discovery), and are modified under the GCD (generalized class discovery) setting. XCon is a method for the GCD setting. The results in the general image recognition benchmark test are shown in Table 1.

[0033] As can be seen from Table 1, the method of the application performs best on most classes, with the highest clustering or classification accuracy, and greatly improves over the comparative methods. The method of the application is superior to other methods on all subsets, confirming that the dual use of the image-text CLIP multimodal model enhances performance compared to using only a visual model. The old classes are known classes.

[0034] Table 1 Comparison results of general image recognition data sets An ablation experiment is performed on the method of the application, and the experimental results are shown in Table 2, which compares the clustering accuracy of using different image encoders and whether to use a model using only visual features or visual and text features. DINO indicates that the DINO pre-training weights from GCD are used, while CLIP indicates that the pre-trained ViT-B / 16 backbone network is used. The results show that the image and text features using CLIP are significantly better than using only image features, thereby confirming that the introduction of language information can significantly improve performance compared to only image models. In addition, the retrieval mechanism, i.e. using the cross-modal joint embedding of CLIP for text retrieval, further improves performance.

[0035] Table 2 Image and text clustering accuracy of only image and different image encoders For the contrastive model, the learned representation is influenced by the text encoder, which suggests that text descriptions that describe the image scene content will improve the CLIP model’s transferability. This hypothesis is verified in this embodiment and multiple text corpora are used to quantify the descriptiveness of the text descriptions. This embodiment performs top-k=4 cross-modal retrieval on multiple text corpora, Conceptual Captions (3M), Conceptual Captions (12M), COCO, and LION, as shown in Table 3, which records the accuracy of the model on all classes, old classes, and new classes subsets on each data corpus. As can be seen from Table 3, Conceptual Captions (3M) improves the transferability of the CLIP model the most, and the accuracy of the model is the highest.

[0036] Table 3. Accuracy of the model using different knowledge bases as text corpora This embodiment further demonstrates the results of t-SNE projections using only image features and simultaneously using image and text features in the CIFAR-10 dataset, which are the clustering features of the unlabeled dataset, as shown in Figure 3 and Figure 4 , respectively, Figure 3 is a t-SNE result display using only image features on the CIFAR-10 dataset, Figure 4 is a t-SNE result display using image and text features simultaneously on the CIFAR-10 dataset. As can be seen from Figure 3 and Figure 4 , for the case of using only image features, data points of the same class are usually projected near each other, forming clear clusters, but there is some overlap between classes. In contrast, when using image and text features simultaneously, the image-text features form clear clusters and have more obvious separation, further confirming the effectiveness of the method of increasing text features.

[0037] As shown in Figure 5 , the clustering accuracy of all classes, known classes, and new classes corresponding to different numbers of text descriptions retrieved from the text corpus is shown in Figure 5 In this embodiment, the sensitivity of the model to the number of text descriptions retrieved for each image is also examined, and the clustering accuracy of all classes, known classes, and new classes using Conceptual Captions (12M) on the CIFAR100 dataset is shown. As can be seen from Figure 5 , when K is equal to 4, the clustering accuracy of all classes, known old classes, and new classes is the highest.

[0038] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, several modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for discovering new categories in a visual open world based on a large model, characterized in that, Includes the following steps: The target dataset is divided into a labeled dataset and an unlabeled dataset, wherein the unlabeled dataset contains images of known classes and images of unknown classes; The CLIP image encoder and CLIP text encoder, trained using a labeled dataset, were fine-tuned. The fine-tuned CLIP image encoder was used to extract image features for each unknown class image in the unlabeled dataset; The text features of each unknown class image in the unlabeled dataset are determined based on the fine-tuned CLIP text encoder. The feature encoding of each unknown class image in the unlabeled dataset is obtained by fusing the image features and text features of each unknown class image in the unlabeled dataset; The semi-supervised K-means clustering algorithm is used to iteratively cluster the unlabeled dataset based on the feature encoding of each unknown class image in the unlabeled dataset, so as to obtain the class classification results of all images in the unlabeled dataset.

2. The method for discovering new categories in a visual open world based on a large model according to claim 1, characterized in that, The text features of each unknown class image in the unlabeled dataset are determined based on the fine-tuned CLIP text encoder, including the following steps: For each image of an unknown class in the unlabeled dataset, retrieve K relevant text descriptions from a text corpus; K relevant text descriptions for each unknown class image in the unlabeled dataset are input into the fine-tuned CLIP text encoder to extract K text description features for each image. Average pooling is performed on the K text description features of each unknown class image in the unlabeled dataset to obtain the text features of each unknown class image in the unlabeled dataset.

3. The method for discovering new categories in a visual open world based on a large model according to claim 2, characterized in that, For each image of an unknown class in the unlabeled dataset, retrieve K relevant text descriptions from a text corpus, including the following steps: The CLIP model was used to calculate the similarity between each unknown class image in the unlabeled dataset and all text descriptions in the text corpus. We selected the K text descriptions with the highest similarity to each unknown class image in the unlabeled dataset as relevant text descriptions.

4. The method for discovering new categories in a visual open world based on a large model according to claim 3, characterized in that, The semi-supervised K-means clustering algorithm is used to iteratively cluster the unlabeled dataset based on the feature encoding of each unknown class image in the unlabeled dataset, to obtain the class classification results of all images in the unlabeled dataset. The steps include: Label assignments for known class images in a fixed unlabeled dataset; The feature encodings of all unknown class images in the unlabeled dataset are input into a semi-supervised K-means clustering algorithm. The cluster centers are continuously optimized until the semi-supervised K-means clustering algorithm converges to obtain the clustering results of the unknown class images in the unlabeled dataset. The clustering results of known class images in the unlabeled dataset are obtained by assigning them to unknown class images in the unlabeled dataset according to their labels.

5. The method for discovering new categories in a visual open world based on a large model according to claim 3, characterized in that, 1≤K≤8。

Images