New class discovery method and system based on semantic information

Abstract

The invention relates to a new class discovery method and system based on semantic information. The method comprises the steps of constructing a clustering branch model and a classification branch model, constructing a visual feature prototype, and synchronously optimizing the clustering branch model and the classification branch model. In order to solve the problem that unknown class labels cannot be obtained, a mechanism for providing pseudo labels for unknown class samples by using a clustering branch model and a known class visual feature prototype is designed, high-confidence samples are selected from output of classification branches, the number of the high-confidence samples contained in each clustering cluster is obtained, and the unknown class labels are obtained. Utilizing Hungary matching to enable the clusters to correspond to the class names; the method for maximizing mutual information of the clustering branch model and the classification branch model is designed for solving the problem that a local optimal solution may fall into synchronous optimization of the clustering branch model and the classification branch model, and synchronous optimization of the two paths of models is ensured. According to the method, the clustering task and the classification task can be completed for the categories at the same time on the premise that no extra label-free category labeling is needed.

Description

Technical Field

[0001] The present invention relates to the field of new class discovery, and particularly to a semantic information-based new class discovery method and system without additional unlabeled class image annotation. Background Art

[0002] New class discovery is an important task in the field of computer vision. Its main goal is to perform clustering tasks on unlabeled class images based on the training of labeled class images and unlabeled class pictures. The new class discovery task requires the model to not only recognize the object classes in the old-class labeled images, but also discover the classes of new unlabeled images. New class discovery has important application values in many fields such as medical image processing and recommendation systems.

[0003] Novel class discovery aims to cluster instances in unlabeled data by leveraging the prior knowledge of known classes. Novel class discovery lies in transferring knowledge from the labeled set to the unlabeled set. The novel class discovery task requires a strong semantic similarity between the labeled and unlabeled classes to group new instances. Han et al. generated pairwise pseudo-labels through ranking statistics (Han K, Rebuffi S A, Ehrhardt S, et al. Autonovel: Automatically discovering and learning novel visual categories[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(10): 6767-6781.). Zhao and Han further improved this method by using local information (Zhao B, Han K. Novel visual category discovery with dual ranking statistics and mutual knowledge distillation[J]. Advances in Neural Information Processing Systems, 2021, 34: 22982-22994.). Zhong and Jia used contrastive learning to learn discriminative representations (Jia X, Han K, Zhu Y, et al. Joint representation learning and novel category discovery on single-and multi-modal data[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 610-619.). Fini et al. proposed a unified training objective function based on self-labeling (Fini E, Sangineto E, Lathuiliere S, et al. A unified objective for novel class discovery[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 9284-9292.).Joseph et al. designed a spacing loss to enforce separability in the latent space (Joseph K J, Paul S, Aggarwal G, et al. Spacing loss for discovering novel categories[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:3761-3766.).

[0004] However, all of the above methods and tasks regard image categories as discrete single-point labels. In other words, existing methods ignore the semantic information of labeled and unlabeled categories, which can provide important inter-category clues for recognition or clustering. At the same time, existing methods can only perform clustering tasks for new categories and not classification tasks, which is lacking in practical applications. Summary of the Invention

[0005] Aiming at the deficiencies of existing methods, the present invention proposes a new method and system for discovering new classes based on semantic information: the method includes: constructing a clustering branch model and a classification branch model, constructing visual feature prototypes, and synchronously optimizing the clustering branch model and the classification branch model.

[0006] To achieve the above objectives, the technical solutions adopted by the present invention are as follows:

[0007] A method for discovering new classes based on semantic information, without additional data annotation, to achieve performance improvement on unlabeled images when there are category differences between the training data (labeled images) of the target model and the actual deployment environment (unlabeled images), including the following steps:

[0008] (1) Pre-training: Given labeled old-category images, use the labeled old-category images to train the visual feature backbone network to make it have basic performance;

[0009] (2) Creation of the clustering branch: Create a clustering branch model, and use a joint loss function to train the clustering branch model. Eventually, the clustering branch model can achieve traditional new class discovery tasks, that is, perform clustering tasks on unlabeled images;

[0010] (3) Creation of the classification branch; Create a classification branch model, construct visual prototypes including labeled images and unlabeled images, and the classification branch model uses the visual prototypes and their corresponding semantic information to classify unlabeled images;

[0011] (4) Joint optimization of the clustering branch model and the classification branch model;

[0012] (5) For the image to be processed, use the jointly optimized clustering branch model and classification branch model to obtain the clustering result and classification result.

[0013] Furthermore, in step (1), use the labeled data to train the vision feature backbone network based on multi-head ResNet to make it have the basic performance of the target domain.

[0014] Furthermore, in step (2), with the parameters of the vision feature backbone network in step (1) as the initial, given the labeled and unlabeled image data, after calculating the probability distribution of the image prediction, introduce a joint loss function. For the labeled data, its supervision is the true label; for the unlabeled data, by using two perspectives of a picture to fake labels for each other as supervision. In this way, the present invention trains a high-confidence clustering branch model.

[0015] Furthermore, in step (3), extract the image features of different categories. For a specific category, select a fixed number of samples, and take the average of their features as the visual prototype. The selection of unlabeled images will be described in detail in the next step. Assume that the visual prototypes of the labeled and unlabeled categories have been obtained. Given an image, after extracting its features, calculate the similarity between its vector and the visual prototype in the visual space respectively, and adjust the similarity in the visual space through the semantic space similarity between different categories. Finally, obtain the similarity that comprehensively considers the visual space and the semantic space, and use it as the final prediction result.

[0016] Furthermore, in step (4), integrate the results of the clustering branch and the classification branch. Specifically, it includes two aspects. On the one hand, assign high-confidence fake labels to the unlabeled samples. First, for the prediction results of the classification branch of each category (when the unlabeled samples are not included), select several samples with the highest confidence; these samples themselves will also be in a clustering result. The number of high-confidence unlabeled category samples contained in each cluster can be obtained, and the Hungarian matching algorithm can be used to obtain the high-probability category name of each cluster. Assign this category name to the entire cluster, and randomly select samples from it and insert them into the visual prototype of the unlabeled category. On the other hand, maximize the mutual information between the probability distributions of the clustering branch and the classification branch to make them influence each other.

[0017] A novel class discovery system based on semantic information adopting the above method, which includes:

[0018] A vision feature backbone network, which is used to extract features from images, adopts a residual convolutional neural network structure, and is shared by the clustering branch model and the classification branch model;

[0019] The clustering branch module is used to create a clustering branch model and cluster new categories using a joint loss.

[0020] The classification branch module is used to create a classification branch model and classify unlabeled new categories using the constructed visual prototypes.

[0021] The joint optimization module is used to simultaneously optimize the clustering branch model and the classification branch model and generate an interactive mechanism.

[0022] The beneficial effects of the present invention are as follows:

[0023] Aiming at the problem of not being able to obtain unknown class labels, the present invention designs a mechanism to provide pseudo-labels for unknown class samples using a clustering branch model and visual feature prototypes of known classes. Select samples with high confidence from the output of the classification branch, obtain the number of samples with high confidence in each clustering cluster, and use the Hungarian matching to correspond the clusters with class names. Aiming at the problem that the clustering branch model and the classification branch model may fall into local optimal solutions during synchronous optimization, a method to maximize the mutual information between the clustering branch model and the classification branch model is designed to ensure the synchronous optimization of the two-way models. The present invention can simultaneously complete the clustering task and the classification task for these categories without the need for additional annotation of unlabeled categories.

[0024] The reason why the present invention has the above-mentioned invention effects is that: on the one hand, the present invention uses language prior knowledge derived from the semantic space to bridge the difference between the semantic space and the visual space; on the other hand, by transferring excellent results in the clustering space to the classification branch, the performance of the classification branch is improved. At the same time, the improved performance of the classification branch will in turn act on the clustering branch, forming a virtuous cycle. Description of the Drawings

[0025] Figure 1 is the flowchart of new class discovery based on semantic information of the present invention. Detailed Embodiment

[0026] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0027] An unsupervised cross-domain object detection method of the present invention has a process as Figure 1 shown and specifically includes the following steps:

[0028] (1) Creation of the clustering branch

[0029] The clustering branch is mainly completed by the clustering module. Based on the baseline model, the present invention constructs a clustering branch model and trains it using a joint unified loss function. Finally, this branch can achieve the traditional new class discovery task, that is, clustering tasks for unlabeled data. During the training process, both labeled data and unlabeled data can be obtained simultaneously. For labeled data, its supervision is the true label; for unlabeled data, false labels are given to each other through two perspectives of a picture as supervision. Denote the number of labeled data categories as C l , denote the number of unlabeled data categories as C u , and finally the output of the clustering branch is a vector with a length of C l +C u dimensions. The present invention uses the cross-entropy loss function to supervise the clustering branch, and denote the loss function of the clustering branch as L cluster .

[0030] In this step, semantic information is not considered in the clustering process. In other words, "clustering" here treats each class as completely homogeneous samples without considering the differences between classes. A separate clustering branch model will correctly classify "cat", "dog", and "bicycle" into three different classes, but it is considered that there is no difference between the distance between "cat" and "dog" and the distance between "cat" and "bicycle". In steps (2), (3), and (4), the new class discovery method based on semantic information proposed by the present invention will improve this.

[0031] (2) Creation of the classification branch

[0032] The present invention proposes a zero-shot classification branch that combines the visual space and the semantic space. For each labeled category, first randomly select K image features, take their mean and put it in the position corresponding to the semantic information label (word vector) to construct a visual prototype. In the inference stage, after extracting the features of a picture, it will calculate the visual similarity with these visual prototypes to obtain the visual space classification score s. At the same time, the present invention constructs a semantic similarity matrix M, and M[i][j] is used to measure the cosine similarity between the word vector of the i-th category and the word vector of the j-th category. The present invention will use this matrix to transfer the similarity in the visual space to the similarity in the semantic space, so as to better complete the classification task.

[0033] Formally, p = σ(s·M), where p is the prediction result combined with the semantic space information, that is, the semantic space classification score.

[0034] Meanwhile, the visual prototype of the present invention is constantly updated and changed during the training process. Its update strategy is a first-in, first-out strategy. Specifically, since the encoder changes during the training process, in order to make the prototype features better reflect the visual vector features generated by the current encoder, the visual prototype needs to be continuously adjusted to be consistent with the encoder. After each training round, the visual features of the newly labeled images in the current round are obtained. These features will replace the visual features in the original positions and become the new visual prototypes. This simple and effective update strategy greatly guarantees the classification performance of the model.

[0035] (3) Joint optimization of the clustering branch and the classification branch

[0036] On the one hand, the present invention jointly uses the results of the clustering branch and the classification branch to generate pseudo-labels for each cluster of unlabeled classes. First, for the prediction results of the classification branch of each class (when there are no unlabeled samples included), select the K samples with the highest confidence. For example, among all the samples predicted as "cat", select the K samples with the highest classification score p for the "cat" class. At the same time, these samples themselves also have a clustering score g, indicating their clustering results. The number of high-confidence unlabeled class samples contained in each cluster can be obtained, and the Hungarian matching algorithm can be used to obtain the class names with high probabilities for each cluster.

[0037] Suppose there are two clusters. In the first cluster, there are five samples with high confidence predicted as "cat" and one sample with high confidence predicted as "dog"; in the second cluster, there are five samples with high confidence predicted as "dog" and one sample with high confidence predicted as "cat". Then it is reasonable to believe that the first cluster is mostly "cat" and the second cluster is mostly "dog". Assign this class name to the entire cluster and randomly select from it to insert into the visual prototype.

[0038] On the other hand, the present invention maximizes the mutual information between the probability distributions of the clustering branch and the classification branch, allowing the two to influence each other. Mutual information can reflect the connection between two distributions. Using mutual information instead of distance measures such as mean squared error to measure these two distributions, the reason is that the clustering results do not correspond one-to-one with specific classes. If the mean squared error measurement method is used, it is likely to fail. This more "soft" supervision method of mutual information is consistent with the idea of the present invention.

[0039] The following experimental results show that compared with the existing methods, the new class discovery method based on semantic information of the present invention can achieve higher clustering accuracy, and at the same time can additionally obtain relatively high classification accuracy under zero-shot prediction.

[0040] The embodiments of this method adopt four publicly available evaluation datasets: CIFAR10, CIFAR100-20, CIFAR100-50, and ImageNet. Among them, in the CIFAR10 evaluation dataset, 5 categories are labeled categories and 5 categories are unlabeled categories, including 25,000 labeled data and 25,000 unlabeled data. In the CIFAR100-20 evaluation dataset, 80 categories are labeled categories and 20 categories are unlabeled categories, including 40,000 labeled data and 10,000 unlabeled data; in the CIFAR100-50 evaluation dataset, 50 categories are labeled categories and 50 categories are unlabeled categories, including 25,000 labeled data and 25,000 unlabeled data; in the ImageNet evaluation dataset, 882 categories are labeled categories and 30 categories are unlabeled categories, including 1,250,000 labeled data and 30,000 unlabeled data.

[0041] Existing method 1: The method proposed by Han et al. in the literature "Automatically Discovering and Learning New Visual Categories with Ranking Statistics."

[0042] Existing method 2: The method proposed by Han et al. in the literature "AutoNovel: Automatically Discovering and Learning Novel Visual Categories."

[0043] Existing method 3: The method proposed by Fini et al. in the literature "A unified objective for novel class discovery."

[0044] The experiment uses clustering accuracy and classification accuracy to evaluate the accuracy of object detection. The clustering accuracy is the maximum possible accuracy found after matching the clustering results with the true labels. The classification accuracy is consistent with traditional classification tasks.

[0045] As can be seen from Table 1, the present invention surpasses the previous methods in terms of performance and achieves significant improvements. On the four datasets of CIFAR10, CIFAR100-20, CIFAR100-50, and ImageNet, the clustering performance has been improved compared with the existing methods. At the same time, the problem that the existing methods cannot classify is solved, and better performance is achieved in the zero-shot classification task.

[0046] Table 1. Comparative experiment results with existing methods (clustering accuracy / classification accuracy, unit: %)

[0047] dataset CIFAR10 CIFAR100-20 CIFAR100-50 ImageNet Existing method 1 90.4 / - 73.2 / - 39.2 / - 82.5 / - Existing method 2 91.7 / - 75.2 / - 44.1 / - 82.5 / - Existing method 3 93.6 / - 90.2 / - 61.0 / - 91.1 / - The present invention 93.8 / 40.1 93.7 / 57.8 62.2 / 21.6 92.5 / 26.5

[0048] The key points of the present invention include:

[0049] (1) The classification branch and its classification strategy for new class discovery based on semantic information;

[0050] (2) The co - optimization mechanism of the clustering branch and the classification branch for new class discovery based on semantic information;

[0051] (3) The visual prototype construction and update strategy for new class discovery based on semantic information.

[0052] Based on the same inventive concept, another embodiment of the present invention provides an unsupervised cross - domain object detection system adopting the above - mentioned method, which includes:

[0053] A visual feature backbone network, which is used to extract features from images, adopts a residual convolutional neural network structure, and is shared by the clustering branch model and the classification branch model;

[0054] A clustering branch module, which is used to create a clustering branch model and cluster new classes by using a joint loss;

[0055] A classification branch module, which is used to create a classification branch model and classify unlabeled new classes by using the constructed visual prototype;

[0056] A joint optimization module, which is used to simultaneously optimize the clustering branch model and the classification branch model and generate an interactive mechanism.

[0057] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (such as a computer, a server, a smart phone, etc.), which includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing each step in the method of the present invention.

[0058] Based on the same inventive concept, another embodiment of the present invention provides a computer - readable storage medium (such as ROM / RAM, a disk, an optical disc). The computer - readable storage medium stores a computer program, and when the computer program is executed by a computer, it realizes each step of the method of the present invention.

[0059] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these changes and modifications.

Claims

1. A new class discovery method based on semantic information, characterized in that, It includes the following steps: Given the old-class images with labels, use the old-class images with labels to train the visual feature backbone network to make it have basic performance; Create a clustering branch model, and use the joint loss function to train the clustering branch model so that the clustering branch model can achieve the clustering task for unlabeled images; Create a classification branch model, construct visual prototypes including labeled images and unlabeled images, and the classification branch model uses the visual prototypes and their corresponding semantic information to classify unlabeled images; Jointly optimize the clustering branch model and the classification branch model; For the image to be processed, use the jointly optimized clustering branch model and classification branch model to obtain the clustering result and the classification result.

2. The method according to claim 1, characterized in that, The visual feature backbone network is a visual feature backbone network based on multi-head ResNet.

3. The method according to claim 1, characterized in that, The creation of the clustering branch model includes: Based on the parameters of the visual feature backbone network, given labeled images and unlabeled images, after calculating the probability distribution of the image prediction, introduce the joint loss function; for labeled data, its supervision is the true label; for unlabeled data, false labels are given to each other from two perspectives of a picture as supervision, so as to train a high-confidence clustering branch model.

4. The method according to claim 1, wherein The creation of the classification branch model includes: Extract the image features of different classes. For a specific class, select a fixed number of samples and take the average of their features as the visual prototype; given an image, after extracting its features, calculate the similarity between its vector and the visual prototype in the visual space respectively, and adjust the similarity in the visual space through the semantic space similarity between different classes. Finally, obtain the similarity considering both the visual space and the semantic space and use it as the final prediction result.

5. The method according to claim 4, characterized in that The visual prototype is constantly updated and changed during the training process, and its update strategy is the first-in first-out strategy; The first-in first-out strategy includes: Continuously adjust the visual prototype during the training process to make it consistent with the encoder. After each training round, the visual features of the new labeled-class images in the current round will be obtained, and these features will replace the visual features in the original position and become the new visual prototype.

6. The method according to claim 1, wherein The joint optimization of the clustering branch model and the classification branch model includes two aspects: On the one hand, use the results of the clustering branch and the classification branch to jointly generate false labels for each cluster of unlabeled classes. For the prediction results of the classification branch model of each class, select several samples with the highest confidence to obtain the number of high-confidence unlabeled class samples included in each cluster, and use the Hungarian matching algorithm to obtain the high-probability class name of each cluster, assign the class name to the whole cluster, and randomly select samples from it to insert into the visual prototype of the unlabeled class; on the other hand, maximize the mutual information between the probability distributions of the clustering branch model and the classification branch model to make them influence each other.

7. A new class discovery system based on semantic information, characterized in that It includes: A visual feature backbone network, which is used to extract features from images, adopts a residual convolutional neural network structure, and is shared by the clustering branch model and the classification branch model; The clustering branch module is used to create a clustering branch model and cluster new categories using a joint loss; The classification branch module is used to create a classification branch model and classify unlabeled new categories using the constructed visual prototypes; The joint optimization module is used to simultaneously optimize the clustering branch model and the classification branch model and generate an interactive mechanism.

8. A computer device, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images