Generalized class discovery model training method and device, generalized class discovery model recognition method and device, equipment and medium

Abstract

The embodiment of the invention provides a generalized class discovery model training method and device, an image recognition method and device, equipment and a medium, and the method comprises the steps: respectively inputting a marked image and an unmarked image in a sample image into a class knowledge guide sample similarity learning module of a generalized class discovery model, obtaining a first similarity matrix between the marked image and the unmarked image; inputting label information of the marked image into a category similarity module of a generalized category discovery model to obtain a similarity score of a label category; inputting the sample image into a label estimation module of the generalized class discovery model to obtain a clustering result of the sample image; and training a prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges, thereby solving the problem of low generalization ability of the target classifier to the new class of the image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a generalized class discovery model training method, an image recognition method, device, electronic device and storage medium. Background Art

[0002] In recent years, machine learning has been widely used in natural language processing, computer vision, speech recognition and other fields. In the field of computer vision, image classification is one of the most popular and widely used tasks. Various classification technologies emerge in an endless stream and their performance continues to improve.

[0003] In machine learning tasks, supervised learning methods that use a large number of manually annotated images to achieve classification are traditional methods of image classification. Since traditional methods mainly rely on visual information to complete model training, they ignore the semantic information contained in the image category itself, thereby reducing the classifier's generalization ability to new categories. Summary of the invention

[0004] The technical problem to be solved by the embodiments of the present application is to provide a generalized class discovery model training method, an image recognition method, device, electronic device and storage medium to solve the problem of low generalization ability of the target classifier for new image classes.

[0005] In a first aspect, the present application provides a method for training a generalized class discovery model, comprising:

[0006] Inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model respectively, and obtaining a first similarity matrix between the labeled image and the unlabeled image;

[0007] Inputting the label information of the labeled image into a category similarity module of a generalized category discovery model to obtain similarity scores between label categories;

[0008] Inputting the sample image into the label estimation module of the generalized class discovery model to obtain a clustering result of the sample image;

[0009] The prototype network of the generalized class discovery model is trained according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges.

[0010] Optionally, the step of inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model to obtain a first similarity matrix between the labeled image and the unlabeled image comprises:

[0011] Inputting the labeled image and the unlabeled image into the feature extraction network respectively to obtain a first embedding vector corresponding to the labeled image and a second embedding vector corresponding to the unlabeled image;

[0012] The first embedding vector and the second embedding vector are input into a sample relationship network to obtain a first similarity matrix between the labeled image and the unlabeled image.

[0013] Optionally, inputting the label information of the marked image into a category similarity module of a generalized category discovery model to obtain similarity scores between label categories includes:

[0014] Obtaining a semantic description vector corresponding to the label information of the marked image;

[0015] The semantic description vector is input into the sample relationship network to obtain the similarity scores between the label categories.

[0016] Optionally, the step of inputting the sample image into a label estimation module of the generalized class discovery model to obtain a clustering result of the sample image comprises:

[0017] Re-representing the sample image to obtain a labeled sample image;

[0018] Inputting the labeled sample image into the feature extraction network to obtain a third embedding vector corresponding to the labeled sample image;

[0019] Inputting the third embedding vector into a sample relationship network to obtain a second similarity matrix between the labeled sample image and the labeled image;

[0020] The second similarity matrix is ​​clustered using a semi-supervised K-means clustering algorithm to obtain a clustering result, wherein the clustering result includes: the number of clusters of unlabeled images, the number of clusters of labeled images, and pseudo labels of unlabeled images.

[0021] Optionally, the prototype network includes: a known category prototype of the labeled image in the sample image and a new category prototype of the sample image, wherein the semantic description vector corresponding to the label information of the labeled image is used as the known category prototype of the prototype network, and the third embedding vector is used as the new category prototype;

[0022] The step of training the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix, and the similarity score until the generalized class discovery model converges comprises:

[0023] Determining a category similarity loss value according to the second similarity matrix and the similarity score;

[0024] Determining a clustering loss value based on the second embedding vector of the labeled image and the known category prototype of the labeled image;

[0025] Inputting the unlabeled image into the prototype network to obtain a matching score;

[0026] Determine a classification loss value based on the matching score, the pseudo label and the label information of the labeled image;

[0027] The network parameters of the prototype network are updated according to the class similarity loss value, the clustering loss value and the classification loss value until the generalized class discovery model converges.

[0028] In a second aspect, an embodiment of the present application provides an image recognition method, including:

[0029] Obtaining a pre-trained generalized class discovery model, wherein the generalized class discovery model is trained by the training method;

[0030] Inputting an unlabeled image in the sample image into a feature extraction network of a generalized class discovery model to obtain a fourth embedding vector of the unlabeled image;

[0031] Inputting the fourth embedding vector into the prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image, wherein the prototype network includes: known category prototypes of the labeled image in the sample image and new category prototypes of the sample image;

[0032] Determine predicted label information of the unlabeled image according to the matching score.

[0033] Optionally, inputting the fourth embedding vector into a prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image comprises:

[0034] A first matching score between the unlabeled image and the known category prototype and a second matching score between the unlabeled image and the new category prototype are calculated.

[0035] Optionally, determining the predicted label information of the unlabeled image according to the matching score includes:

[0036] determining a maximum matching score from the first matching score and the second matching score by maximizing a likelihood function;

[0037] The category prototype corresponding to the maximum matching score is used as the predicted label information of the unlabeled image.

[0038] In a third aspect, an embodiment of the present application provides a training device for a generalized class discovery model, including:

[0039] A class knowledge learning module, used for inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model respectively, to obtain a first similarity matrix between the labeled image and the unlabeled image;

[0040] A category module, used for inputting the label information of the labeled image into a category similarity module of a generalized category discovery model to obtain similarity scores between label categories;

[0041] A label module, used for inputting the sample image into the label estimation module of the generalized class discovery model to obtain a clustering result of the sample image;

[0042] A training module is used to train the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges.

[0043] In a fourth aspect, an embodiment of the present application provides an image recognition device, including:

[0044] An acquisition module, used to acquire a pre-trained generalized class discovery model, wherein the generalized class discovery model is trained by the training method;

[0045] An input module, used for inputting an unlabeled image in the sample image into a feature extraction network of a generalized class discovery model to obtain a fourth embedding vector of the unlabeled image;

[0046] A matching score module, configured to input the fourth embedding vector into a prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image, wherein the prototype network includes: known category prototypes of the labeled image in the sample image and new category prototypes of the sample image;

[0047] A label module is used to determine the predicted label information of the unlabeled image according to the matching score.

[0048] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0049] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the training method or the image recognition method is implemented when the processor executes the computer program.

[0050] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that implements the training method or the image recognition method when executed by a processor.

[0051] Compared with the prior art, the embodiments of the present application have the following advantages:

[0052] In an embodiment of the present application, by introducing category prior knowledge, a sample similarity learning module guided by category knowledge is designed to obtain a first similarity matrix between the labeled image and the unlabeled image, and the label information of the labeled image is input into the category similarity module of the generalized class discovery model to obtain a similarity score between the label categories; the sample image is input into the label estimation module of the generalized class discovery model to obtain a clustering result of the sample image; the prototype network of the generalized class discovery model is trained according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges. Compared with the prior art, since label information is used in the training process, the rich semantic information of the label information itself can be fully utilized, and the close association between the label categories obtained through the label information can be obtained, so that new categories can be inferred based on these association information, thereby improving the generalization ability of the generalized class discovery model for new categories.

[0053] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the structure of the training process of the generalized class discovery model provided in the embodiment of the present application;

[0055] Figure 2 A flowchart of the steps of a training method for a generalized class discovery model provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structure of a trained generalized class discovery model provided in an embodiment of the present application;

[0057] Figure 4 A flowchart of an image recognition method provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of the structure of a training device for a generalized class discovery model provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of the structure of an image recognition device provided in an embodiment of the present application;

[0060] Figure 7A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0062] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0063] Reference Figure 1 , shows a schematic diagram of the training process of the generalized class discovery model provided in an embodiment of the present application, and the training method may include the following steps:

[0064] exist Figure 1 In the paper, the generalized category discovery (GCD) model is a common task in semi-supervised learning and unsupervised learning, which aims to discover new categories using a small amount of labeled data and a large amount of unlabeled data. The generalized category discovery model improves the classification and clustering effect of the generalized category discovery model by combining the information of labeled data and unlabeled data.

[0065] Among them, the generalized class discovery model includes: class knowledge-guided sample similarity learning module, category similarity module, label estimation module and prototype network.

[0066] based on Figure 1 As shown in FIG. 1 , the training process of the generalized class discovery model is described in detail. The training process mainly includes the following steps:

[0067] Step 201: inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model respectively, and obtaining a first similarity matrix between the labeled image and the unlabeled image.

[0068] Assumption: There are N labeled images in the sample image, and the set of labeled images is Among them, x i is the labeled image, y i is the label information corresponding to the labeled image, where [I] is the input space, is a set of known categories, There are M unlabeled images in the sample images, and the set of unlabeled images is Among them, x j is an unlabeled image, is the set of all categories and make is a set of new categories. In the training phase, the labeled image set is used and the unlabeled image set D Y To train a generalized class discovery model.

[0069] Specifically, the labeled image is input into the feature extraction network to obtain a first embedding vector corresponding to the labeled image; the unlabeled image is input into the feature extraction network to obtain a second embedding vector corresponding to the unlabeled image, that is, N labeled images and M unlabeled images are respectively input into the feature extraction network f e (·; Θ) to obtain the embedding vector H∈R (N+M)×d .

[0070] Among them, the feature extraction network is initialized using the pre-trained model ViT / B-16.

[0071] After the first embedding vector and the second embedding vector are input into the sample relationship network, a first similarity matrix S∈R between the labeled image and the unlabeled image is obtained. (N+M)×(N+M) .

[0072] For example: labeling image x i and the unlabeled image x j , their eigenvectors are respectively the eigenvectors of the samples are respectively h i and h j ,s ij is defined as follows:

[0073] s ij =sim(h i ,h j )

[0074] Among them, s ij It represents the similarity between labeled image i and unlabeled image j. sim(.) is a similarity function used to calculate the similarity between two feature vectors. Generally, cosine similarity is used. The cosine similarity measures the similarity between two vectors by calculating the cosine value of the angle between the two vectors.

[0075] Step 202: input the label information of the labeled image into the category similarity module of the generalized category discovery model to obtain the similarity scores between the label categories.

[0076] Specifically, a semantic description vector corresponding to the label information of the marked image is obtained; the semantic description vector is input into the sample relationship network to obtain a similarity score between label categories.

[0077] The first similarity matrix S1∈R corresponding to the labeled image using the category similarity module N×N Therefore, the following category similarity loss function L is designed rank :

[0078]

[0079] Where N is the total number of labeled patterns, y i and j is the label of image i and j, I(y i =y j ) is the indicator function, when y i =y j 1 when the value is 0, otherwise, margin ij is the margin (predefined boundary value) between the labeled images i and j, usually a predefined constant, s ij The similarity score or match score between labeled images i and j.

[0080] Indicator function I(y i =y j ):

[0081] When i =y j When I(y i =y j )=1, marking images i and j as belonging to the same category.

[0082] When i ≠y j When I(y i =y j )=0, marking images i and j as belonging to different categories.

[0083]

[0084] k ij is a constant representing the similarity between labeled images i and j.

[0085]

[0086] and They are the labels y corresponding to the labeled image i i and the label y corresponding to the labeled image j j The embedding vector of is the inner product of the embedding vector, and is the norm of the embedding vector.

[0087] Step 203: Input the sample image into the label estimation module of the generalized class discovery model to obtain the clustering result of the sample image.

[0088] Specifically, the sample image is re-represented to obtain a labeled sample image; the labeled sample image is input into the feature extraction network to obtain a fourth embedding vector corresponding to the labeled sample image; the fourth embedding vector is input into the sample relationship network to obtain a second similarity matrix between the labeled sample image and the labeled image; the second similarity matrix is ​​clustered using a semi-supervised K-means clustering algorithm to obtain a clustering result, and the clustering result includes: the number of clusters of the unlabeled image, the number of clusters of the labeled image and the pseudo-label of the unlabeled image.

[0089] The semi-supervised K-means clustering step assigns sample images to the nearest cluster center: for labeled images, they are directly assigned to the corresponding category. For unlabeled images, their distance to each cluster center is calculated and they are assigned to the cluster center with the nearest distance.

[0090] For labeled images in clustering results, their original category labels are used directly. For unlabeled images, category labels are assigned according to the final clustering results. Unlabeled class division: For unlabeled images, category labels are assigned according to the final clustering results, that is, corresponding pseudo labels are assigned to the unlabeled images, and the pseudo labels are used as the input of the prototype network for training.

[0091] The sample images include: N labeled images and M unlabeled images, and the N+M sample images are re-represented, that is, the N+M sample images can be represented by N labeled samples to obtain labeled sample images, and the labeled sample images are input into the feature extraction network to obtain a fourth embedding vector corresponding to the labeled sample images; the fourth embedding vector is input into the sample relationship network to obtain a second similarity matrix S2∈R between the labeled sample images and the labeled images (N+M)×N At this point, each sample image is re-expressed as an N-dimensional vector, where each dimension represents the similarity between the sample image and each labeled image.

[0092] Use the semi-supervised K-means clustering algorithm to cluster the rows in S2 to obtain clustering results, where the clustering results include: the number of clusters of unlabeled images π A , the number of clusters of labeled images π Y and pseudo labels for M unlabeled images Then the supervision information L is applied to the output of the labeled image. clu (Clustering loss function).

[0093]

[0094] Among them, π A is the number of clusters of unlabeled images, π k is the sample set of the kth cluster, h i The feature vector h corresponding to the labeled image i , μ k is the cluster center of the kth cluster, and ||.|| represents the Euclidean distance (matching score).

[0095] The cluster center is obtained by a clustering algorithm (such as K-means) and serves as the category prototype of each category.

[0096]

[0097] Among them, I(y i = = k) is the indicator function, when y i =k is 1, otherwise it is 0.

[0098] Step 204: training the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges.

[0099] The prototype network includes: a known category prototype of the marked image in the sample image and a new category prototype of the sample image, wherein the semantic description vector corresponding to the label information of the marked image is used as the known category prototype of the prototype network, and the third embedding vector is used as the new category prototype.

[0100] Specifically, training the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix and the similarity score includes: determining the category loss function according to the second similarity matrix and the similarity score; determining the clustering loss function according to the second embedding vector of the labeled image and the known category prototype of the labeled image; inputting the unlabeled image into the prototype network to obtain a matching score; determining the classification loss function according to the matching score, the pseudo label and the real label of the labeled image; updating the network parameters of the prototype network according to the category similarity loss value, the clustering loss value and the classification loss value until the generalized class discovery model converges. By training the model according to the category loss value, the clustering loss value and the classification loss value, the classification and clustering performance of the model can be improved. By reasonably selecting and optimizing the loss function, the training effect and application effect of the model can be improved.

[0101] The total loss is calculated by taking the weighted sum of the category loss value, clustering loss value and classification loss value, and the model parameters are updated using the gradient descent method or other optimization algorithms to minimize the total loss and make the model converge, thereby improving the performance and generalization ability of the model.

[0102] Among them, the prototype network is f p (·; P), Among them, p1,…,p L is a known class prototype, p L+1 ,…,p c It is an unknown category prototype. The semantic description vector corresponding to the label information of the labeled image is used as the known category prototype of the prototype network. The fourth embedding vector corresponding to the labeled sample image is used as the unknown category prototype. Then, the prototype network is trained to obtain the new category prototype. It is continuously updated during the training process. The corresponding category prototype representation is learned according to the data distribution of the unlabeled image. The specific learning process is:

[0103] Suppose we have N labeled images and M unlabeled images, and the feature vectors corresponding to the labeled images and the unlabeled images are h i and h j , the category labels are Y i and Y U , and then calculate the classification loss value L cls .

[0104]

[0105] f p (h i ; P)∈R (M+N)×c is the prototype network for the labeled image h i In order to make the output vector satisfy the probability distribution, we use softmax to normalize and get the matching score The prototype network is used for the unlabeled image h j The matching score, Y i is the labeled image h i Tag information, is the prototype network for the unlabeled image h j Pseudo labels of , both of which are one-hot vectors.

[0106] The total loss function of the generalized class discovery model is: L total =L rank +L cls +L clu , and use the stochastic gradient descent algorithm to update the network parameters. Through the above steps, the training of the generalized class discovery model is completed to obtain the trained generalized class discovery model.

[0107] In the embodiment of the present application, by introducing category prior knowledge, a sample similarity learning module guided by category knowledge is designed, and a corresponding category loss is designed. At the same time, a prototype network is designed to realize the prediction of new categories by the generalized category discovery model, and corresponding classification loss and semi-supervised clustering loss are designed to update the parameters of the prototype network. Compared with the prior art, since label information is used in the training process, the rich semantic information of the label information itself can be fully utilized. The close association between label categories can be obtained through label information, so new categories can be inferred based on these association information, thereby improving the generalization ability of the generalized category discovery model for new categories.

[0108] Reference Figure 3 , showing a schematic diagram of the structure of the trained generalized class discovery model provided in the embodiment of the present application, based on Figure 3 See the structural diagram shown in Figure 4 The process of an image recognition method provided by an embodiment of the present application is described in detail. The image recognition method may include the following steps:

[0109] Step 401: Obtain a pre-trained generalized class discovery model.

[0110] Among them, obtain the pre-trained generalized class discovery model, that is, Figure 3 , Figure 3 The generalized class discovery model in Figure 1 The training method is obtained by training. For specific training methods, see Figure 1 That’s it, no more details.

[0111] Step 402: inputting the unlabeled image in the sample image into the feature extraction network of the generalized class discovery model to obtain a fourth embedding vector of the unlabeled image.

[0112] The sample images include unlabeled images and labeled images, the unlabeled images do not have labels, and the labeled images have labels. In a specific application, the unlabeled images in the sample images are input into the feature extraction network of the generalized class discovery model to obtain the fourth embedding vector of the unlabeled images. For example, there are M unlabeled images, and the M labeled images are input into the generalized class discovery model to obtain the fourth embedding vectors corresponding to the M unlabeled images.

[0113] Step 403: Input the fourth embedding vector into the prototype network of the generalized class discovery model to obtain the matching scores between the unlabeled image and the prototypes of each category corresponding to the sample image.

[0114] The prototype network includes: known category prototypes of labeled images in the sample images and new category prototypes of the sample images.

[0115] Among them, the category prototype refers to the center of the category.

[0116] exist Figure 3 In this generalized class discovery model, there are two networks: feature extraction network f e (·; Θ) and the prototype network f p (·; P), the prototype network includes: known category prototypes of the labeled images in the sample image and new category prototypes of the sample image.

[0117] For example: unlabeled sample x j The set D of unlabeled images Y , that is, x∈D U , input to the feature extraction network f e (·; Θ), we get the first embedding vector of the unlabeled image, that is, h = f e (x; Θ * ), and then input into the prototype network based on the first embedding vector, calculate the first matching score between the unlabeled image and the known category prototype and the second matching score f between the unlabeled image and the new category prototype p (h; P * )∈R (M+N)×c .

[0118] In practical applications, the first embedding vector is input into the prototype network, a first matching score between the unlabeled image and the known category prototype and a second matching score between the unlabeled image and the new category prototype are calculated, and then the first matching score and the second matching score are normalized, and then the maximum matching score is determined from the first matching score and the second matching score by maximizing the likelihood function.

[0119] For example: given an unlabeled image x∈D U , the embedding representation is obtained through the feature extraction network Local optimal solution.

[0120] Step 404: Determine a maximum matching score from the first matching score and the second matching score by maximizing a likelihood function.

[0121] Step 405: Using the category prototype corresponding to the maximum matching score as the predicted label information of the unlabeled image.

[0122] In practical applications, the category prototype with the largest matching score between the first matching score and the second matching score is selected as the predicted label information of the unlabeled image.

[0123] In practical applications, the maximum matching score can be determined from the first matching score and the second matching score by maximizing the likelihood function; the category prototype corresponding to the maximum matching score is used as the predicted label information of the unlabeled image, and the label information is the category of the image, for example: the input data is an image, and the predicted label is the category of the image (such as cat, dog, car, etc.).

[0124] In an embodiment of the present application, the unlabeled image is processed using a feature extraction network of a generalized class discovery model to obtain an embedding vector of the unlabeled image, and then a matching score between the unlabeled image and each category prototype corresponding to the sample image is obtained based on a prototype network in the generalized class discovery model. The label information of the unlabeled image is determined based on the matching score. Compared with the prior art, the relationship between the unlabeled image and each category prototype is taken into account, thereby improving the generalization ability of the generalized class discovery model.

[0125] Reference Figure 5 , shows a schematic diagram of the structure of a training device for a generalized class discovery model provided in an embodiment of the present application, specifically comprising:

[0126] The class knowledge learning module 501 is used to input the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model respectively, and obtain a first similarity matrix between the labeled image and the unlabeled image;

[0127] A category module 502, used to input the label information of the labeled image into a category similarity module of a generalized category discovery model to obtain similarity scores between label categories;

[0128] A label module 503, used for inputting the sample image into the label estimation module of the generalized class discovery model to obtain a clustering result of the sample image;

[0129] A training module 504 is used to train the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges. Optionally, the class knowledge learning module includes:

[0130] An input unit, used to input the marked image and the unmarked image into the feature extraction network respectively, to obtain a first embedding vector corresponding to the marked image and a second embedding vector corresponding to the unmarked image;

[0131] A similarity unit is used to input the first embedding vector and the second embedding vector into a sample relationship network to obtain a first similarity matrix between the labeled image and the unlabeled image.

[0132] Optionally, the category module includes:

[0133] A semantic description unit, used to obtain a semantic description vector corresponding to the label information of the marked image;

[0134] The label unit is used to input the semantic description vector into the sample relationship network to obtain the similarity score between the label categories.

[0135] Optionally, the label module includes:

[0136] a re-representation unit, used for re-representing the sample image to obtain a marked sample image;

[0137] An embedding unit, used for inputting the labeled sample image into the feature extraction network to obtain a third embedding vector corresponding to the labeled sample image;

[0138] A matrix unit, used for inputting the third embedding vector into a sample relationship network to obtain a second similarity matrix between the labeled sample image and the labeled image;

[0139] A clustering unit is used to cluster the second similarity matrix using a semi-supervised K-means clustering algorithm to obtain a clustering result, wherein the clustering result includes: the number of clusters of unlabeled images, the number of clusters of labeled images, and pseudo labels of unlabeled images.

[0140] Optionally, the prototype network includes: a known category prototype of the labeled image in the sample image and a new category prototype of the sample image, wherein the semantic description vector corresponding to the label information of the labeled image is used as the known category prototype of the prototype network, and the third embedding vector is used as the new category prototype;

[0141] The training module includes:

[0142] A category similarity loss unit, used to determine a category similarity loss value according to the similarity matrix and the similarity score;

[0143] A clustering loss unit, configured to determine a clustering loss value according to the second embedding vector of the labeled image and a known class prototype of the labeled image;

[0144] A matching unit, used for inputting the unlabeled image into the prototype network to obtain a matching score;

[0145] a classification loss unit, configured to determine a classification loss value according to the matching score, the pseudo label and the label information of the labeled image;

[0146] An updating unit is used to update the network parameters of the prototype network according to the class similarity loss value, the clustering loss value and the classification loss value until the generalized class discovery model converges.

[0147] In the embodiment of the present application, by introducing category prior knowledge, a sample similarity learning module guided by category knowledge is designed, and a corresponding category loss is designed. At the same time, a prototype network is designed to realize the prediction of new categories by the generalized category discovery model, and corresponding classification loss and semi-supervised clustering loss are designed to update the parameters of the prototype network. Compared with the prior art, since label information is used in the training process, the rich semantic information of the label information itself can be fully utilized. The close association between label categories can be obtained through label information, so new categories can be inferred based on these association information, thereby improving the generalization ability of the generalized category discovery model for new categories.

[0148] Reference Figure 6 , shows a schematic diagram of the structure of an image recognition device provided in an embodiment of the present application, specifically comprising:

[0149] The acquisition module 601 is used to acquire a pre-trained generalized class discovery model, where the generalized class discovery model is trained by the training method.

[0150] The input module 602 is used to input the unlabeled image in the sample image into the feature extraction network of the generalized class discovery model to obtain a fourth embedding vector of the unlabeled image.

[0151] A matching score module 603 is used to input the fourth embedding vector into the prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image, wherein the prototype network includes: a known category prototype of the labeled image in the sample image and a new category prototype of the sample image.

[0152] The label module 604 is used to determine the predicted label information of the unlabeled image according to the matching score.

[0153] Optionally, the matching score module includes:

[0154] A calculation unit is used to calculate a first matching score between the unlabeled image and the known category prototype and a second matching score between the unlabeled image and the new category prototype.

[0155] Optionally, the label module includes:

[0156] A selection unit is used to determine a maximum matching score from the first matching score and the second matching score by maximizing a likelihood function; and use a category prototype corresponding to the maximum matching score as predicted label information of the unlabeled image.

[0157] In an embodiment of the present application, the unlabeled image is processed using a feature extraction network of a generalized class discovery model to obtain an embedding vector of the unlabeled image, and then a matching score between the unlabeled image and each category prototype corresponding to the sample image is obtained based on a prototype network in the generalized class discovery model. The true label of the unlabeled image is determined based on the matching score. Compared with the prior art, the relationship between the unlabeled image and each category prototype is considered, thereby improving the generalization ability of the generalized class discovery model.

[0158] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the training method or the image recognition method when executing the computer program.

[0159] Figure 7 FIG. 7 is a schematic diagram showing the structure of an electronic device 700 according to an embodiment of the present invention. Figure 7 As shown, the electronic device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0160] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, a microphone, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0161] The various processes and processing described above may be performed by the processing unit 701. For example, the method of any of the above embodiments may be implemented as a computer software program, which is tangibly contained in a computer-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU 701, one or more actions in the method described above may be performed.

[0162] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program that implements the training method or the image recognition method when executed by a processor.

[0163] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0164] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminals (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal so that a series of operating steps are executed on the computer or other programmable terminal to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable terminal. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0169] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or terminal including the elements.

[0170] The above is a detailed introduction to a training method for a generalized class discovery model, an image recognition method, a training device for a generalized class discovery model, an image recognition device, an electronic device and a computer-readable storage medium provided in the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A training method for a generalized class discovery model, characterized in that: include: Inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model respectively, and obtaining a first similarity matrix between the labeled image and the unlabeled image; Inputting the label information of the labeled image into the category similarity module of the generalized category discovery model to obtain similarity scores between label categories; Inputting the sample image into the label estimation module of the generalized class discovery model to obtain a clustering result of the sample image; The prototype network of the generalized class discovery model is trained according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges.

2. The training method according to claim 1, characterized in that: The step of inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model to obtain a first similarity matrix between the labeled image and the unlabeled image comprises: Inputting the labeled image and the unlabeled image into the feature extraction network respectively to obtain a first embedding vector corresponding to the labeled image and a second embedding vector corresponding to the unlabeled image; The first embedding vector and the second embedding vector are input into a sample relationship network to obtain a first similarity matrix between the labeled image and the unlabeled image.

3. The training method according to claim 1, characterized in that: Inputting the label information of the labeled image into the category similarity module of the generalized category discovery model to obtain the similarity score between the label categories includes: Obtaining a semantic description vector corresponding to the label information of the marked image; The semantic description vector is input into the sample relationship network to obtain the similarity scores between the label categories.

4. The training method according to claim 1, characterized in that: The step of inputting the sample image into the label estimation module of the generalized class discovery model to obtain the clustering result of the sample image comprises: Re-representing the sample image to obtain a labeled sample image; Inputting the labeled sample image into the feature extraction network to obtain a third embedding vector corresponding to the labeled sample image; Inputting the third embedding vector into a sample relationship network to obtain a second similarity matrix between the labeled sample image and the labeled image; The second similarity matrix is ​​clustered using a semi-supervised K-means clustering algorithm to obtain a clustering result, wherein the clustering result includes: the number of clusters of unlabeled images, the number of clusters of labeled images, and pseudo labels of unlabeled images.

5. The method according to claim 4, characterized in that The prototype network includes: a known category prototype of the labeled image in the sample image and a new category prototype of the sample image, wherein the semantic description vector corresponding to the label information of the labeled image is used as the known category prototype of the prototype network, and the third embedding vector is used as the new category prototype; The step of training the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix, and the similarity score until the generalized class discovery model converges comprises: Determining a category similarity loss value according to the second similarity matrix and the similarity score; Determining a clustering loss value based on the second embedding vector of the labeled image and the known category prototype of the labeled image; Inputting the unlabeled image into the prototype network to obtain a matching score; Determine a classification loss value based on the matching score, the pseudo label and the label information of the labeled image; The network parameters of the prototype network are updated according to the class similarity loss value, the clustering loss value and the classification loss value until the generalized class discovery model converges.

6. An image recognition method, characterized in that: include: Obtain a pre-trained generalized class discovery model, wherein the generalized class discovery model is trained by the training method according to any one of claims 1 to 5; Inputting an unlabeled image in the sample image into a feature extraction network of a generalized class discovery model to obtain a fourth embedding vector of the unlabeled image; Inputting the fourth embedding vector into the prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image, wherein the prototype network includes: known category prototypes of the labeled image in the sample image and new category prototypes of the sample image; Determine predicted label information of the unlabeled image according to the matching score.

7. The method according to claim 6, characterized in that Inputting the fourth embedding vector into the prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image comprises: A first matching score between the unlabeled image and the known category prototype and a second matching score between the unlabeled image and the new category prototype are calculated.

8. The method according to claim 7, characterized in that Determining the predicted label information of the unlabeled image according to the matching score includes: determining a maximum matching score from the first matching score and the second matching score by maximizing a likelihood function; The category prototype corresponding to the maximum matching score is used as the predicted label information of the unlabeled image.

9. A training device for a generalized class discovery model, characterized in that: include: A class knowledge learning module, used for inputting the labeled image and the unlabeled image in the sample image into the class knowledge guided sample similarity learning module of the generalized class discovery model respectively, to obtain a first similarity matrix between the labeled image and the unlabeled image; A category module, used for inputting the label information of the labeled image into a category similarity module of a generalized category discovery model to obtain similarity scores between label categories; A label module, used for inputting the sample image into the label estimation module of the generalized class discovery model to obtain a clustering result of the sample image; A training module is used to train the prototype network of the generalized class discovery model according to the clustering result, the first similarity matrix and the similarity score until the generalized class discovery model converges.

10. An image recognition device, characterized in that: include: An acquisition module, used to acquire a pre-trained generalized class discovery model, wherein the generalized class discovery model is trained by the training method according to any one of claims 1 to 5; An input module, used for inputting an unlabeled image in the sample image into a feature extraction network of a generalized class discovery model to obtain a fourth embedding vector of the unlabeled image; A matching score module, configured to input the fourth embedding vector into a prototype network of the generalized class discovery model to obtain a matching score between the unlabeled image and each category prototype corresponding to the sample image, wherein the prototype network includes: known category prototypes of the labeled image in the sample image and new category prototypes of the sample image; A label module is used to determine the predicted label information of the unlabeled image according to the matching score.

11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the training method described in any one of claims 1 to 5 or the image recognition method described in any one of claims 6 to 8 when executing the computer program.

12. A computer-readable storage medium, characterized in that: When the computer-readable storage medium stores a computer program that, when executed by a processor, implements the training method according to any one of claims 1 to 5 or the image recognition method according to any one of claims 6 to 8.

Images