Network-supervised fine-grained image recognition method based on partial label learning

Abstract

The present invention discloses a network-supervised fine-grained image recognition method based on partial label learning, comprising: using a pre-trained deep neural network model to perform deep descriptor transformation to evaluate the positive correlation between network images, detecting open-set label noise in the data set based on the correlation matrix, and removing the open-set label noise; using a loss function to drive the model to exhibit a high recall rate, so that the label set of each sample contains as many labels of the sample's true category as possible; selecting the true label of the sample from the sample's label set, correcting the closed-set label noise, and feeding the clean data and the closed-set label noise with the corrected label into a deep neural network for training. The present invention effectively removes the open-set label noise while using partial label learning to convert the closed-set label noise into training images with accurate labels, thereby increasing the number of available samples in the network data set and ensuring further improvement in the learning performance of the neural network model.

Description

Technical Field

[0001] The present invention belongs to the field of network supervised image recognition, and in particular relates to a network supervised fine-grained image recognition method based on partial label learning. Background Art

[0002] Constructing fine-grained datasets requires domain experts to correctly classify the subtle differences between fine-grained subcategories, making it a difficult task. To reduce the reliance on manual annotation for constructing fine-grained datasets and learn more practical models, it has become increasingly popular to directly collect images of relevant categories from the internet to construct network datasets and train them. However, the constructed network datasets contain a lot of data noise, and direct training can lead to model overfitting, thus affecting accuracy. Fine-grained network datasets generally have two types of noise, namely open-set label noise and closed-set label noise. Open-set label noise is usually caused by "cross-domain", that is, the noise does not belong to any category in the same fine-grained domain. Closed-set noise refers to images with incorrect labels in a fine-grained domain.

[0003] Methods for dealing with general label noise include sample selection, soft labels, or related loss functions. Although these methods have achieved good classification results, they have the risk of 1) discarding some clean images and 2) the inability to convert closed-set label noisy images into accurate training images. Summary of the Invention

[0004] The purpose of the present invention is to provide a network-supervised fine-grained image recognition method based on partial label learning.

[0005] The technical solution for achieving the purpose of the present invention is as follows: In a first aspect, the present invention provides a network-supervised fine-grained image recognition method based on partial label learning, comprising the following steps:

[0006] Step 1: Use a pre-trained deep neural network model to perform deep descriptor transformation to evaluate the positive correlation between network images, detect the open-set label noise in the dataset based on the correlation matrix reflecting the correlation, and remove the open-set label noise;

[0007] Step 2: Use the loss function to drive the model so that the label set of each sample contains the label of the sample's true category as much as possible;

[0008] In step 3, the idea of ​​partial label learning is used to select the true label of the sample from the sample's label set, thereby correcting the closed-set label noise.

[0009] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0010] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0011] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0012] Compared with the prior art, the present invention has the following significant advantages: (1) The present invention proposes an open-set label noise removal strategy and a closed-set label noise correction strategy to handle practical but challenging network supervised fine-grained recognition tasks; (2) The present invention uses a pre-trained deep model to perform deep descriptor transformation to estimate the positive correlation between network images, and effectively detects and removes open-set label noise based on the correlation value; (3) The candidate label set for each sample is independently constructed, and the loss function is used to drive the model to show a high recall rate, so that the candidate label set of each sample contains the true label of the sample as much as possible, ensuring the performance of partial label learning; (4) Partial label learning is used to convert closed-set label noise into training images with accurate labels, and while correcting the closed-set label noise, the closed-set label noise is converted into training data, which increases the number of available samples in the network dataset and ensures the improvement of the learning performance of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a flow chart of the network-supervised fine-grained image recognition method based on partial label learning of the present invention. DETAILED DESCRIPTION

[0014] Combine Figure 1 , a network-supervised fine-grained image recognition method based on partial label learning, specifically comprising the following steps:

[0015] Step 1: Use a pre-trained deep neural network model to perform deep descriptor transformation to evaluate the positive correlation between network images, detect the open-set label noise in the dataset based on the correlation matrix reflecting the correlation, and remove the open-set label noise;

[0016] set up is the label space, For each random batch of n images, By having a pre-trained convolutional neural network Φ preExtract a feature map set containing n feature maps

[0017]

[0018] Where H, W and d represent the feature map t i In order to be able to Find a more universal high-level feature expression and use principal component analysis to generate the eigenvector with the largest eigenvalue For a given set of feature maps, each feature map is channel-weighted summed with the feature vector p to obtain a heat map and form a heat map set Calculate the i-th feature map t i Corresponding The i-th heat map H i :

[0019]

[0020] in Each heat map is then upsampled to the input image size to obtain the correlation matrix C. The correlation matrix consists of correlation values, where positive values ​​indicate The positive correlation of the universal expression similarity exists, and the negative value indicates negative correlation. The larger the absolute value, the stronger the correlation. The number and size of the positive values ​​in the correlation matrix can effectively determine whether it is open set label noise. Therefore, the present invention sets a threshold δ to judge each sample and determine whether the i-th sample is noise:

[0021]

[0022] If the sample does not meet this condition, it is regarded as open-set label noise and removed from the sample space, so that a sample space consisting of clean data and closed-set label noise can be obtained.

[0023] Step 2: Use the loss function to drive the model to achieve a higher recall rate, so that the label set of each sample contains the label of the sample's true category as much as possible;

[0024] The present invention defines the label space as The sample space is in Indicates that the label belongs to the i-th category y i At the beginning of training, we randomly select C categories to generate a batch. For each selected category y i , randomly select n * samples of this category. Based on batch a=n* ×C can be obtained through the convolutional neural network Φ CNN to obtain the embedded features For the samples in it can be obtained through the convolutional neural network Φ to obtain the embedded feature f CNN : i where c is the length of the embedded feature f

[0025]

[0026] The similarity matrix is obtained by calculating the cosine similarity between the embedded features i where the cosine similarity between the i-th query image and the j-th support image is calculated as follows:

[0027]

[0028] The similarities s obtained by each query image and other images q,: are arranged, and the top k images with high similarities are put into the set . The present invention defines the images that belong to the same category as the query image but are not in the set as positive images, and the images that are in the set but are not of the same category as the query image as negative images. The positive images not in the set constitute the set where represents the complement of in and y q is the label of the query image. The present invention sets n * <k, and the set composed of negative images can be obtained where is the number of images that belong to the same category as the query image outside the set and s n is a matrix of similarity scores that only contains negative images. Therefore, the loss function is defined as

[0029]

[0030] Applying this loss function can ensure that each image generates a label set containing the true category as much as possible

[0031] Step 3: Using the idea of partial label learning, select the true label of the sample from the label set of the sample, so as to correct the closed-set label noise

[0032] Obtain as many label sets containing true categories as possible through step 2 Finally, a true label is determined from the label set of closed set label noise. In the encoding stage, the encoding matrix M∈{+1,-1} is constructed by randomly generating N-bit column codes. N×L , where N represents the number of categories, L represents the number of binary classifiers, and the encoding matrix is ​​used to divide the samples during training. A randomly generated column encoding v = [v1, v2, ..., v N ] T ∈{+1,-1} N The label space can be divided into positive label space and negative label space

[0033]

[0034]

[0035] Use the positive and negative label space to select positive and negative samples. Given a training sample in The present invention visual tag collection As a whole to help build a binary classifier. All categories fall into or When the sample These positive and negative samples then form a binary classification training set.

[0036] In the decoding stage, a connected set is constructed for each class. The connected set of class j can be expressed as:

[0037]

[0038] According to the connected set ε y Generate performance matrix G N×L To reflect the ability of the classifier, the jth class is in the tth classifier g t The performance calculation on is as follows:

[0039]

[0040] in Is the indicator function. In order to obtain the relative performance of the classifier on each class, the performance matrix G is normalized row by row:

[0041]

[0042] in For a closed set of label noise You can get the class predictions by:

[0043]

[0044] Finally, the closed set label noise obtains the pseudo label The clean samples and the closed-set label noise with pseudo labels are combined and fed into the convolutional neural network for training.

[0045] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A network-supervised fine-grained image recognition method based on partial label learning, characterized in that: The following steps are involved: Step 1: Use a pre-trained deep neural network model to perform deep descriptor transformation to evaluate the positive correlation between network images, detect the open-set label noise in the dataset based on the correlation matrix reflecting the correlation, and remove the open-set label noise; Step 2: Use the loss function to drive the model so that the label set of each sample contains the label of the sample's true category as much as possible; Step 3: Using the idea of ​​partial label learning, select the true label of the sample from the sample label set to correct the closed set label noise. Specifically: In the encoding phase, the encoding matrix M∈{+1,-1} is constructed by randomly generating N-bit column codes. N×L , where N represents the number of categories, L represents the number of binary classifiers, and the encoding matrix is ​​used to divide the samples during training; a randomly generated column encoding The label space can be divided into positive label space and negative label space Use the positive and negative label space to select positive and negative samples. Given a training sample in View Tag Collection As a whole to help build a binary classifier; when the label set All categories fall into or When the sample will be used as positive or negative samples; thus these positive and negative samples constitute the two-category training set In the decoding stage, a connected set is constructed for each class, and the connected set of the jth class is expressed as: According to the connected set ε y Generate performance matrix G N×L To reflect the ability of the classifier, the jth class is in the tth classifier g t The performance calculation on is as follows: in is the indicator function that normalizes the performance matrix G row by row: in For a closed set of label noise Get the class prediction via: Finally, the closed set label noise obtains the pseudo label The clean samples and the closed-set label noise with pseudo labels are combined and fed into the convolutional neural network for training.

2. The network-supervised fine-grained image recognition method based on partial label learning according to claim 1 is characterized in that: Step 1: Use a pre-trained deep neural network model to perform deep descriptor transformation to evaluate the positive correlation between network images, detect the open set label noise in the dataset based on the correlation matrix reflecting the correlation, and remove the open set label noise; set up is the label space, For each random batch of n images, By having a pre-trained convolutional neural network Φ pre Extract a feature map set containing n feature maps Where H, W and d represent the feature map t i The height, width and depth of For a given set of feature maps, each feature map is channel-weighted summed with the feature vector p to obtain a heat map and form a heat map set Calculate the i-th feature map t i Corresponding The i-th heat map H i : in Then each heat map is upsampled to the input image size to obtain the correlation matrix C; the correlation matrix consists of correlation values, and positive values ​​indicate There is a positive correlation between the universal expression similarity, and a negative value indicates a negative correlation. The larger the absolute value, the stronger the correlation. The number and size of the positive values ​​in the correlation matrix are used to determine whether it is open set label noise. Therefore, a threshold δ is set to judge each sample to determine whether the i-th sample is noise: If the sample does not meet this condition, it is regarded as open-set label noise and removed from the sample space, thereby obtaining a sample space consisting of clean data and closed-set label noise.

3. The network-supervised fine-grained image recognition method based on partial label learning according to claim 2 is characterized in that: Step 2 uses the loss function to drive the model to show a higher recall rate; Define the label space as The sample space is in Indicates that the label belongs to the i-th category y i In the sample selection phase at the beginning of training, C categories are randomly selected to generate a batch For each selected category y i , randomly select n * samples of this category; based on batch a=n * ×C, through the convolutional neural network Φ CNN Get embedded features For Samples in Through the convolutional neural network Φ CNN Get the embedded feature f i : where c is the embedding feature f i The length of the embedding features is calculated by calculating the cosine similarity between the similarity matrix The cosine similarity between the i-th query image and the j-th support image is calculated as follows: The similarity s obtained for each query image and other images q,: is arranged, and the top k images with high similarity are put into the set ; Define the images that belong to the same category as the query image but are not in the set as positive images, and the images that are in the set but not of the same category as the query image as negative images; The positive images not in the set constitute the set where denotes the complement of in, and y q is the label of the query image; Set n * < k, and obtain the set consisting of negative images where is the number of images that belong to the same category as the query image outside the set n is the matrix of similarity scores containing only negative images; Thus, the loss function is defined as Applying this loss function ensures that each image generates a label set that contains the true category as much as possible.

4. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 3 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images