An image classification method based on instance-dependent complementary label learning

Abstract

The present invention discloses an image classification method based on instance-dependent complementary label learning, comprising: constructing a transfer matrix network and a classifier network on a backbone network to estimate the complementary label transfer matrix of an instance and predict the true category of the instance; embedding instance features into feature vectors of specified dimensions, and mining the nearest neighbors of the instance using the Euclidean distance of the feature vectors as a metric; training the classifier network in combination with complementary label classification and semantic clustering to estimate the true class posterior probability of the instance, selecting instances with the top 1% of probabilities as approximate anchor points to initialize the transfer matrix network; introducing a constraint - "neighboring instances in feature space have more similar complementary label transfer matrices" - to limit the degrees of freedom of the transfer matrix, and constructing an end-to-end joint training framework to collaboratively optimize the parameters of the transfer matrix network and the classifier network, and using the optimized classifier network as the final prediction network to achieve image classification based on instance-dependent complementary label learning.

Description

Technical Field

[0001] The present invention belongs to the field of weakly supervised learning and image classification, and particularly relates to an image classification method based on instance-dependent complementary label learning. Background Art

[0002] With the rapid development of deep learning, deep neural networks have achieved results that surpass traditional methods in multiple image classification tasks. The success of deep learning relies on large amounts of accurately labeled image data. However, in real-world scenarios, due to limitations such as the cost of expert labeling and privacy protection, large amounts of accurately labeled data are often difficult to obtain. To address these issues, a weakly supervised learning method called complementary label learning has been proposed in recent years and has been widely used in fields such as image classification, medical image diagnosis, and questionnaire surveys. Complementary labels represent labels that do not belong to the category of the instance. Compared to deep learning, which requires accurately selecting the only correct label from a large number of labels, complementary labels are easier to obtain and have the advantages of privacy and security, which greatly improves the difficulties faced by deep learning in image classification tasks. Through complementary label learning, a classifier that can predict the true category label of an instance can be trained from a dataset that only has complementary labels.

[0003] In complementary label learning, the complementary label transfer matrix is ​​crucial for building statistically consistent multi-class classifiers. It represents the conversion relationship between the true class posterior probability of an instance and the complementary class posterior probability. Existing complementary label learning techniques assume that the generation of complementary labels depends solely on the true class label of the instance. However, in real-world image classification tasks, image instances of the same category exhibit significant feature differences, so the complementary label annotation process also depends on the instance's feature information. Because the relationship between complementary labels and instance features is not considered, existing complementary label learning techniques lack practicality. In the instance-dependent setting, different image instances have different transfer matrices. Without any additional constraints, the instance-dependent complementary label transfer matrix is ​​unidentifiable. Specifically, the feasible solution of the transfer matrix is ​​not unique. Therefore, how to accurately estimate the complementary label transfer matrix in the instance-dependent setting and thus build a reliable image classifier remains a difficult and urgent problem.

[0004] Related technical solutions for image classification problems based on instance-dependent complementary label learning, including the following literature and invention patents:

[0005] Reference 1: A Medical Image Classification Method for Uncertain Labeled Data (202110203910.X)

[0006] Reference 2: A label noise estimation method based on manifold regularized transfer matrix (202210192794.0)

[0007] Literature 1 constructs a deep learning model based on the different annotation results of medical image instances by multiple annotators. The classifier network is trained using consistent annotated data and positive category labels, while the counterfactual network is trained using inconsistent data and its complementary labels. Then, an anti-attention mechanism module is constructed to assist the learning of the classifier network with the learned counterfactual information. Finally, the trained classifier network is used as a prediction network to predict the true category of the medical image instance.

[0008] Reference 2 first pre-trains the first network to distill the image dataset, and forms a sub-dataset with credible instances whose probability of the category to which the image instances output by the first network belong is greater than the probability threshold; then the obtained sub-dataset is input into the second network to obtain the probability of the category to which the image instances in the sub-dataset belong and the instance-dependent noise label transfer matrix, calculates the cross-entropy loss function of the second network according to the category label of the instance, and constructs an association matrix for the neighboring instances in the sub-dataset where the instances belong to the same manifold (the instance category labels are the same) and a penalty matrix for the instances belonging to different manifolds (the instance category labels are different), and constructs a manifold regularization loss function based on the two matrices to constrain the noise label transfer matrix; finally, the second network is trained by reducing the above two loss functions, and then the trained second network is used to complete the image instance classification task.

[0009] The above literature has the following main problems: The algorithm proposed in literature 1 requires multiple annotators to annotate the image dataset and uses inconsistently labeled data and complementary labels to assist in the training of the classification network. This method does not consider the dependency between complementary labels and instance features and the true category, and simply assumes that the transition probability of complementary labels is uniformly distributed, which is inconsistent with actual application scenarios and makes this method lack of practicality. Literature 2, based on label noise learning, trains deep neural networks using a credible sub-dataset whose true class posterior probability is greater than a probability threshold. To reduce the degrees of freedom of the instance-dependent noisy label transfer matrix, this method constructs an association matrix and a penalty matrix for neighboring instances based on the category of the image instances in the sub-dataset as constraints on the transfer matrix. However, compared with label noise learning, the supervision information carried by complementary labels is weaker, and the network's estimation error of the class posterior probability of complementary labeled instances is larger. It is difficult to obtain a credible sub-dataset based solely on the relationship between the class posterior probability and the threshold. Therefore, constructing a manifold regularized loss function based on the category of the image instances in the sub-dataset to constrain the complementary label transfer matrix will result in large matrix estimation errors, making this method difficult to apply to complementary label learning. Therefore, how to introduce reasonable constraints to reduce the estimation error of instance-dependent complementary label transfer matrix remains an urgent problem to be solved. Summary of the Invention

[0010] In order to address the deficiencies of the existing technical solutions, the present invention provides an image classification method based on instance-dependent complementary label learning.

[0011] The present invention is achieved by adopting the following technical solutions:

[0012] An image classification method based on instance-dependent complementary label learning, comprising:

[0013] First, a transfer matrix network and a classifier network are constructed on top of the backbone network to estimate the complementary label transfer matrix of the instance and predict the true category of the instance; secondly, the instance features are embedded into feature vectors of specified dimensions, and the K-nearest neighbors of the instance are mined using the Euclidean distance of the feature vectors as the metric; thirdly, the classifier network is trained by combining complementary label classification and semantic clustering to estimate the true class posterior probability of the instance, and the instances with the top 1% of probabilities are selected as approximate anchors to initialize the transfer matrix network; finally, a constraint is introduced - "neighboring instances in the feature space have more similar complementary label transfer matrices" - to limit the degrees of freedom of the transfer matrix, and an end-to-end joint training framework is constructed to collaboratively optimize the parameters of the transfer matrix network and the classifier network. The optimized classifier network is used as the final prediction network to achieve image classification based on instance-dependent complementary label learning.

[0014] A further improvement of the present invention is that the method specifically comprises the following steps:

[0015] 1) Construction of transfer matrix network and classifier network

[0016] A convolutional neural network is constructed as the backbone network based on the feature dimensions of the image instances in the training set to extract the features of different image instances. Then, based on the number of target categories of the classification task, a transfer matrix network and a classifier network are constructed on top of the backbone network to estimate the complementary label transfer matrix of the instance and predict the true category label of the instance, respectively.

[0017] 2) Feature vector extraction and nearest neighbor mining of image instances

[0018] The features of image instances are extracted based on the backbone network and embedded into feature vectors of specified dimensions. The K nearest neighbors of the instances are then mined using the Euclidean distance between feature vectors of different instances.

[0019] 3) Approximate anchor point estimation and transfer matrix network initialization based on complementary annotation classification and semantic clustering

[0020] Based on the distance between image instances in the feature space, nearest neighbor semantic clustering is performed. Complementary labels are used to improve the cross-entropy loss function for classification. The classifier network is trained by combining complementary label classification and semantic clustering, outputting the true class posterior probability of the instance, and extracting the instances with the top 1% probability of each category as approximate anchor points. Afterwards, the feature vectors of the approximate anchor instances output by the backbone network are used as the input of the transfer matrix network, and the corresponding instance-dependent complementary label transfer matrix is ​​output. The complementary labels of the instances are derived by combining the Bayesian optimal labels of the approximate anchor points predicted by the classifier. A loss function is constructed based on the derived complementary labels and the true complementary labels to initialize the transfer matrix network parameters.

[0021] 4) Construction of an end-to-end joint training framework and prediction of the true category of image instances

[0022] By introducing the constraint that "adjacent instances in the feature space have more similar complementary label transfer matrices" the feasible solution space of the transfer matrix is ​​restricted, and the parameters of the transfer matrix network and the classifier network are collaboratively optimized by constructing an end-to-end joint training framework. The feature vectors of the image instances are input into the transfer matrix network and the classifier network respectively, and the output instance-dependent complementary label transfer matrix and the instance's true class posterior probability are combined with the instance's complementary label to construct a cross-entropy loss function. The network parameters are optimized by reducing the loss function value, and the optimized classifier network is used as the final prediction network to predict the true category label of the image instance.

[0023] A further improvement of the present invention is that in step 1), the construction of the transfer matrix network and the classifier network specifically includes the following steps:

[0024] Step 1: Build a backbone network

[0025] A convolutional neural network is constructed as the backbone network, which has a four-layer network structure consisting of input layer, convolution layer, pooling layer and fully connected layer. The first input layer performs image instance feature mapping, the second convolution layer performs feature mining and extraction, the third pooling layer performs max-pooling on the output of the second layer, and the last fully connected layer maps the output of the previous layer into a d-dimensional feature vector.

[0026] Step 2: Construct a transfer matrix network

[0027] A fully connected layer is added on top of the backbone network to construct a transfer matrix network. Assuming that the number of target categories in the image classification task is K, the transfer matrix network maps the d-dimensional feature vector output by the backbone network to a K×K dimensional vector output. The output vector is then converted to a K×K matrix, and a softmax operation is performed on each row of the matrix, followed by normalization, to obtain the complementary label transfer matrix corresponding to the image instance.

[0028] Step 3: Build a classifier network

[0029] A fully connected layer is added on top of the backbone network to construct a classifier network. The classifier network maps the d-dimensional feature vector output by the backbone network into a K-dimensional vector output, performs a softmax operation on this vector, and performs normalization processing to obtain the prediction results of the posterior probability of each category of the instance.

[0030] A further improvement of the present invention lies in the feature vector extraction and nearest neighbor mining of image instances in step 2): since the distribution of image data features is highly complex and nonlinear, in order to accurately reflect the intrinsic distribution pattern of the data to achieve accurate prediction, the backbone network is first pre-trained to fully mine the feature information of the image instance. Based on the idea that similar instances have similar feature representations, the backbone network outputs similar feature vectors for the original instance and its data enhancement, which is used to improve the robustness and feature mining capabilities of the network. Then, the Euclidean distance between the feature vectors is used as a metric to mine the nearest neighbors of the instance.

[0031] A further improvement of the present invention is that, in step 2), the specific implementation steps are as follows:

[0032] Step 1: Construct similar image instances based on data augmentation

[0033] Assume that the complementary labeled image training set is There are n training instances and K instance categories, x i For any image instance in the training set, For instance x i The corresponding complementary label, for the image instance x in the training set i Use data augmentation techniques to construct similar image instances φ(x i );

[0034] Step 2: Backbone network pre-training

[0035] In order to fully mine the feature information of image instances, the backbone network is pre-trained by learning the common features between similar instances and distinguishing the differences between non-similar instances, so that the backbone network outputs similar feature vectors for similar image instances and makes the distance between feature vectors of different image instances as far as possible; Assuming that the parameter of the backbone network is θ, the original instance x i And the data augmented instance φ(x i ) respectively input the backbone network mapping to obtain the feature vector g θ (x i ) and g θ (φ(x i )), will be achieved by minimizing gθ (x i ) and g θ (φ(x i )) to pre-train the backbone network. The specific form of the training objective is as follows:

[0036]

[0037] Among them, d(·,·) is the Euclidean distance, g θ (x i ) and g θ (φ(x i )) is the d-dimensional feature vector output by the backbone network;

[0038] The target is continuously optimized through the gradient descent method and back propagated through the BP algorithm to update the parameters of the backbone network and realize the backbone network pre-training;

[0039] Step 3: Mining the nearest neighbors of image instances

[0040] Select a positive integer N to represent the number of neighbors of the image instance, and calculate the Euclidean distance between the feature vectors of different image instances output by the backbone network for each instance x i The N nearest neighbors of

[0041] A further improvement of the present invention lies in that, in step 3), approximate anchor point estimation and transfer matrix network initialization are based on complementary label classification and semantic clustering: complementary labels are used as supervisory information to guide the classifier network to predict the true class posterior probability of the instance, and semantic clustering is used to enhance the consistency and smoothness of the network output to reduce the network's estimation error of the true class posterior probability, and then the approximate anchor point is estimated based on the instance class posterior probability output by the classifier network.

[0042] A further improvement of the present invention is that, in step 3), the specific implementation steps are as follows:

[0043] Step 1: Optimize the classifier network based on complementary label classification

[0044] Assume that η is the parameter of the classifier network, f(x i ;η)∈[0,1] K Output instance x for the classifier network i The true class posterior probability is ; Since the classifier is trained using the complementary labeled image training set, and the complementary label represents the category to which the instance does not belong, the cross entropy loss function is improved according to the characteristics of the complementary label. Specifically, the posterior probability prediction of the category corresponding to the complementary label is made close to 0, which is expressed as: The specific form of the loss function is:

[0045]

[0046] in, Output instance x for the classifier network i Belong to the category represented by complementary labels probability;

[0047] Step 2: Optimize the classifier network based on semantic clustering

[0048] Semantic clustering is performed based on the nearest neighbors of the image instances mined in step 2) to enhance the discrimination ability of the classifier network. Reducing the distance between the feature vectors of neighboring instances through semantic clustering helps improve the classification ability of the network. At the same time, the regularization term is introduced to ensure the relative balance of the distribution of each category, thereby preventing the classifier from assigning all instances to the same category. Assuming that the number of target categories for the image classification task is K, the image instances in the training set are assigned to K classes through semantic clustering. The specific form of the loss function is as follows:

[0049]

[0050] Where n is the number of image instances in the training set, For instance x i The nearest neighbor set of f(x i ;η)∈[0,1] K Output instance x for the classifier network i The cluster posterior probability, f c (x i ; n) is an instance x i The probability of belonging to the cth class, c∈{1,...,K}, <·> is the dot product operation, where the first term helps the function f(·; η) to the instance x i and its neighbors Make consistent predictions, where the second term evenly distributes the predictions across the K classes, and λ is the weight hyperparameter between the two terms;

[0051] Step 3: Approximate anchor point estimation

[0052] Calculate the above loss function value and and will The weight of is set to α, and the stochastic gradient descent method is used to minimize the loss function and update the parameters η of the classifier network. The K neurons in the output layer of the classifier network output the true class posterior probability of the instance, and the instances with the top 1% probability of each category are extracted to form an approximate anchor point set. Assuming that the number of image instances in the approximate anchor point set is L, is the i-th image instance in the approximate anchor set, For example Complementary tags, For example The Bayesian optimal label of , then the approximate anchor point set is expressed as: The specific form of the Bayesian optimal label of an instance is as follows:

[0053]

[0054] in, Output instances for the classifier network The probability of belonging to category k;

[0055] Step 4: Initialize the transfer matrix network parameters based on the approximate anchor point set

[0056] Initialize the transfer matrix network parameters based on the obtained approximate anchor point set. Assuming that the parameters of the transfer matrix network are ω, the image instance x output by the backbone network is a The eigenvector of is used as the input of the transfer matrix network, and the corresponding K×K complementary label transfer matrix is ​​output, satisfying the relationship: in Represents x a The element of the complementary label transfer matrix in row i and column j converts the instance x a The complementary label transfer matrix and Bayesian optimal labeling Combined, we can deduce the instance x a The complementary tags are:

[0057]

[0058] in, is the one-hot encoded vector;

[0059] Constructing derived complementary labels and true complementary labels The loss function between is used to initialize the parameters of the transfer matrix network. Its specific expression is as follows:

[0060]

[0061] Where L is the number of image instances in the approximate anchor set, and is the one-hot encoded vector, An example of the transfer matrix network output The complementary label transfer matrix of

[0062] During the training process, the above loss function is optimized and back-propagated through the BP algorithm to initialize the parameters of the transfer matrix network.

[0063] A further improvement of the present invention lies in that, in step 4), an end-to-end joint training framework is constructed and the true category of the instance is predicted: a constraint is introduced - "adjacent instances in the feature space have more similar transfer matrices" - to limit the feasible solution space of the parameters of the transfer matrix network. Specifically, a similarity loss function between the transfer matrices of neighboring instances is constructed to constrain the parameters of the transfer matrix network. By constructing an end-to-end joint training framework, the parameters of the transfer matrix network and the classifier network are collaboratively optimized. Finally, the optimized classifier network is used as a prediction network to predict the true category of the instance, thereby realizing image classification based on complementary label learning.

[0064] A further improvement of the present invention is that, in step 4), the specific implementation steps are as follows:

[0065] Step 1: Construct a similarity loss function based on the transfer matrix between neighboring instances

[0066] Based on the nearest neighbors of the image instances mined in step 2), a similarity loss function is constructed between transfer matrices, so that the transfer matrix network outputs similar complementary label transfer matrices for the image instances of the nearest neighbors in the feature space; by introducing the constraint of matrix similarity, the linear system can be reduced. The degree of freedom of the transfer matrix Q(x) is reduced, and the specific form of the loss function is as follows:

[0067]

[0068] in, For image instance x i The nearest neighbor set of is x i The nearest neighbor instance, Q j ,:(x i ;ω) is the transfer matrix network output instance x i The jth row of the complementary label transfer matrix, Q j ,:(x e ;ω) is the transfer matrix network output instance x e The jth row of the complementary label transfer matrix, ||·||2 is the l2 norm;

[0069] Step 2: Build an end-to-end joint training framework

[0070] By constructing an end-to-end joint training framework, the parameters of the transfer matrix network and the classifier network are collaboratively optimized. Specifically, the classifier network f(·; η) predicts the true class posterior probability of the instance and provides the true class posterior probability of the instance to the transfer matrix network. The transfer matrix network Q(·; ω) models the conversion relationship between the true class posterior probability of the instance and the complementary class posterior probability. After the output of the classifier network, a transformation layer is constructed using the output of the transfer matrix network. The forward loss correction method is used to convert the true class posterior probability of the instance into the complementary class posterior probability, and a cross entropy loss function is constructed between the complementary label and the converted complementary class posterior probability. Its specific expression is as follows:

[0071]

[0072] Step 3: True category prediction of image instance

[0073] Calculate the above loss function value and and will The loss weight is set to δ, and the stochastic gradient descent method is used to minimize the loss function. The parameters of the classifier network and the transfer matrix network are updated to train the final classifier network. After the training, the optimized classifier network f(·; η) is used as the prediction network, and the feature vector of the image instance x output by the backbone network is used as the input of the classifier network. The corresponding prediction vector f(x; η) is output, where f i (x; η), i∈{1, ..., K} is the i-th component of f(x; ηη), representing the posterior probability P(Y=i|X=x) that x belongs to the i-th class, and computing the Bayesian optimal label And serves as the true category label of the final predicted image instance.

[0074] The present invention has at least the following beneficial technical effects:

[0075] The present invention provides an image classification method based on instance-dependent complementary label learning and constructs a reliable image classifier based on the instance-dependent complementary label transfer matrix. Compared with existing technical solutions, the present invention has the following technical advantages:

[0076] (1) The present invention models the instance-dependent complementary label transfer matrix and introduces a constraint—"neighboring instances in the feature space have more similar complementary label transfer matrices"—to limit the feasible solution space of the transfer matrix. Unlike existing technical solutions that do not consider the dependency between complementary labels and instance features, the present invention reduces the estimation error of the instance-dependent complementary label transfer matrix, thereby constructing a more reliable image classification model.

[0077] (2) The present invention proposes an approximate anchor point estimation method. To address the problem that existing technical solutions are difficult to mine credible sub-datasets based on the class posterior probabilities of complementary annotated image instances, the present invention enhances the network's discrimination ability by combining complementary annotation classification and semantic clustering, thereby reducing the network's estimation error of the class posterior probabilities of complementary annotated instances, and thus being able to mine a more credible approximate anchor point set from the complementary annotated dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 This is the overall framework flow chart.

[0079] Figure 2 Flowchart for the construction of the transfer matrix network and the classifier network.

[0080] Figure 3 Flowchart for feature vector extraction and nearest neighbor mining of image instances.

[0081] Figure 4 Flowchart for approximate anchor point estimation and transfer matrix network initialization based on complementary label classification and semantic clustering.

[0082] Figure 5 Flowchart for building an end-to-end joint training framework and predicting the true category of image instances.

[0083] Figure 6 Schematic diagram of the classifier network for approximate anchor point estimation.

[0084] Figure 7 Initialize the diagram for the transfer matrix network.

[0085] Figure 8 Schematic diagram of the end-to-end joint training framework.

[0086] Figure 9 This is a graph showing the classification accuracy results of the present invention and the comparative method on the selected image dataset. DETAILED DESCRIPTION

[0087] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0088] like Figure 1As shown, in a specific implementation of the present invention, an image classification method based on instance-dependent complementary label learning includes the following steps:

[0089] Step 1. Construction of transfer matrix network and classifier network

[0090] like Figure 2 As shown in the figure, a convolutional neural network is constructed as the backbone network based on the feature dimensions of the image instances in the training set to extract the features of different image instances. Then, based on the number of target categories of the classification task, a transfer matrix network and a classifier network are constructed on top of the backbone network to estimate the complementary label transfer matrix of the instance and predict the true category label of the instance, respectively. The specific steps include the following:

[0091] S101. Build a backbone network

[0092] In this embodiment, a convolutional neural network is constructed as the backbone network, which has a total of four network layers: input layer, convolution layer, pooling layer, and fully connected layer. The first input layer performs feature mapping on the image instance with an input size of 28×28; the second convolution layer performs feature mining and extraction on the image instance, using convolution templates of sizes 5×5×32 and 5×5×64 to perform convolution operations on the feature matrix of the previous layer; the third pooling layer performs max-pooling on the output of the second layer, and the size of the pooling template is 2×2; the last fully connected layer maps the output of the previous layer to a 512-dimensional feature vector ( Figure 2 S101).

[0093] S102. Constructing a transfer matrix network

[0094] A fully connected layer is added on top of the backbone network to construct a transfer matrix network. In this embodiment, the number of target categories for the image classification task is 10. The transfer matrix network maps the 512-dimensional feature vector output by the backbone network to a 100-dimensional vector output, and then converts the output vector into a 10×10 matrix. Softmax operations are performed on each row of the matrix and normalized to obtain the complementary label transfer matrix of the instance ( Figure 2 S102).

[0095] S103. Build a classifier network

[0096] A fully connected layer is added on top of the backbone network to construct a classifier network. In this embodiment, the classifier network maps the 512-dimensional feature vector output by the backbone network into a 10-dimensional prediction vector output, performs a softmax operation on the vector, and performs normalization processing to obtain the prediction results of the posterior probability of each category of the instance ( Figure 2 S103).

[0097] Step 2. Feature vector extraction and nearest neighbor mining of image instances

[0098] like Figure 3 As shown, the present invention pre-trains the backbone network to fully mine the feature information of the image instance. Based on the idea that similar image instances have similar feature representations, the backbone network outputs similar feature vectors for the original instance and its data enhancement, and mines the nearest neighbors of the image instance using the Euclidean distance between the feature vectors as a metric. Specifically, the present invention includes the following steps:

[0099] S201. Constructing similar image instances based on data augmentation

[0100] Assume that the complementary labeled image training set is In this embodiment, n=60000, the number of target categories in the training set K=10, x i For any image in the training set, For image instance x i The corresponding complementary label for instance x i Use weak augmentation operations such as flipping, cropping, and shifting to construct similar image instances φ(x i )( Figure 3 S201).

[0101] S202. Backbone network pre-training

[0102] In order to fully exploit the feature information of image instances, the present invention pre-trains the backbone network by learning the common features between similar instances and distinguishing the differences between non-similar instances, so that the backbone network outputs similar feature vectors for similar image instances and makes the distance between the feature vectors of different image instances as far as possible. In this embodiment, assuming that the parameter of the backbone network is θ, the original instance x is i And the data augmented instance φ(x i ) are input into the backbone network mapping to obtain a 512-dimensional feature vector g θ (x i ) and g θ (φ(x i ), by minimizing g θ (x i ) and g θ (φ(x i )) to enhance the robustness of the backbone network. The specific form of the training objective is as follows:

[0103]

[0104] The target is continuously optimized by the gradient descent method and back propagated through the BP algorithm to update the parameters of the backbone network and realize the backbone network pre-training ( Figure 3S202).

[0105] S203. Mining the nearest neighbors of image instances

[0106] In this embodiment, a positive integer N=20 is selected to represent the number of neighbors of an image instance, and the Euclidean distance between the feature vectors of different image instances output by the backbone network is calculated for each instance x. i The 20 nearest neighbors of ( Figure 3 S203).

[0107] Step 3. Approximate anchor point estimation and transfer matrix network initialization based on complementary annotation classification and semantic clustering

[0108] like Figure 4 As shown in the figure, complementary labels are used as supervisory information to guide the classifier network to predict the true class posterior probability of the instance, and semantic clustering is used to enhance the consistency and smoothness of the network output to reduce the network's estimation error of the true class posterior probability. Then, an approximate anchor point is estimated based on the instance class posterior probability output by the classifier network. The specific steps include:

[0109] S301. Optimizing Classifier Networks Based on Complementary Label Classification

[0110] In this embodiment, the number of target categories for the image classification task is 10, assuming that η is the parameter of the classifier network, f(x i ;η)∈[0,1] 10 Output instance x for the classifier network i Since the classifier network is trained using a complementary labeled image training set, and the complementary label represents the category to which the instance does not belong, it is necessary to improve the traditional cross entropy loss function based on the characteristics of the complementary label. Specifically, the posterior probability prediction of the category corresponding to the complementary label is made close to 0, which is expressed as: The specific form of the loss function is as follows ( Figure 4 S301):

[0111]

[0112] in, Output instance x for the classifier network i Belong to the category represented by complementary labels probability.

[0113] S302. Optimizing Classifier Networks Based on Semantic Clustering

[0114] Semantic clustering is performed based on the nearest neighbors of the image instances mined in step 2 to enhance the discrimination ability of the classifier network. Since image instances that are similar in feature space usually belong to the same category, reducing the distance between the feature vectors of neighboring instances through semantic clustering helps to improve the classification ability of the network. At the same time, the regularization term is introduced to ensure the relative balance of the distribution of each category, thereby preventing the classifier from assigning all image instances to the same category. In this embodiment, the number of target categories for the image classification task is 10, and the image instances in the training set are assigned to 10 categories through semantic clustering. The specific form of the semantic clustering loss function is shown as follows ( Figure 4 S302):

[0115]

[0116] Where n = 60,000 is the number of image instances in the training set, For instance x i The nearest neighbor set of f(x i ;η)∈[0,1] 10 Output instance x for the classifier network i The cluster posterior probability, f c (x i ; n) is an instance x i The probability of belonging to the cth class, c∈{1, 2…, 10}, <·> is the dot product operation, where the first term helps the function f(·; η) to the instance x i and its neighbors Make consistent predictions. The second term in the formula evenly distributes the prediction results to the 10 classes. λ is the weight hyperparameter between the two terms. In this embodiment, λ is set to 5.

[0117] S303. Approximate anchor point estimation

[0118] Calculate the above loss function value and and will The weight of is set to α, and the stochastic gradient descent method is used to minimize the loss function and update the parameter η of the classifier network. In this embodiment, α is set to 0.1. The 10 neurons in the output layer of the classifier network output the true class posterior probability of the instance. The instances with the top 1% probability of each category are extracted to form an approximate anchor point set; the number of image instances in the approximate anchor point set is L = 600. is the i-th image instance in the approximate anchor set, For example Complementary tags, For example The Bayesian optimal label of , then the approximate anchor point set can be expressed as: The specific form of the Bayesian optimal label of the instance is as follows ( Figure 4 S303):

[0119]

[0120] in, Output instances for the classifier network The probability of belonging to class k.

[0121] S304. Initialize the transfer matrix network parameters based on the approximate anchor point set

[0122] The parameters of the transfer matrix network are initialized based on the obtained approximate anchor point set. In this embodiment, it is assumed that the parameters of the transfer matrix network are ω, and the image instance x output by the backbone network is a The eigenvector of is used as the input of the transfer matrix network, and the corresponding 10×10 complementary label transfer matrix is ​​output, satisfying the relationship: in Represents x a The element of the complementary label transfer matrix in row i and column j converts the instance x a The complementary label transfer matrix and Bayesian optimal labeling Combined, we can deduce the instance x a The complementary tags are:

[0123]

[0124] in, is the one-hot encoded vector.

[0125] Constructing derived complementary labels and true complementary labels The loss function between is used to initialize the parameters of the transfer matrix network. Its specific expression is as follows:

[0126]

[0127] Where L = 600 is the number of image instances in the approximate anchor set, and is the one-hot encoded vector, Output instances for the transfer matrix network The complementary label transfer matrix.

[0128] During the training process, the above loss function is optimized and back-propagated through the BP algorithm to initialize the parameters of the transfer matrix network. It should be noted that the instances belonging to the i-th class in the approximate anchor set can only be used to learn the i-th row of the transfer matrix, and the loss will not be calculated for other rows. The values ​​of other rows will be learned from instances other than the i-th class. Specifically, the parameters of the transfer matrix network can be divided into specific row parameters and public shared parameters. Through back-propagation, both types of parameters are initialized ( Figure 4 S304).

[0129] Step 4. End-to-end joint training framework construction and image instance true category prediction

[0130] like Figure 5 As shown in the figure, a constraint is introduced—"neighboring instances in the feature space have more similar transfer matrices"—to limit the feasible solution space of the transfer matrix network parameters. Specifically, a similarity loss function between the transfer matrices of neighboring instances is constructed to constrain the parameters of the transfer matrix network. An end-to-end joint training framework is then constructed to collaboratively optimize the parameters of the transfer matrix network and the classifier network. Finally, the optimized classifier network is used as the prediction network to predict the true category of the instance, realizing image classification based on complementary label learning. The specific steps include the following:

[0131] S401. Construct similarity loss function based on the transfer matrix between neighboring instances

[0132] Based on the nearest neighbors of the image instances mined in step 2, a similarity loss function is constructed between the transfer matrices, so that the transfer matrix network outputs similar complementary label transfer matrices for the instances of the nearest neighbors in the feature space. By introducing the constraint of matrix similarity, the linear system can be reduced. The degree of freedom of the transfer matrix Q(x) is reduced, and the specific form of the loss function is as follows ( Figure 5 S401):

[0133]

[0134] in, For image instance x i The nearest neighbor set of is x i The nearest neighbor instance, Q j ,:(x i ;ω) is the transfer matrix network output instance x i The jth row of the complementary label transfer matrix, Q j ,:(x e ;ω) is the transfer matrix network output instance x e The jth row of the complementary label transfer matrix is ​​, and ||·||2 is the l2 norm.

[0135] S402. Build an end-to-end joint training framework

[0136] After the transfer matrix network is initialized, the present invention collaboratively optimizes the parameters of the transfer matrix network and the classifier network by constructing an end-to-end joint training framework. Specifically, the classifier network f(·; η) predicts the true class posterior probability of the instance and provides the true class posterior probability of the instance to the transfer matrix network. The transfer matrix network Q(·; ω) models the conversion relationship between the true class posterior probability of the instance and the complementary class posterior probability. After the output of the classifier network, a transformation layer is constructed using the output of the transfer matrix network. The forward loss correction method is used to convert the true class posterior probability of the instance into the complementary class posterior probability, and a cross entropy loss function between the complementary label and the converted complementary class posterior probability is constructed. Its specific expression is as follows ( Figure 5 S402):

[0137]

[0138] S403. True category prediction of image instances

[0139] During the training process, the image training set is divided into a training set and a validation set in a ratio of 9:1, and the above loss function value is calculated on the training set. and and will The loss weight is set to δ, and the stochastic gradient descent method is used to minimize the loss function. The parameters {η, ω} of the classifier network and the transfer matrix network are updated. The optimal network parameters {η, ω} are then selected on the validation set to obtain the final classifier network. In this embodiment, δ = 1 is selected. The optimized classifier network f(·; η) is used as the final prediction network. The feature vector of the image instance x in the test set is input into the classifier network, and the corresponding prediction vector f(x; ηη) is output, where f i (x; η), i∈{1, ..., K} is the i-th component of f(x; ηη), representing the posterior probability P(Y=i|X=x) that x belongs to the i-th class, and computing the Bayesian optimal label And as the final predicted image instance true category label ( Figure 5 S403).

[0140] Example:

[0141] Combined with artificially generated complementary label data, this paper conducts experimental tests on three commonly used benchmark image datasets: MNIST, Kuzushiji-MNIST, and Fashion-MNIST, and compares them with four classification methods based on complementary label learning in the prior art: PC / S [1] ,NN[2] 、L-W [3] 、Forward [4] and one classification method based on label noise learning: PTD [5] Conducted a comparative experiment.

[0142] [1] Ishida T, Niu G, Hu W, et al. Learning from complementary labels[J]. Advances in neural information processing systems, 2017, 30.

[0143] [2] Ishida T, Niu G, Menon A, et al. Complementary-label learning for arbitrary losses and models[C] / / International Conference on Machine Learning. PMLR, 2019: 2971-2980.

[0144] [3] Gao Y, Zhang M L. Discriminative Complementary-Label Learning with Weighted Loss[C] / / International Conference on Machine Learning. PMLR, 2021: 3587-3597.

[0145] [4] Yu X, Liu T, Gong M, et al. Learning with biased complementary labels[C] / / Proceedings of the European conference on computer vision (ECCV). 2018: 68-83.

[0146] [5] Xia X, Liu T, Han B, et al. Part-dependent label noise: Towards instance-dependent label noise[J]. Advances in Neural Information Processing Systems, 2020, 33: 7597-7610.

[0147] The present invention selects classification accuracy as the evaluation index of the model to evaluate the accuracy of the classifier as a whole. Assuming that the number of instances in the data set is N, for category i∈{1, 2, ..., K}, n i It represents the number of correctly classified instances (both the actual and predicted labels are i). The classification accuracy is the ratio of the sum of correctly classified instances of each category to all instances. Its specific expression is as follows:

[0148]

[0149] Based on the synthetic benchmark image dataset, the classification accuracy of the present invention and the comparison method is as follows: Figure 9 shown.

[0150] Through comparative experiments, it can be seen that the algorithm proposed in the present invention achieves the best classification accuracy on the three synthetic benchmark image datasets, which demonstrates the effectiveness and superiority of the present invention.

[0151] It will be easily understood by those skilled in the art that the above description is merely an embodiment of the method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image classification method based on instance-dependent complementary label learning, characterized in that: include: First, a transfer matrix network and a classifier network are built on top of the backbone network to estimate the complementary label transfer matrix of the instance and predict the true category of the instance; Secondly, the instance features are embedded into a feature vector of a specified dimension, and the K nearest neighbors of the instance are mined using the Euclidean distance of the feature vector as a metric; Thirdly, the classifier network is trained by combining complementary label classification and semantic clustering to estimate the true class posterior probability of the instance, and the instances with the top 1% probability are selected as approximate anchor points to initialize the transfer matrix network; nearest neighbor semantic clustering is performed based on the distance between image instances in the feature space, and the cross entropy loss function is improved using complementary labels for classification. The classifier network is trained by combining complementary label classification and semantic clustering to output the true class posterior probability of the instance, and the instances with the top 1% probability of each category are extracted as approximate anchor points; then, the feature vectors of the approximate anchor instances output by the backbone network are used as the input of the transfer matrix network, and the corresponding instance-dependent complementary label transfer matrix is ​​output. The complementary labels of the instances are derived by combining the Bayesian optimal labels of the approximate anchors predicted by the classifier, and a loss function is constructed based on the derived complementary labels and the true complementary labels to initialize the transfer matrix network parameters; Finally, the constraint "neighboring instances in the feature space have more similar complementary label transfer matrices" is introduced to limit the feasible solution space of the transfer matrix, and the parameters of the transfer matrix network and the classifier network are collaboratively optimized by constructing an end-to-end joint training framework; the feature vectors of the image instance are input into the transfer matrix network and the classifier network respectively, and the output instance-dependent complementary label transfer matrix and the instance's true class posterior probability are combined with the instance's complementary label to construct a cross-entropy loss function. The network parameters are optimized by reducing the loss function value, and the optimized classifier network is used as the final prediction network to predict the true category label of the image instance; a similarity loss function between the transfer matrices of neighboring instances is constructed to constrain the parameters of the transfer matrix network. By constructing an end-to-end joint training framework, the parameters of the transfer matrix network and the classifier network are collaboratively optimized. Finally, the optimized classifier network is used as the prediction network to predict the true category of the image instance, realizing image classification based on complementary label learning.

2. The image classification method based on instance-dependent complementary label learning according to claim 1, characterized in that: The method specifically comprises the following steps: 1) Construction of transfer matrix network and classifier network; A convolutional neural network is constructed as the backbone network based on the feature dimensions of the image instances in the training set to extract the features of different image instances. Then, based on the number of target categories of the classification task, a transfer matrix network and a classifier network are constructed on top of the backbone network to estimate the complementary label transfer matrix of the instance and predict the true category label of the instance, respectively. 2) Feature vector extraction and nearest neighbor mining of image instances; The features of image instances are extracted based on the backbone network and embedded into feature vectors of specified dimensions. The K nearest neighbors of the instances are then mined using the Euclidean distance between feature vectors of different instances. 3) Approximate anchor point estimation and transfer matrix network initialization based on complementary annotation classification and semantic clustering; 4) Construction of an end-to-end joint training framework and prediction of the true category of image instances; The constraint "nearby instances in the feature space have more similar transfer matrices" is introduced to limit the parameter feasible solution space of the transfer matrix network.

3. The image classification method based on instance-dependent complementary label learning according to claim 2, characterized in that: In step 1), the construction of the transfer matrix network and the classifier network specifically includes the following steps: Step 1: Build a backbone network A convolutional neural network is constructed as the backbone network, which has a four-layer network structure: input layer, convolution layer, pooling layer, and fully connected layer. The first input layer performs image instance feature mapping, the second convolution layer performs feature mining and extraction, the third pooling layer performs max-pooling on the output of the second layer, and the last fully connected layer maps the output of the previous layer into a one-dimensional feature vector. Step 2: Construct a transfer matrix network A fully connected layer is added on top of the backbone network to construct a transfer matrix network. Assuming that the number of target categories in the image classification task is K, the transfer matrix network maps the d-dimensional feature vector output by the backbone network to a K×K dimensional vector output. The output vector is then converted to a K×K matrix, and a softmax operation is performed on each row of the matrix, followed by normalization, to obtain the complementary label transfer matrix corresponding to the instance. Step 3: Build a classifier network A fully connected layer is added on top of the backbone network to construct a classifier network. The classifier network maps the d-dimensional feature vector output by the backbone network into a K-dimensional vector output, performs a softmax operation on this vector, and performs normalization processing to obtain the prediction results of the posterior probability of each category of the instance.

4. The image classification method based on instance-dependent complementary label learning according to claim 3, characterized in that: In step 2), feature vector extraction and nearest neighbor mining of image instances: the backbone network is pre-trained to fully mine the feature information of image instances. Based on the idea that similar image instances have similar feature representations, the backbone network outputs similar feature vectors for the original instance and its data enhancement, which is used to improve the robustness and feature mining capabilities of the network. The nearest neighbors of the image instances are then mined using the Euclidean distance between feature vectors as a metric.

5. The image classification method based on instance-dependent complementary label learning according to claim 4, characterized in that: In step 2), the specific implementation steps are as follows: Step 1: Construct similar image instances based on data augmentation Assume that the complementary labeled image training set is There are n training instances and K instance categories, x i For any image instance in the training set, For instance x i The corresponding complementary label, for the image instance x in the training set i Use data augmentation techniques to construct similar image instances φ(x i ); Step 2: Backbone network pre-training In order to fully mine the feature information of image instances, the backbone network is pre-trained by learning the common features between similar instances and distinguishing the differences between non-similar instances, so that the backbone network outputs similar feature vectors for similar image instances and makes the distance between feature vectors of different image instances as far as possible; Assuming that the parameter of the backbone network is θ, the original instance x i And the data augmented instance φ(x i ) respectively input the backbone network mapping to obtain the feature vector g θ (x i ) and g θ (φ(x i )), will be achieved by minimizing g θ (x i ) and g θ (φ(x i )) to pre-train the backbone network. The specific form of the training objective is as follows: Among them, d(·,·) is the Euclidean distance, g θ (x i ) and g θ (φ(x i ) is the d-dimensional feature vector output by the backbone network; The target is continuously optimized through the gradient descent method and back propagated through the BP algorithm to update the parameters of the backbone network and realize the backbone network pre-training; Step 3: Mining the nearest neighbors of image instances Select a positive integer N to represent the number of neighbors of the image instance, and calculate the Euclidean distance between the feature vectors of different image instances output by the backbone network for each instance x i The N nearest neighbors of 6. The image classification method based on instance-dependent complementary label learning according to claim 5, characterized in that: In step 3), approximate anchor point estimation and transfer matrix network initialization based on complementary label classification and semantic clustering: complementary labels are used as supervision information to guide the classifier network to predict the true class posterior probability of the instance, and semantic clustering is used to enhance the consistency and smoothness of the network output to reduce the network's estimation error of the true class posterior probability, and then the approximate anchor point is estimated based on the instance class posterior probability output by the classifier network.

7. The image classification method based on instance-dependent complementary label learning according to claim 6, characterized in that: In step 3), the specific implementation steps are as follows: Step 1: Optimize the classifier network based on complementary label classification Assume that η is the parameter of the classifier network, f(x i ;η)∈[0,1] K Output instance x for the classifier network i The true class posterior probability is ; Since the classifier network is trained using the complementary labeled image training set, and the complementary label represents the category to which the instance does not belong, the cross entropy loss function is improved according to the characteristics of the complementary label. Specifically, the posterior probability prediction of the category corresponding to the complementary label is made close to 0, which is expressed as: The specific form of the loss function is: in, Output instance x for the classifier network i Belong to the category represented by complementary labels probability; Step 2: Optimize the classifier network based on semantic clustering Semantic clustering is performed based on the nearest neighbors of the image instances mined in step 2) to enhance the discrimination ability of the classifier network. Reducing the distance between the feature vectors of neighboring instances through semantic clustering helps improve the classification ability of the network. At the same time, the regularization term is introduced to ensure the relative balance of the distribution of each category, thereby preventing the classifier from assigning all instances to the same category. Assuming that the number of target categories for the image classification task is K, the image instances in the training set are assigned to K classes through semantic clustering. The specific form of the loss function is as follows: Where n is the number of image instances in the training set, For instance x i The nearest neighbor set of f(x i ;η)∈[0,1] K Output instance x for the classifier network i The cluster posterior probability, f c (x i ; n) is instance x i The probability of belonging to the cth class, c∈{1,...,K}, <·> is the dot product operation, where the first term helps the function f(·; η) to the instance x i and its neighbors Make consistent predictions, where the second term evenly distributes the predictions across the K classes, and λ is the weight hyperparameter between the two terms; Step 3: Approximate anchor point estimation Calculate the above loss function value and and will The weight of is set to α, and the stochastic gradient descent method is used to minimize the loss function and update the parameters η of the classifier network. The K neurons in the output layer of the classifier network output the true class posterior probability of the instance, and the instances with the top 1% probability of each category are extracted to form an approximate anchor point set. Assuming that the number of image instances in the approximate anchor point set is L, is the i-th instance in the approximate anchor set, For example Complementary tags, For example The Bayesian optimal label of , then the approximate anchor point set is expressed as: The specific form of the Bayesian optimal label of an instance is as follows: in, Output instances for the classifier network The probability of belonging to category k; Step 4: Initialize the transfer matrix network parameters based on the approximate anchor point set Initialize the transfer matrix network parameters based on the obtained approximate anchor point set. Assuming that the parameters of the transfer matrix network are ω, the image instance x output by the backbone network is a The eigenvector of is used as the input of the transfer matrix network, and the corresponding K×K complementary label transfer matrix is ​​output, satisfying the relationship: in Represents x a The element of the complementary label transfer matrix in row i and column j converts the instance x a The complementary label transfer matrix and Bayesian optimal labeling Combined, we can deduce the instance x a The complementary tags are: in, is the one-hot encoded vector; Constructing derived complementary labels and true complementary labels The loss function between is used to initialize the parameters of the transfer matrix network. Its specific expression is as follows: Where L is the number of image instances in the approximate anchor set, and is the one-hot encoded vector, An example of the transfer matrix network output The complementary label transfer matrix of During the training process, the above loss function is optimized and back-propagated through the BP algorithm to initialize the parameters of the transfer matrix network.

8. The image classification method based on instance-dependent complementary label learning according to claim 7, characterized in that: In step 4), the specific implementation steps are as follows: Step 1: Construct a similarity loss function based on the transfer matrix between neighboring instances Based on the nearest neighbors of the image instances mined in step 2), a similarity loss function is constructed between transfer matrices, so that the transfer matrix network outputs similar complementary label transfer matrices for the image instances that are neighbors in the feature space; By introducing the matrix similarity constraint, the linear system can be reduced The degree of freedom of the transfer matrix Q(x) is reduced, and the specific form of the loss function is as follows: in, For image instance x i The nearest neighbor set of is x i The nearest neighbor instance, Q j，： (x i ;ω) is the transfer matrix network output instance x i The jth row of the complementary label transfer matrix, Q j，: (x e ;ω) is the transfer matrix network output instance x e The jth row of the complementary label transfer matrix, ||·||2 is the l2 norm; Step 2: Build an end-to-end joint training framework By constructing an end-to-end joint training framework, the parameters of the transfer matrix network and the classifier network are collaboratively optimized. Specifically, the classifier network f(·; η) predicts the true class posterior probability of the instance and provides the true class posterior probability of the instance to the transfer matrix network. The transfer matrix network Q(·; ω) models the conversion relationship between the true class posterior probability of the instance and the complementary class posterior probability. After the output of the classifier network, a transformation layer is constructed using the output of the transfer matrix network. The forward loss correction method is used to convert the true class posterior probability of the instance into the complementary class posterior probability, and a cross entropy loss function is constructed between the complementary label and the converted complementary class posterior probability. Its specific expression is as follows: Step 3: True category prediction of image instance Calculate the above loss function value and and will The loss weight is set to δ, and the stochastic gradient descent method is used to minimize the loss function. The parameters of the classifier network and the transfer matrix network are updated to train the final classifier network. After the training, the optimized classifier network f(·; η) is used as the prediction network, and the feature vector of the image instance x output by the backbone network is used as the input of the classifier network. The corresponding prediction vector f(x; η) is output, where f i (x; η), i∈{1, ..., K} is the i-th component of f(x; η), representing the posterior probability P(Y=i|X=x) that x belongs to the i-th class, and computing the Bayesian optimal label And used as the final predicted image instance true category label.

Images