Robust training method of deep multi-label classification network based on information entropy maximization regularization mechanism
By using the information entropy maximization regularization mechanism, the problems of false negative label noise and imbalance in large-scale multi-label image datasets are solved, improving the generalization performance of deep multi-label classification networks and making them suitable for ultra-large-scale multi-label image datasets.
Patent Information
- Application Number
- CN202111514507.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-13
- Publication Date
- 2026-01-09
- Estimated Expiration
- 2041-12-13
AI Technical Summary
Existing technologies suffer from false negative label noise and positive-negative label imbalance when processing large-scale multi-label image datasets, resulting in poor training performance of deep multi-label classification networks. Furthermore, existing regularization methods are difficult to implement in practical applications or increase the training burden.
We employ an information entropy maximization regularization mechanism, which maximizes the information entropy of the prediction probability of unknown labels. We utilize a binary cross-entropy loss function regularization mechanism based on information entropy maximization to avoid the influence of false negative label noise and use real positive labels to provide supervision information for learning. This method is suitable for ultra-large-scale multi-label image datasets.
It improves the generalization performance of deep multi-label classification networks, enabling highly discriminative predictions even with extremely missing labels. It boasts excellent performance without increasing learnable parameters and is suitable for multi-label datasets of various sizes.
Smart Images
Figure QLYQS_1 
Figure QLYQS_4 
Figure QLYQS_8
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of machine learning, in particular to a deep multi-label classification network robust training method based on information entropy maximization regularization mechanism. BACKGROUND
[0002] Unlike single-label image classification, the purpose of multi-label image classification is to predict all corresponding labels for an image. Because a natural scene naturally contains multiple objects and concepts, multi-label image classification is more in line with the setting in the real world. However, it is very time-consuming and laborious to make detailed label annotations on large-scale multi-label image datasets, and the cost of human and material resources consumed by the annotation is much higher than that of single-label image datasets, because the number of potential labels corresponding to an image can be large, and small objects and rare objects are often ignored by annotators. Therefore, the existing open-source large-scale multi-label image datasets are widely considered to be missing some labels, which promotes the research on multi-label learning problems under the condition of missing labels in the academic field. Considering an extreme case of missing labels, that is, only a single positive label is annotated for each sample, and the remaining labels are not annotated, which is called the Single Positive Multi-label Learning (SPML) problem. The setting that each sample has only a single positive label conforms to most actual application scenarios, such as collecting image datasets through queries from the Internet, etc., which indicates that the SPML problem has wide practical application value. In addition, in-depth research on the SPML problem and the proposal of a proper solution can relax the annotation requirements of large-scale multi-label image datasets and significantly reduce their annotation costs, which has great social and economic benefits. Therefore, SPML is a multi-label learning variant worthy of further exploration, and has become one of the challenging emerging research directions in the field of deep learning in recent years.
[0003] The learning paradigm of the deep multi-label classification network widely used in related technologies is to convert the prediction problem of multiple labels into multiple individual binary classification problems. Due to the particularity of the multi-label setting, in large-scale multi-label image datasets, the number of negative labels is much larger than the number of positive labels. Therefore, an intuitive method is to regard all unknown labels as negative labels, and then use the standard binary cross-entropy loss function for training, which is called Assuming Negatives (AN), which is the baseline method for solving the SPML problem. Assuming a sample pair (x (n) ,z (n) ), where x (n) is the nth input image, and z (n) is the label vector corresponding to the nth image. The loss function of the AN method is:
[0004]
[0005] where f (n) is the output of the model with respect to x (n) , is the label prediction probability of the model on the c-th class. C is the number of classes, and denote the existence of positive and negative labels for the c-th class, respectively. The loss function of the AN method follows the standard binary cross-entropy loss function, and the training of the deep multi-label classification network is based on the label assumption condition of "regarding all unknown labels as negative labels".
[0006] To solve the problem of degeneration in the training process of the deep multi-label classification network due to the lack of negative labels, the Regularized Online Label Estimation (ROLE) method uses the average number of real positive labels of each sample in the data set to constrain the output of the deep multi-label classification network, so that the sum of the output f (n) of the model with respect to x (n) is not too large, which is equivalent to introducing a regularization term, called Expected Positive Regularization (EPR), to the loss function. In order to make predictions for unknown labels to provide more supervision information, the ROLE method additionally adds a label estimation module to the original deep multi-label classification network, and jointly trains the two in a mutual supervision manner, using the estimated labels of the label estimation module as the labels for the deep multi-label classification network to calculate the loss, and vice versa, using the predictions of the deep multi-label classification network as supervision information for the label estimation module to estimate labels. Similarly, the ROLE method also introduces EPR to the label estimation module to constrain the estimated labels. When making predictions, the ROLE method can discard the previously trained label estimation module and only use the trained deep multi-label classification network to make label predictions.
[0007] However, the related art has the following disadvantages: 1) Although negative labels account for the majority of label annotations in large-scale multi-label image datasets, the assumption made by the AN method (i.e., considering all unknown labels as negative labels) introduces a large amount of false negative label noise, which provides a large amount of false supervision information for the training of the deep multi-label classification network, making it difficult to make correct positive and negative label predictions. In addition, this assumption also exacerbates the positive and negative label imbalance in multi-label learning, making the network more inclined to predict negative labels, which further weakens the classification performance of the deep multi-label classification network. 2) The EPR regularization method introduced by the ROLE method needs to statistically calculate the average number of real positive labels for each sample in the dataset in advance, which is unrealistic in practical application scenarios, because the number of real positive labels in the dataset is unknown in the first place. In addition, the label estimation module used by the ROLE method is essentially a huge estimated label matrix, which introduces additional large amounts of learnable parameters into the original network model, greatly increasing the time and space requirements for network training, and the number of learnable parameters introduced is positively correlated with the number of classes and the number of samples in the dataset, which indicates that the ROLE method is difficult to handle super large-scale multi-label image datasets. SUMMARY
[0008] To solve the problems in the prior art, the present application proposes a deep multi-label classification network robust training method based on information entropy maximization regularization mechanism, which uses a binary cross-entropy loss function regularization mechanism based on information entropy maximization to solve the SPML problem, the purpose of which is to maximize the information entropy of the deep multi-label classification network for unknown label prediction probability, so that the model can be immune to false negative label noise, learn from the supervision information provided by the real positive labels, and make predictions with strong distinguishability for unknown labels. The method is easy to implement and does not introduce additional learnable parameters, and can be combined with any deep multi-label classification network, suitable for handling super large-scale multi-label image datasets.
[0009] To achieve the above purpose, the present application provides a deep multi-label classification network robust training method based on information entropy maximization regularization mechanism, comprising: the prediction of each unknown label of each sample by the deep multi-label classification network is regarded as a discrete random variable, including prediction as a positive label and prediction as a negative label, the probability of prediction as a positive label being the unknown label prediction probability output by the deep multi-label classification network, and the information entropy of the unknown label prediction probability output by the deep multi-label classification network is maximized.
[0010] Further, in the method, it is assumed that a sample pair (x (n) ,z (n) ) is given, where x (n) is the nth input image, and z (n)The label vector corresponding to the nth image pair is y
[0011]
[0012] where f (n) is the output of the model with respect to x (n) , is the predicted probability of the model on the cth class, C is the number of classes, represents the existence of a positive label for the cth class, represents the existence of a negative label for the cth class, is a regularization term.
[0013] Further, the regularization term is defined as:
[0014]
[0015] where α is a hyperparameter that controls the strength of regularization.
[0016] Further, the value of the hyperparameter α is adjusted between 0 and 1.
[0017] Further, in the method, the probability of predicting a positive label is the unknown label prediction probability of the network output The probability of predicting a negative label is
[0018] Further, the method controls the unknown label prediction probability to tend to 0.5.
[0019] Further, in the method, it is assumed that there is a discrete random variable X ∈ {x1, x2,..., x n}, then the definition of its information entropy is:
[0020] H(X) = E[I(X)] = E[-ln(P(X))]
[0021] Where P(X) is the probability mass function of X, E[·] is the expectation function, and I(X) is the information amount of X.
[0022] Further, in the method, if P(X) is known or has been estimated, then the formula of information entropy is:
[0023]
[0024] Where b is the base used for logarithm.
[0025] Further, in the method, when b = 2, the unit of information entropy is bit; when b = e, the unit of information entropy is nat; when b = 10, the unit of information entropy is Hart.
[0026] Further, in the method, the prediction of each unknown label of each sample by the deep multi-label classification network is regarded as a discrete random variable X={x1, x2}, the event x1 is that the label is predicted to be positive, and the event x2 is that the label is predicted to be negative, P(x1) is the prediction probability of the unknown label output by the network, and P(x2)=1-P(x1) is the probability of predicting the label to be negative.
[0027] Compared with the prior art, the present application aims to solve the robust learning problem of the deep multi-label classification network in the case that each sample has only a single positive label and the remaining labels are missing, and the binary cross-entropy loss function regularization mechanism based on information entropy maximization, i.e. the EMR method, regards the unknown labels in the data set from a unique perspective, i.e. without making any assumptions about these unknown labels and admitting that they are unknown, maximizes the information entropy of the prediction probability of the unknown labels by the deep multi-label classification network in the loss function during model training, solves the SPML problem, so that the model can be immune to false negative label noise, learns using the supervision information provided by the true positive label, and makes a strong distinguishable prediction for the unknown label. The EMR method is easy to implement, does not introduce additional learnable parameters, can be combined with any deep multi-label classification network, has excellent performance, and is suitable for processing super large-scale multi-label image data sets. Using the method proposed in the present application for training, the generalization performance of the deep multi-label classification network can be greatly improved and performs well in the case of extreme label missing. DETAILED DESCRIPTION
[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Therefore, the detailed description of the embodiments of the present application is not intended to limit the scope of the claimed application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0029] The present application provides a deep multi-label classification network robust training method based on information entropy maximization regularization mechanism, comprising: the prediction of each unknown label of each sample by the deep multi-label classification network is regarded as a discrete random variable, including predicting a positive label and predicting a negative label, the probability of predicting a positive label is the prediction probability of the unknown label output by the deep multi-label classification network, and the information entropy of the prediction probability of the unknown label output by the deep multi-label classification network is maximized.
[0030] In information theory, information entropy refers to the average amount of information contained in each message received, also known as source entropy, average self-information, etc., which can be used to measure the uncertainty of the message. Here, "message" represents events, samples or features from a distribution or data stream, which in the method proposed in the present invention refers to the unknown label prediction probability output by the deep multi-label classification network. The unit of entropy is usually bit, but also measured in Sh, nat, Hart, etc., depending on the base of the logarithm used in the definition of entropy. Suppose there is a discrete random variable X∈{x1,x2,...,x n}, the definition of its information entropy is:
[0031] H(X)=E[I(X)]=E[-ln(P(X))]
[0032] Where P(X) is the probability mass function of X, E[·] is the expectation function, and I(X) is the information amount of X.
[0033] If P(X) is known or has been estimated, the formula of information entropy can be expressed as:
[0034]
[0035] Where b is the base used for logarithm. When b=2, the unit of entropy is bit; when b=e, the unit of entropy is nat; and when b=10, the unit of entropy is Hart. In the method of the present invention, the prediction of each unknown label of each sample by the deep multi-label classification network can be regarded as a discrete random variable X={x1,x2}, where event x1 is predicted as a positive label and event x2 is predicted as a negative label. Then P(x1) is the unknown label prediction probability output by the network, and correspondingly, the probability of predicting a negative label is P(x2)=1-P(x1). Based on this setting, the present invention can further complete the design of the binary cross-entropy loss function regularization mechanism based on the maximization of information entropy.
[0036] In order to avoid the influence of false negative label noise caused by false assumptions, and to prevent the deep multi-label classification network from overfitting to a large number of negative labels, the present application adopts a naive but effective idea to treat unknown labels in the data set, that is, not making any assumptions about these unknown labels and admitting that they are unknown. Based on this idea, the prediction probability of the deep multi-label classification network for the unknown label should be as close to 0.5 as possible, that is, tend to 0.5, that is, the information entropy of the prediction probability should be as large as possible. Therefore, the present application uses a binary cross-entropy loss function regularization mechanism based on information entropy maximization to solve the SPML problem, the purpose is to maximize the information entropy of the prediction probability of the deep multi-label classification network for the unknown label, called the (Entropy Maximization Regularizer, EMR) method. Assuming that a sample pair (x (n) ,z (n) ) is given, where x (n) is the nth input image, and z (n) is the label vector corresponding to the nth image. The loss function of the EMR method is:
[0037]
[0038] Where f (n) is the output of the model with respect to x (n) , is the prediction probability of the model on the cth class. C is the number of classes, indicates that the cth class has a positive label, indicates that the cth class has a negative label, is a regularization term introduced by the EMR method, which is defined as follows:
[0039]
[0040] Where alpha is a hyperparameter that controls the regularization strength, and its value is generally adjusted between 0 and 1. In the EMR method, the prediction of each unknown label by the deep multi-label classification network for each sample can be regarded as a discrete random variable, which has two possibilities of predicting a positive label and predicting a negative label. The probability of predicting a positive label is the prediction probability of the unknown label output by the network Correspondingly, the probability of predicting a negative label is In this way, the EMR method can impose maximum information entropy supervision on unknown labels, rather than treating them all as negative labels, which is consistent with the idea adopted by the present application. The EMR method enables the deep multi-label classification network to learn from the supervision information provided by the true positive label and make a more distinguishable prediction for the unknown label, rather than making a too confident or even wrong judgment, which greatly improves the generalization ability of the network.
[0041] The binary cross-entropy loss function regularization mechanism based on information entropy maximization, i.e., the EMR method, regards unknown labels in the data set from a unique perspective, i.e., does not make any assumptions about the unknown labels and acknowledges that they are unknown, maximizes the information entropy of the prediction probability of the unknown labels by the deep multi-label classification network in the loss function during model training, is used to solve the SPML problem, so that the model can be immune to false negative label noise, learns by using the supervision information provided by the true positive label, and makes a strong distinguishable prediction for the unknown label, the EMR method is easy to implement, does not introduce additional learnable parameters, can be combined with any deep multi-label classification network, has excellent performance, and is suitable for processing super large-scale multi-label image data sets.
[0042] Compared with the ROLE method, the EPR regularization method introduced by the ROLE method needs to statistically calculate the average number of true positive labels of each sample in the data set in advance, however, this is difficult to realize in actual application scenarios, because the number of true positive labels in the data set is unknown. In addition, the label estimation module used in the ROLE method introduces a large number of additional learnable parameters to the original network model, greatly increases the time and space requirements during network training, and the number of introduced learnable parameters is positively correlated with the number of categories and the number of samples in the data set, which indicates that the ROLE method is not suitable for super large-scale multi-label image data sets. The method proposed in the application does not introduce additional learnable parameters, is more widely applicable, can be combined with any deep multi-label classification network and any size of multi-label data set, and has a performance far superior to that of the ROLE method.
[0043] In order to verify the superiority of the application, the EMR method of the application and the AN method and the ROLE method of the prior art are respectively used to perform experimental verification on four large-scale multi-label image data sets, i.e., PASCAL VOC 2012 (VOC), MS-COCO 2014 (COCO), NUS-WIDE (NUS) and CUB-200-2011 (CUB), and the experimental results show that the method proposed in the application has excellent performance and surpasses all existing technical solutions. The experimental results for the optimal performance in each data set are shown in the following table:
[0044]
[0045] The present application aims to solve the robust learning problem of deep multi-label classification network in the case that each sample only has a single positive label and the rest of the labels are all missing. The present application proposes a binary cross-entropy loss function regularization mechanism based on information entropy maximization. The generalization performance of the deep multi-label classification network can be greatly improved by using the method proposed in the present application, and it performs well in the case of extreme label missing.
[0046] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made by those skilled in the art based on the spirit and principles of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A deep multi-label classification network robust training method based on an information entropy maximization regularization mechanism, characterized in that, Comprise: The deep multi-label classification network regards the prediction of each unknown label of each sample as a discrete random variable, including two events of predicting as a positive label and predicting as a negative label, and the probability of predicting as a positive label is the unknown label prediction probability output by the deep multi-label classification network, and the information entropy of the unknown label prediction probability output by the deep multi-label classification network is maximized; The method assumes that a sample pair with only a single positive label is given wherein is the n th input image, is the n th image corresponding label vector, then the loss function is: wherein, is the output of the model with respect to , is the predicted probability of the model on the c th class, C is the number of classes. The regularization term is defined as: wherein, α is a hyperparameter that controls the strength of the regularization.
2. The deep multi-label classification network robust training method based on information entropy maximization regularization mechanism according to claim 1, characterized in that, The hyperparameters α are adjusted between 0 and 1. 3.The deep multi-label classification network robust training method based on information entropy maximization regular mechanism according to claim 2, characterized in that, The probability of being predicted as a positive label in the method is the unknown label prediction probability of the network output , the probability of being predicted as a negative label is 1- . 4.The deep multi-label classification network robust training method based on information entropy maximization regularization mechanism according to claim 1, wherein, The method controls the unknown label prediction probability to tend to 0.
5. 5.The deep multi-label classification network robust training method based on information entropy maximization regularization mechanism according to claim 1, wherein, In the method, a discrete random variable is assumed The information entropy of the discrete random variable is defined as H X ) =E[ I X )]=E[-ln( P X ))] where P X ) is the probability mass function of X E [.] is the expected function, I X ) is the information content of X . 6.The method of claim 5, wherein, In the method, if the information entropy is known or has been estimated P X ) after the estimate is made, the information entropy is represented by the formula: wherein b is the base used for the logarithm.
7. The method of claim 6, wherein, In the method when b = 2, the unit of information entropy is bit; When b = e , the unit of information entropy is nat; When b = 10, the unit of information entropy is Hart. 8.The deep multi-label classification network robust training method based on information entropy maximization regular mechanism according to claim 7, wherein, The deep multi-label classification network in the method regards the prediction of each unknown label of each sample as a discrete random variable , event that is, predicted as a positive label, event that is, predicted as a negative label, then that is, the unknown label prediction probability output by the network, that is, the probability of being predicted as a negative label.