A Noisy Multi-Label Classification Method Based on the Max-Margin Mechanism

Abstract

The present invention proposes a noisy multi-label classification method based on the maximum margin mechanism, which is applicable to the multi-label data classification scenario where the label space contains noise. The method includes the following steps: (1) The user collects multi-label data on the premise of ensuring complete acquisition of true labels; (2) Use label confidence to represent label information and initialize the confidence value for each label; (3) Convert the noisy multi-label classification problem into a convex optimization problem of minimizing the empirical loss, hinge loss, and confidence constraint, and obtain the optimal solution of the convex optimization problem to generate a multi-label classification model; (4) Predict the unseen examples according to the trained classification model to obtain the classification results of each category; (5) If the user is satisfied with the prediction results, end, otherwise go to step (2) to re-initialize the confidence value for each label.

Description

Technical Field

[0001] The present invention relates to a multi-label data classification method, specifically a noisy multi-label classification method based on the maximum margin mechanism, belonging to the technical field of weak supervision classification. Background Art

[0002] In multi-label learning, an example is associated with multiple correct semantic labels. The goal of the learning system is to construct a mapping relationship from the input space (feature space) to the output space (label space) using the multi-label dataset, for predicting a set of correct labels for unseen examples. The most critical assumption in traditional multi-label learning tasks is that each example in the multi-label dataset is assigned its correct set of semantic labels. However, in real life, it is very difficult to accurately assign labels to each example. In many tasks, there may be situations such as mislabeling, missing labels, or redundant labels, resulting in noise in the label information of the multi-label dataset. Frameworks for studying such multi-label datasets are collectively referred to as the noisy multi-label learning framework.

[0003] The present invention studies the noisy multi-label learning task with redundant labels. Each example is associated with a set of candidate labels, and the true labels of the examples are unknown but hidden in the set of candidate labels. The goal of the learning system is to learn the mapping relationship from the input space to the output space under the known weak supervision information for predicting a set of correct labels for unseen examples. To solve the problem of noise in the dataset, the existing methods are mainly divided into two categories: disambiguation methods and non-disambiguation methods. The disambiguation methods are divided into two stages. In the first stage, the set of candidate labels in the dataset is disambiguated to obtain a dataset without noisy labels. In the second stage, the disambiguated dataset is used to train the model using multi-label learning methods. The key point of such methods lies in how to disambiguate the dataset, which will directly affect the classification performance of the learning model. The non-disambiguation methods directly use the information of the original noisy dataset for model learning. Summary of the Invention

[0004] In the noisy multi-label classification task, due to the noise in the set of candidate labels, traditional multi-label classification models cannot perform excellently. In the existing methods, methods such as meta-learning and generative adversarial networks are often used to handle the noisy multi-label classification problem, while traditional machine learning models are rarely used. Therefore, the present invention proposes a noisy multi-label classification method based on traditional machine learning models, improving the traditional SVM model to make it suitable for the noisy multi-label classification task. Directly learn on the noisy multi-label dataset, construct a prediction model, and improve the performance of the multi-label classification model.

[0005] Technical solution: The present invention proposes a noisy multi-label classification method based on the maximum margin mechanism, which integrates noisy label information into the traditional SVM framework, learns the confidence for each label in the candidate label set, and transforms the noisy multi-label classification problem into a convex optimization problem of minimizing the empirical loss, hinge loss and confidence constraint. The method includes the following steps: (1) The user collects multi-label data (inevitably containing noise) on the premise of ensuring complete acquisition of true labels; (2) Use label confidence to represent label information and initialize the confidence value for each label; (3) Transform the noisy multi-label classification problem into a convex optimization problem of minimizing the empirical loss, hinge loss and confidence constraint, and obtain the optimal solution of the convex optimization problem to get the classification model; (4) Predict the unseen examples according to the trained classification model to obtain the classification results of each category; (5) If the user is satisfied with the prediction results, end, otherwise go to step (2) to re-initialize the confidence value for each label.

[0006] The present invention is further preferably, and the step (2) specifically includes:

[0007] Suppose represents the d-dimensional example space, and Y = {y1, y2,..., y q} represents the label space containing q categories. Given the noisy multi-label data set D = {(x1, S1), (x2, S2),..., (x m , S m )}, which contains m examples, where is the d-dimensional attribute vector, and is a set of candidate label sets corresponding to the example x i . In addition, represents a set of non-candidate label sets corresponding to the example x i ; Y i is the true label set of the example x i ;

[0008] In order to better distinguish the true labels and noise labels in the candidate label set, define the confidence θ ij to represent the possibility that the label y j can be the true label of the example x i . The larger θ ij is, the greater the possibility that the label y j is the true label of the example x i ; when θ ij = 1, it means that the label y j is the true label of the example x i , and when θ ij = 0, it means that the label y j is an irrelevant label of the example x i ; Obviously, when When θ ij =0, which is the same as example x i All tags in the corresponding non-candidate tag set are irrelevant tags for the example;

[0009] Therefore, when initializing the label confidence, when y j ∈S i hour, when When θ ij =0.

[0010] The present invention is further preferred, wherein step (3) specifically comprises:

[0011] Based on the above symbolic definition, for each mark y in the mark space Y p Learn a linear classifier f respectively p (x i )=<ω p ,x i >+b p , where ω p ∈R d is the weight vector of the linear classifier, b p ∈R is the bias of the linear classifier; for this, the noisy multi-label classification problem is converted into a convex optimization problem of minimizing the empirical loss, hinge loss and confidence constraint. The objective function is as follows:

[0012]

[0013]

[0014] The first term in the above formula is the structural loss function of the model, which constrains the complexity of the model itself and prevents the model from overfitting to a certain extent; the second term is the empirical loss function of the model, which optimizes this loss function to make the model fit the training samples and prevent the model from underfitting; where ξ ik is a slack variable used to measure the degree to which the sample violates the constraint. The greater the degree to which the sample violates the constraint, the greater the value of the slack variable; ξ ik is through example x i Mark y in the non-candidate marker set k The learned slack variable is shown in formula (2). Considering the ambiguity of the sample, in order to ensure the fairness of the model in learning the label confidence, the confidence is constrained in the third term.

[0015]

[0016] The objective of formula (1) is to minimize the loss error of the model, including the structural loss, empirical loss of the model, and the constraint on the confidence value. Since there are many variables to be learned in the formula, an alternating iterative optimization strategy can be adopted. The optimization is divided into two stages. In the first stage, the model weights ω and biases b are solved while fixing the confidence θ. In the second stage, the confidence θ is solved while fixing the model weights ω and biases b. In each stage, a quadratic programming toolkit is used to solve the problem.

[0017] Further preferably, in the present invention, (3.1) The first stage: When the confidence θ is fixed, the optimization objective of formula (1) is as follows:

[0018]

[0019] Construct the Lagrangian function of formula (3) as follows:

[0020]

[0021] In the above formula, α ik ≥0, β ik ≥0;

[0022] Take the derivatives of formula (4) with respect to ω p , b p , ξ ik respectively and set the derivatives to 0, then the following formulas can be obtained:

[0023]

[0024]

[0025] C = α ik + β ik (7) In the above formula, II(·) is the indicator function, which takes the value of 1 when · is true and 0 otherwise. Substitute formulas (5)-(7) into formula (4) to obtain the dual problem of the original problem The optimization objective is as follows:

[0026]

[0027] (3.2) The second stage: When the model weights ω and biases b are fixed, the optimization objective of formula (1) is as follows:

[0028]

[0029] Advantages of the present invention:

[0030] 1. The present invention converts the problem of noisy multi-label classification into a convex optimization problem of minimizing empirical loss, hinge loss, and confidence constraints, and learns a classification model for each label through an alternating iterative optimization strategy, maximizing the margin between the weighted output of the model in the candidate label set and the maximum output of the model in the non-candidate label set.

[0031] 2. Learn label confidence and multi-label classification models simultaneously on a dataset with noisy labels. The effectiveness of traditional machine learning models is verified, and the impact of noisy labels is reduced to a certain extent. Brief Description of the Drawings

[0032] Figure 1 is the working flowchart of the document classification device;

[0033] Figure 2 is the flowchart of the method of the present invention;

[0034] Figure 3 is the flowchart of the classification model construction stage. Detailed Embodiments

[0035] The present invention will be further clarified below in conjunction with the drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. It should be noted that the terms "front", "rear", "left", "right", "up" and "down" used in the following description refer to the directions in the drawings, and the terms "inner" and "outer" refer to the directions towards or away from the geometric center of a specific component, respectively.

[0036] As Figure 1 shown, take the process of document classification as an example. The document storage device contains a large number of ambiguous topic documents, each document is labeled with multiple topics, and only a small number of topics belong to the true topics of this document, and the number of true topics is unknown. When judging the true topics of documents, the user first selects N sample documents from all the ambiguous documents and submits them to the document classifier (computer model) for processing. At this time, features need to be extracted from the documents to be classified. This step generally uses some feature dimensionality reduction techniques for Chinese texts, such as Term Frequency (denoted as TF), Term Frequency-Inverse Document Frequency (denoted as TF-IDF), etc. After extracting the features of the documents, the document classifier proposed by the present invention first initializes the confidence value for each topic and generates a linear classifier for each topic, and then obtains the final classification model by solving a convex optimization problem of minimizing empirical loss, hinge loss, and confidence constraints. When predicting a document with an unknown topic, features are extracted in the same way, and then this classification model is used to predict and return the classification result. As Figure 1As shown. If the user is not satisfied with the obtained result, more sample documents can be selected from the document library and fed back to the document classification device.

[0037] The method involved in the present invention is as Figure 2 shown. Step 10 is the starting action. Assume that the set of sample documents submitted by the user is D = {(x1, S1), (x2, S2), …, (x m , S m )}, where the sample is a d-dimensional feature vector, and is a set of candidate label sets corresponding to the sample x i . Y i is the true label set of the sample x i , Y i is unknown and In step 11, the confidence value is initialized for each label. In step 12, first a linear classifier is generated for each label, and then the final classification model is obtained by solving a convex optimization problem that minimizes the empirical loss, hinge loss, and confidence constraint. The specific transformation is introduced in Figure 3 . In step 13, the unclassified documents in the document storage device are classified using the prediction model obtained in step 12, and the classification result is returned. Finally, it enters the end state of step 14.

[0038] Figure 3 gives a detailed description of step 12. Step 1200 is the starting action. Steps 1201 to 1205 form a loop body, and the objective function is iteratively optimized and solved once in each round of the loop. Specifically, for each label y p in the label space Y, a linear classifier f p (x i ) = <ω p , x i > + b p is learned respectively, where ω p ∈R d is the weight vector of the linear classifier, and b p ∈R is the bias of the linear classifier. For this, the noisy multi-label classification problem is converted into a convex optimization problem that minimizes the empirical loss, hinge loss, and confidence constraint, and the objective function is as follows:

[0039]

[0040] In the above formula, the first term is the structural loss function of the model, which restricts the complexity of the model itself to a certain extent to prevent overfitting of the model. The second term is the empirical loss function of the model. Optimizing this loss function allows the model to fit the training samples and prevents the problem of underfitting of the model. Among them, ξ ikis a slack variable, which is used to measure the degree to which a sample violates the constraints. The greater the degree to which a sample violates the constraints, the larger the value of the slack variable. ξ ik is the slack variable learned through the example x i and the label y that is not in the candidate label set k as shown in Equation (2). Considering the polysemy of samples, in order to ensure the fairness of the model's learning of label confidence, the confidence is constrained in the third term.

[0041]

[0042] The objective of Equation (1) is to minimize the loss error of the model, including the structural loss, empirical loss of the model, and the constraint on the confidence value. Since there are many variables to be learned in the equation, an alternating iterative optimization strategy can be adopted. The optimization is divided into two stages. In the first stage, the model weights ω and bias b are solved while fixing the confidence θ. In the second stage, the confidence θ is solved while fixing the model weights ω and bias b. In each stage, a quadratic programming toolkit is used to solve. When the confidence θ is fixed in step 1203, the optimization objective of Equation (1) is as follows:

[0043]

[0044] Construct the Lagrangian function of Equation (3) as follows:

[0045]

[0046] In the above formula, α ik ≥0, β ik ≥0. Take the derivatives of Equation (4) with respect to ω p , b p , ξ ik respectively and set the derivatives to 0 to obtain the following formulas:

[0047]

[0048]

[0049] C = α ik + β ik (7)

[0050] In the above formula, II(·) is the indicator function. When · is true, the value of the indicator function is 1, otherwise the value is 0. Substitute Equations (5)-(7) into Equation (4) to obtain the optimization objective of the dual problem of the original problem as follows:

[0051]

[0052] When the model weights ω and biases b are fixed in step 1204, the optimization objective of formula (1) is as follows:

[0053]

[0054]

[0055] After T rounds of alternating iterations, the weights ω and biases b of the final classification model are obtained, and then it enters the end state of step 1206.

[0056] The present invention provides a noisy multi-label classification method based on the maximum margin mechanism, integrates noisy label information into the traditional SVM framework, learns the confidence for each label in the candidate label set, and converts the noisy multi-label classification problem into a convex optimization problem of minimizing the empirical loss, hinge loss and confidence constraint. By using the alternating iteration optimization strategy, a classification model is learned for each label, so that the margin between the weighted output of the model in the candidate label set and the maximum output of the model in the non-candidate label set is maximized. Briefly speaking, this algorithm simultaneously learns the label confidence and the multi-label classification model on the dataset with noisy labels. Learning and training based on this method can improve the effectiveness, stability and robustness of the multi-label classification model.

[0057] The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features.

Claims

1. A noisy multi-label classification method based on the maximum margin mechanism, characterized in that: The method comprises the following steps: Step (1): The user collects multi-label data on the premise of ensuring complete acquisition of true labels. Step (2): The label confidence is used to represent label information, and a confidence value is initialized for each label. The specific steps of step (2) include: Hypothesis denotes a d-dimensional example space, and Y = {y1, y2, …, y q} denotes a label space with q categories; Given a noisy multi-label dataset D = {(x1, S1), (x2, S2), …, (x m , S m ), which contains m examples, where is a d-dimensional attribute vector, and is a set of candidate label sets corresponding to the example x i ; In addition,[[]] denotes a set of non-candidate label sets corresponding to the example x i ; Y i is the true label set of the example x i ;​ To better distinguish the true tags from the noise tags in the candidate tag set, the confidence θ is defined ij represents the tag y j has the possibility of being the true tag of the example x i , θ ij The larger θ is, the greater the possibility that the tag y j is the true tag of the example x i ; when θ ij = 1, it means that the tag y j is the true tag of the example x i ; when θ ij = 0, it means that the tag y j is an irrelevant tag of the example x i ; obviously, when , θ ij = 0, that is, all tags in the non-candidate tag set corresponding to the example x i are irrelevant tags of this example; Therefore, when initializing the marker confidence, when y j ∈S i at that time, When at that time, θ ij = 0; Step (3): The noisy multi-label classification problem is converted into a convex optimization problem of minimizing empirical loss, hinge loss and confidence constraint, and the optimal solution of the convex optimization problem is obtained to generate a multi-label classification model. The specific steps of step (3) include: For each label y in the label space Y p learn a linear classifier f p (x i ) = <ω p , x i > + b p , where ω p ∈ R d is the weight vector of the linear classifier, and b p ∈ R is the bias of the linear classifier; for this, the noisy multi-label classification problem is converted into a convex optimization problem of minimizing the empirical loss, hinge loss, and confidence constraint, and the objective function is as follows: The first term in the above formula is the structural loss function of the model, which constrains the complexity of the model itself and prevents the model from overfitting to a certain extent; the second term is the empirical loss function of the model, which optimizes this loss function to make the model fit the training samples and prevent the model from underfitting; where ξ ik is a slack variable used to measure the degree to which the sample violates the constraint. The greater the degree to which the sample violates the constraint, the greater the value of the slack variable; ξ ik is through example x i Mark y in the non-candidate marker set k The learned slack variable is shown in formula (2). Considering the ambiguity of the sample, in order to ensure the fairness of the model in learning the label confidence, the confidence is constrained in the third term. The objective of formula (1) is to minimize the loss error of the model, including the structural loss, empirical loss of the model and the constraint on the confidence value. Since there are many variables to be learned in the formula, an alternating iterative optimization strategy is adopted. The optimization is divided into two stages. In the first stage, the model weights ω and bias b are solved while fixing the confidence θ. In the second stage, the confidence θ is solved while fixing the model weights ω and bias b. In each stage, a quadratic programming toolkit is used to solve. Among them, (3.1) The first stage: When the confidence θ is fixed, the optimization objective of formula (1) is as follows: Construct the Lagrangian function of formula (3) as follows: In the above formula, α ik ≥ 0, β ik ≥ 0; Derive formula (4) with respect to ω p , b p , ξ ik Taking the derivative and setting it to zero gives the following formula: C＝α ik +β ik (7) In the above formula is an indicator function. When · is true, the value of the indicator function is 1; otherwise, the value is 0. Substituting formulas (5)-(7) into formula (4) gives the dual problem of the original problem The optimization objective is as follows: (3.2) The second stage: When the model weights ω and bias b are fixed, the optimization objective of formula (1) is as follows: Step (4): Predict the unseen examples according to the trained classification model to obtain the classification results of each category. Step (5): If the user is satisfied with the prediction result, end. Otherwise, go to step (2) to re-initialize the confidence value for each label.

Images