A Multi-Label Classification Method and System Based on Label Dependency Measurement

Abstract

This invention discloses a multi-label classification method and system based on label dependency measurement, belonging to the field of natural language processing technology. The technical problem to be solved by this invention is the difficulty in classifying relationships between multiple labels and the difficulty in convergence. The technical solution adopted is as follows: S1, process the given text; S2, define the correlation matrix C between multiple labels; S3, input the text tensor T into the embedding layer and the core layer in sequence to obtain the text embedding vector; S4, initialize the label embedding using the number of categories in the classification label set D; S5, use the label information contained in the multi-label correlation coefficient matrix C to calculate the inner product attention of the text embedding and the label embedding, thereby integrating the relationship matrix information into the text embedding vector X for re-representation; S6, input the re-represented text embedding vector in step S5 into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a multi-label classification method and system based on label dependency measurement. Background Technology

[0002] Currently, various fields have accumulated massive amounts of diverse data. To extract the deeper information contained within this data, traditional methods of manually summarizing, analyzing, integrating, and classifying data are inefficient and have been replaced by neural network methods, represented by deep learning. Classification is a core problem in data analysis; classifying data based on its features is a common approach during initial data processing.

[0003] Classification problems can be divided into single-label classification problems and multi-label classification problems. In single-label classification, the original data is associated with only one label in the label set, while in multi-label classification, the original data can be associated with multiple labels in the label set. Classification problems are widely used in the field of natural language processing. For example, models are used to identify email content and determine whether the email is legitimate or spam. However, compared to single-label classification problems that assume that the labels are independent, multi-label classification problems are more common in real life. For example, determining the attribute of a news article promoting a star athlete's participation in football training could fall under either the "sports" category or the "person" category. Multi-label classification of raw data is a common way for humans to process daily data, and it is more in line with reality and human cognition. Multi-label classification not only needs to determine the category of all texts, but also needs to consider the correlation, inclusion, and exclusion relationships between labels. For example, when determining a person's occupation based on the article's description, if the initial occupation is determined to be {poet}, then the probability of classifying the person's occupation as {poet, writer} is often higher than classifying it as {poet, athlete}. This utilizes the relationships between multiple labels. Therefore, solving multi-label classification problems is more challenging, valuable, and difficult.

[0004] In the field of multi-label classification, there are several categories: rule-based multi-label classification, traditional machine learning-based multi-label classification, and deep learning-based multi-label classification. Rule-based methods summarize existing classification experience from a large number of examples, finding keywords, key terms, and classification logic, accumulating vast classification experience for each category. This method is labor-intensive, the classification rules are relatively fixed, and the generalization ability is weak. Machine learning models treat multi-label classification as multiple binary classification problems, using algorithms such as Support Vector Machines (SVM) and maximum entropy models for multiple binary classifications. However, this method does not consider the relationships between labels, resulting in poor classification performance. K-nearest neighbor algorithms describe the correlation between labels, but the classification effect relies excessively on the variable initialization process. Deep learning-based multi-label classification uses multi-layered MLP networks to characterize the relationships between labels, attempting to learn several higher-order relationships through iterative training. However, the lack of a training objective during model iteration makes convergence difficult and also leads to excessive computational complexity. Summary of the Invention

[0005] The technical objective of this invention is to provide a multi-label classification method and system based on label dependency measurement to solve the problems of difficulty in classifying relationships between multiple labels and difficulty in convergence.

[0006] The technical objective of this invention is achieved as follows: a multi-label classification method based on label dependency measurement, the specific method of which is as follows:

[0007] S1. Process the given text: Set the maximum character length in the text to L, and pad any insufficient characters with "padding"; and generate a text tensor T with dimension R using a dictionary index mapping table. L Where R represents the real number space and L is a positive integer;

[0008] S2. Define the correlation matrix C between multiple labels, and define the set of category labels as D = {d1, d2, ..., d...} i ,…,d m There are m categories in total; and the set of category labels to which the current text belongs is set as I = {i1, i2, ..., i...} k} is a subset of set D; where d i Let represent the i-th category, k ≤ m, and m and k are positive integers;

[0009] S3. After sequentially inputting the text tensor T into the embedding layer and the core layer, we obtain the text embedding vector X, which has dimension R. L×E Where E represents the embedding dimension and is a positive integer;

[0010] S4. Using the number of categories in the category label set D, initialize the label embedding Y, with dimension R. m×E ;

[0011] S5. Utilize the label information contained in the multi-label correlation coefficient matrix C to calculate the inner product attention between the text embedding and the label embedding, thereby integrating the relation matrix information into the text embedding vector X for re-representation;

[0012] S6. Input the re-represented text embedding vector from step S5 into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters.

[0013] S7. For each category label embedding in set I, traverse the multi-label correlation matrix C and calculate the label dependency loss function between each pair of categories; and perform a secondary parameter update on the category label embeddings involved, while other parameters are not updated.

[0014] S8. During the training phase, input text and classification labels, calculate cross-entropy loss and label dependency metric loss respectively, and update the parameters multiple times (twice) according to the methods in steps S6 and S7.

[0015] S9. In the prediction phase, input text and output the multi-class label prediction results of the text.

[0016] Preferably, the correlation matrix C between the multiple tags is as follows:

[0017]

[0018] Where, d i With d j Relevance includes linear correlation and inclusion; other situations include d i With d j Exclusive and non-exclusive relationships; d i ∈D,d j ∈D, where D is the set of category labels, i≤m, j≤m;

[0019] The core layer is a text sentence representation processing layer, and the core layer adopts LSTM, BERT or Roberta network structure.

[0020] Preferably, the label information contained in the multi-label correlation coefficient matrix C is used to calculate the inner product attention between the text embedding and the label embedding, thereby incorporating the relation matrix information into the text embedding vector X for re-representation, as follows:

[0021] S501. Calculate the inner product of the text embedding vector X and the label embedding Y:

[0022] Where X = {x1, x2, ..., x} L};Y={y1,y2,…,y m};x i ∈X; y j ∈Y;

[0023] S502. Calculate the weight allocation coefficients in the attention mechanism, using the following formula:

[0024]

[0025] Where C is the multi-label correlation coefficient matrix; the dimension of α is R. L×m The softmax function performs normalization on the embedding dimension;

[0026] S503. Utilizing the ensemble effect of the weight allocation coefficient α, the text embedding vector X is re-represented as X', where the dimension of X' is R. L×E The formula is as follows:

[0027] X' = ​​α*X.

[0028] Preferably, the text embedding vector X is input into the Maxpooling layer and the linear classification layer, the forward cross-entropy loss is calculated, and the gradient update of all parameters is performed using the stochastic gradient descent algorithm, as follows:

[0029] S601. Input the text embedding vector X' into the Maxpooling layer to obtain semantic information O, which has more obvious features in the sentence length dimension. The output dimension of semantic information O is R. E The formula is:

[0030] O = Maxpooling(X');

[0031] S602. Integrate the information from O in step S601, select an MLP network layer for linear classification to obtain the classification tensor O', and the output dimension of the classification tensor O' is R. 2 The formula is:

[0032] O' = MLP(O)

[0033] S603, Using the classification tensor O' and the classification label i ∈I calculate the cross-entropy loss, and then calculate the overall loss using the following formula:

[0034] loss = ∑ i∈I Cross_entropy(O',label i ).

[0035] More preferably, for each category label embedding in set I, the multi-label correlation matrix C is traversed, and the pairwise label dependency measurement loss function is calculated as follows:

[0036] S701. Transforming Classification Label Embedding into a Probability Distribution: Input Y into the softmax function, which operates on the last dimension. The formula is as follows:

[0037] P = softmax(Y);

[0038] S702, Based on the classification label set I = {i1, i2, ..., i} of the current sample. k}, and filter out the tag relationship set D from the multi-tag correlation matrix. + With D - Specifically:

[0039] D + ={(i,j)|c ij =1, i∈I, j∈D};

[0040] D - ={(i,j)|c ij =0, i∈I, j∈D};

[0041] Among them, c ij ∈C; set D + Indicates the classification label i of the current sample * ∈I all labels that have a correlation; set D - Indicates the classification label i of the current sample * All tags that have an exclusionary relationship or no relationship;

[0042] S703. Based on steps S701 and S702, calculate the label dependency metric loss function, as shown in the formula:

[0043]

[0044] Where, p i ,p j ,p q ,p v The probability distribution P and M are derived from step S701. s (p i ,p j () represents the distance based on KL divergence, and the formula is:

[0045]

[0046] Among them, D KL(p||q) is the KL divergence of probability distribution q based on probability distribution p, used to measure the difference between probability distribution q and probability distribution p; M s (p,q) satisfies the three elements of distance: the distance between p and q is nonnegative, the distance between p and q is symmetric, and the distance between p and q satisfies the triangle inequality. KL divergence is not a distance in the true sense.

[0047] A multi-label classification system based on label dependency metric, the system comprising,

[0048] The definition module is used to build a pre-classification label system and define the relationships between labels;

[0049] The acquisition module is used to input the natural language text model into the word embedding layer and the core layer to obtain text embedding vectors;

[0050] A module is established to combine the multi-label correlation coefficient matrix to build a representation system of text embedding vectors and label embeddings, and to perform forward calculation of loss values, and then update all parameters once through the backpropagation algorithm;

[0051] The update module is used to perform a secondary update on the label embedding vector parameters by building a label dependency metric loss function based on the multi-label correlation coefficient matrix.

[0052] The training module is used during the training phase to take input text and classification labels, calculate cross-entropy loss and label dependency metric loss respectively, and update the parameters twice.

[0053] The prediction module is used to take text as input and output multi-class label prediction results for the text during the prediction phase.

[0054] Preferably, the definition module includes,

[0055] The processing submodule is used to process the given text. Specifically, it sets the maximum character length in the text to L, pads any insufficient characters with "padding", and generates a text tensor T with dimension R using a dictionary index mapping table. L Where R represents the real number space and L is a positive integer;

[0056] Define a submodule to define the correlation matrix C between multiple labels, and define the set of category labels as D = {d1, d2, ..., d...} i ,…,d m There are m categories in total, and the set of category labels to which the current text belongs is set as I = {i1, i2, ..., i}. k} is a subset of set D; where d i Let represent the i-th category, k ≤ m, and m and k are positive integers;

[0057] The acquisition module sequentially inputs the text tensor T into the embedding layer and the core layer to obtain a text embedding vector X with dimension R. L×E Where E represents the embedding dimension and is a positive integer;

[0058] The establishment module includes,

[0059] The initialization submodule is used to initialize the label embedding Y with dimension R using the number of categories in the category label set D. m×E ;

[0060] The computation submodule is used to calculate the inner product attention of text embedding and label embedding using the label information contained in the multi-label correlation coefficient matrix C, thereby integrating the relation matrix information into the text embedding vector X for re-representation;

[0061] The update submodule is used to input the re-represented text embedding vector into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters.

[0062] More preferably, the correlation matrix C between the multiple tags is as follows:

[0063]

[0064] Where, d i With d j Relevance includes linear correlation and inclusion; other situations include d i With d j Exclusive and non-exclusive relationships; d i ∈D,d j ∈D, where D is the set of category labels, i≤m, j≤m;

[0065] The specific working process of the calculation submodule is as follows:

[0066] (i) Calculate the inner product of the text embedding vector X and the label embedding Y:

[0067] Where X = {x1, x2, ..., x} L};Y={y1,y2,…,y m};x i ∈X; y j ∈Y;

[0068] (II) Calculate the weight allocation coefficients in the attention mechanism, using the following formula:

[0069]

[0070] Where C is the multi-label correlation coefficient matrix; the dimension of α is R. L×m The softmax function performs normalization on the embedding dimension;

[0071] (III) Utilizing the ensemble effect of the weight allocation coefficient α, the text embedding vector X is re-represented as X', where the dimension of X' is R. L×E The formula is as follows:

[0072] X' = ​​α*X;

[0073] The specific working process of the update submodule is as follows:

[0074] (1) Input the text embedding vector X' into the Maxpooling layer to obtain semantic information O with more obvious features in the sentence length dimension. The output dimension of semantic information O is R. E The formula is:

[0075] O = Maxpooling(X');

[0076] (2) Integrate the speech information O from step S601, select an MLP network layer for linear classification to obtain the classification tensor O', and the output dimension of the classification tensor O' is R. 2 The formula is:

[0077] O' = MLP(O)

[0078] (3) Using the classification tensor O' and the classification label i ∈I calculate the cross-entropy loss, and then calculate the overall loss using the following formula:

[0079] loss = ∑ i∈I Cross_entropy(O',label i );

[0080] The calculation process of the label dependency metric loss function is as follows:

[0081] ① Transforming category label embedding into a probability distribution: Input Y into the softmax function, which operates on the last dimension. The formula is:

[0082] P = softmax(Y);

[0083] ② Based on the set of classification labels I = {i1, i2, ..., i} to which the current sample belongs k}, and filter out the tag relationship set D from the multi-tag correlation matrix. + With D - Specifically:

[0084] D+ ={(i,j)|c ij =1, i∈I, j∈D};

[0085] D - ={(i,j)|c ij =0, i∈I, j∈D};

[0086] Among them, c ij ∈C; set D + Indicates the classification label i of the current sample * ∈I all labels that have a correlation; set D - Indicates the classification label i of the current sample * All tags that have an exclusionary relationship or no relationship;

[0087] ③ Based on steps S701 and S702, calculate the label dependency metric loss function, as shown in the formula:

[0088]

[0089] Where, p i ,p j ,p q ,p v The probability distribution P and M are derived from step S701. s (p i ,p j () represents the distance based on KL divergence, and the formula is:

[0090]

[0091] Among them, D KL (p||q) is the KL divergence of probability distribution q based on probability distribution p, used to measure the difference between probability distribution q and probability distribution p; M s (p,q) satisfies the three elements of distance: the distance between p and q is nonnegative, the distance between p and q is symmetric, and the distance between p and q satisfies the triangle inequality. KL divergence is not a distance in the true sense.

[0092] An electronic device includes: a memory and at least one processor;

[0093] The memory contains computer programs;

[0094] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multi-label classification method based on the label dependency metric described above.

[0095] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the multi-label classification method based on label dependency metric as described above.

[0096] The multi-label classification method and system based on label dependency measurement of the present invention have the following advantages:

[0097] (i) This invention solves the problems of difficulty in modeling the relationship between multiple labels and difficulty in model convergence in the existing technical solutions by establishing a label dependency measurement loss function based on the multi-label correlation coefficient matrix, and characterizes the relationship between labels such as correlation, inclusion, and exclusion, thereby improving the accuracy of multi-label classification.

[0098] (II) Based on the pre-classification label system, this invention introduces a multi-label correlation coefficient matrix, which solves the problem that traditional machine learning algorithms cannot characterize the relationship between multiple labels, and makes the label relationship definition more refined.

[0099] (III) After multiple iterations of training, the classification of this invention can not only continue to converge, but also capture the interdependencies between labels through metric learning, thereby further improving the classification accuracy of multi-label text. Attached Figure Description

[0100] The invention will be further described below with reference to the accompanying drawings.

[0101] Appendix Figure 1 The flowchart shows the process of a multi-label classification method based on label dependency measurement.

[0102] Appendix Figure 2 This is an architecture diagram of a multi-label classification system based on label dependency metrics. Detailed Implementation

[0103] The multi-label classification method and system based on label dependency measurement of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0104] Example 1:

[0105] As attached Figure 1 As shown in the figure, this embodiment provides a multi-label classification method based on label dependency measurement, which is as follows:

[0106] S1. Process the given text: Set the maximum character length in the text to L, and pad any insufficient characters with "padding"; and generate a text tensor T with dimension R using a dictionary index mapping table. L Where R represents the real number space and L is a positive integer;

[0107] S2. Define the correlation matrix C between multiple labels, and define the set of category labels as D = {d1, d2, ..., d...} i ,…,d m There are m categories in total; and the set of category labels to which the current text belongs is set as I = {i1, i2, ..., i...} k} is a subset of set D; where d i Let represent the i-th category, k ≤ m, and m and k are positive integers;

[0108] S3. After sequentially inputting the text tensor T into the embedding layer and the core layer, we obtain the text embedding vector X, which has dimension R. L×E Where E represents the embedding dimension and is a positive integer;

[0109] S4. Using the number of categories in the category label set D, initialize the label embedding Y, with dimension R. m×E ;

[0110] S5. Utilize the label information contained in the multi-label correlation coefficient matrix C to calculate the inner product attention between the text embedding and the label embedding, thereby integrating the relation matrix information into the text embedding vector X for re-representation;

[0111] S6. Input the re-represented text embedding vector from step S5 into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters.

[0112] S7. For each category label embedding in set I, traverse the multi-label correlation matrix C and calculate the label dependency loss function between each pair of categories; and perform a secondary parameter update on the category label embeddings involved, while other parameters are not updated.

[0113] S8. During the training phase, input text and classification labels, calculate cross-entropy loss and label dependency metric loss respectively, and update the parameters multiple times (twice) according to the methods in steps S6 and S7.

[0114] S9. In the prediction phase, input text and output the multi-class label prediction results of the text.

[0115] The correlation matrix C between multiple labels in this embodiment is as follows:

[0116]

[0117] Where, d i With d j Relevance includes linear correlation and inclusion; other situations include d i With dj Exclusive and non-exclusive relationships; d i ∈D,d j ∈D, where D is the set of category labels, i≤m, j≤m.

[0118] In this embodiment, the core layer is a text sentence representation processing layer, and the core layer adopts LSTM, BERT, or Roberta network structure.

[0119] In step S5 of this embodiment, the label information contained in the multi-label correlation coefficient matrix C is used to calculate the inner product attention of the text embedding and the label embedding, thereby integrating the relation matrix information into the text embedding vector X for re-representation, as follows:

[0120] S501. Calculate the inner product of the text embedding vector X and the label embedding Y:

[0121] Where X = {x1, x2, ..., x} L};Y={y1,y2,…,y m};x i ∈X; y j ∈Y;

[0122] S502. Calculate the weight allocation coefficients in the attention mechanism, using the following formula:

[0123]

[0124] Where C is the multi-label correlation coefficient matrix; the dimension of α is R. L×m The softmax function performs normalization on the embedding dimension;

[0125] S503. Utilizing the ensemble effect of the weight allocation coefficient α, the text embedding vector X is re-represented as X', where the dimension of X' is R. L×E The formula is as follows:

[0126] X' = ​​α*X.

[0127] In this embodiment, step S6 involves inputting the text embedding vector X into the Maxpooling layer and the linear classification layer, calculating the forward cross-entropy loss, and using the stochastic gradient descent algorithm to update the gradients of all parameters, as detailed below:

[0128] S601. Input the text embedding vector X' into the Maxpooling layer to obtain semantic information O, which has more obvious features in the sentence length dimension. The output dimension of semantic information O is R. E The formula is:

[0129] O = Maxpooling(X');

[0130] S602. Integrate the information from O in step S601, select an MLP network layer for linear classification to obtain the classification tensor O', and the output dimension of the classification tensor O' is R. 2 The formula is:

[0131] O' = MLP(O)

[0132] S603, Using the classification tensor O' and the classification label i ∈I calculate the cross-entropy loss, and then calculate the overall loss using the following formula:

[0133] loss = ∑ i∈I Cross_entropy(O',label i ).

[0134] In this embodiment, step S7 involves iterating through the multi-label correlation matrix C for each category label embedding in set I and calculating the pairwise label dependency measurement loss function as follows:

[0135] S701. Transforming Classification Label Embedding into a Probability Distribution: Input Y into the softmax function, which operates on the last dimension. The formula is as follows:

[0136] P = softmax(Y);

[0137] S702, Based on the classification label set I = {i1, i2, ..., i} of the current sample. k}, and filter out the tag relationship set D from the multi-tag correlation matrix. + With D - Specifically:

[0138] D + ={(i,j)|c ij =1, i∈I, j∈D};

[0139] D - ={(i,j)|c ij =0, i∈I, j∈D};

[0140] Among them, c ij ∈C; set D + Indicates the classification label i of the current sample * ∈I all labels that have a correlation; set D - Indicates the classification label i of the current sample * All tags that have an exclusionary relationship or no relationship;

[0141] S703. Based on steps S701 and S702, calculate the label dependency metric loss function, as shown in the formula:

[0142]

[0143] Where, p i ,p j ,p q ,p v The probability distribution P and M are derived from step S701. s (p i ,p j () represents the distance based on KL divergence, and the formula is:

[0144]

[0145] Among them, D KL (p||q) is the KL divergence of probability distribution q based on probability distribution p, used to measure the difference between probability distribution q and probability distribution p; M s (p,q) satisfies the three elements of distance: the distance between p and q is nonnegative, the distance between p and q is symmetric, and the distance between p and q satisfies the triangle inequality. KL divergence is not a distance in the true sense.

[0146] Example 2:

[0147] As attached Figure 2 As shown, this embodiment provides a multi-label classification system based on label dependency measurement. The system includes,

[0148] The definition module is used to build a pre-classification label system and define the relationships between labels;

[0149] The acquisition module is used to input the natural language text model into the word embedding layer and the core layer to obtain text embedding vectors;

[0150] A module is established to combine the multi-label correlation coefficient matrix to build a representation system of text embedding vectors and label embeddings, and to perform forward calculation of loss values, and then update all parameters once through the backpropagation algorithm;

[0151] The update module is used to perform a secondary update on the label embedding vector parameters by building a label dependency metric loss function based on the multi-label correlation coefficient matrix.

[0152] The training module is used during the training phase to take input text and classification labels, calculate cross-entropy loss and label dependency metric loss respectively, and update the parameters twice.

[0153] The prediction module is used to take text as input and output multi-class label prediction results for the text during the prediction phase.

[0154] The definition module in this embodiment includes,

[0155] The processing submodule is used to process the given text. Specifically, it sets the maximum character length in the text to L, pads any insufficient characters with "padding", and generates a text tensor T with dimension R using a dictionary index mapping table. L Where R represents the real number space and L is a positive integer;

[0156] Define a submodule to define the correlation matrix C between multiple labels, and define the set of category labels as D = {d1, d2, ..., d...} i ,…,d m There are m categories in total, and the set of category labels to which the current text belongs is set as I = {i1, i2, ..., i}. k} is a subset of set D; where d i Let represent the i-th category, k ≤ m, and m and k are positive integers;

[0157] In this embodiment, the acquisition module sequentially inputs the text tensor T into the embedding layer and the core layer to obtain the text embedding vector X, which has a dimension of R. L×E Where E represents the embedding dimension and is a positive integer;

[0158] The establishment module in this embodiment includes,

[0159] The initialization submodule is used to initialize the label embedding Y with dimension R using the number of categories in the category label set D. m×E ;

[0160] The computation submodule is used to calculate the inner product attention of text embedding and label embedding using the label information contained in the multi-label correlation coefficient matrix C, thereby integrating the relation matrix information into the text embedding vector X for re-representation;

[0161] The update submodule is used to input the re-represented text embedding vector into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters.

[0162] The correlation matrix C between multiple labels in this embodiment is as follows:

[0163]

[0164] Where, d i With d j Relevance includes linear correlation and inclusion; other situations include d i With d j Exclusive and non-exclusive relationships; d i ∈D,d j∈D, where D is the set of category labels, i≤m, j≤m.

[0165] The specific working process of the calculation submodule in this embodiment is as follows:

[0166] (i) Calculate the inner product of the text embedding vector X and the label embedding Y:

[0167] Where X = {x1, x2, ..., x} L};Y={y1,y2,…,y m};x i ∈X; y j ∈Y;

[0168] (II) Calculate the weight allocation coefficients in the attention mechanism, using the following formula:

[0169]

[0170] Where C is the multi-label correlation coefficient matrix; the dimension of α is R. L×m The softmax function performs normalization on the embedding dimension;

[0171] (III) Utilizing the ensemble effect of the weight allocation coefficient α, the text embedding vector X is re-represented as X', where the dimension of X' is R. L×E The formula is as follows:

[0172] X' = ​​α*X.

[0173] The specific working process of the update submodule in this embodiment is as follows:

[0174] (1) Input the text embedding vector X' into the Maxpooling layer to obtain semantic information O with more obvious features in the sentence length dimension. The output dimension of semantic information O is R. E The formula is:

[0175] O = Maxpooling(X');

[0176] (2) Integrate the speech information O from step S601, select an MLP network layer for linear classification to obtain the classification tensor O', and the output dimension of the classification tensor O' is R. 2 The formula is:

[0177] O' = MLP(O)

[0178] (3) Using the classification tensor O' and the classification label i ∈I calculate the cross-entropy loss, and then calculate the overall loss using the following formula:

[0179] loss = ∑ i∈ICross_entropy(O',label i ).

[0180] The calculation process of the label dependency metric loss function is as follows:

[0181] ① Transforming category label embedding into a probability distribution: Input Y into the softmax function, which operates on the last dimension. The formula is:

[0182] P = softmax(Y);

[0183] ② Based on the set of classification labels I = {i1, i2, ..., i} to which the current sample belongs k}, and filter out the tag relationship set D from the multi-tag correlation matrix. + With D - Specifically:

[0184] D + ={(i,j)|c ij =1, i∈I, j∈D};

[0185] D - ={(i,j)|c ij =0, i∈I, j∈D};

[0186] Among them, c ij ∈C; set D + Indicates the classification label i of the current sample * ∈I all labels that have a correlation; set D - Indicates the classification label i of the current sample * All tags that have an exclusionary relationship or no relationship;

[0187] ③ Based on steps S701 and S702, calculate the label dependency metric loss function, as shown in the formula:

[0188]

[0189] Where, p i ,p j ,p q ,p v The probability distribution P and M are derived from step S701. s (p i ,p j () represents the distance based on KL divergence, and the formula is:

[0190]

[0191] Among them, D KL(p||q) is the KL divergence of probability distribution q based on probability distribution p, used to measure the difference between probability distribution q and probability distribution p; M s (p,q) satisfies the three elements of distance: the distance between p and q is nonnegative, the distance between p and q is symmetric, and the distance between p and q satisfies the triangle inequality. KL divergence is not a distance in the true sense.

[0192] Example 3:

[0193] This embodiment also provides an electronic device, including: a memory and a processor;

[0194] The memory stores the instructions executed by the computer.

[0195] The processor executes computer execution instructions stored in the memory, causing the processor to perform the multi-label classification method based on label dependency metric in any embodiment of the present invention.

[0196] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0197] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0198] Example 4:

[0199] This embodiment also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the multi-label classification method based on label dependency metric in any embodiment of the present invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0200] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0201] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0202] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0203] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-label classification method based on label dependency measurement, characterized in that, The method is as follows: S1. Process the given text: Set the maximum character length in the text to L, and pad any insufficient characters with "padding"; and generate a text tensor T using a dictionary index mapping table, with the text tensor T having dimensions of L. Where R represents the real number space and L is a positive integer; S2. Define the correlation matrix between multiple tags. At the same time, define the category tag set as ,common There are several categories; and the set of category tags to which the current text belongs is set to... , for set A subset of; in which, Indicates the first Categories ,and and It is a positive integer; S3. After sequentially inputting the text tensor T into the embedding layer and the core layer, the text embedding vector X is obtained. The text embedding vector X has the following dimensions: ;in, Indicates the dimension of the embedding, and is a positive integer; S4. Utilize category label sets The number of categories is calculated, and the initial label embedding Y is defined with dimensions of 1. ; S5. Utilizing multi-label correlation coefficient matrix The implicit label information is used to calculate the inner product attention between the text embedding and the label embedding, thereby incorporating the relation matrix information into the text embedding vector X for re-representation; S6. Input the re-represented text embedding vector from step S5 into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters. S7, For sets Embed each category label and traverse the multi-label correlation matrix. The loss function for measuring the label dependency between each pair of classes is calculated; and the secondary parameters of the relevant classification label embeddings are updated, while the other parameters are not updated. S8. During the training phase, input text and classification labels, calculate cross-entropy loss and label dependency metric loss respectively, and perform multiple parameter updates and iterations according to steps S6 and S7. S9. In the prediction phase, input text and output the multi-class label prediction results of the text. Among them, the correlation matrix between multiple tags Specifically as follows: ； in, and Relevance includes linear correlation and inclusion; other situations include... and It has an exclusive relationship and no relationship; , , For a collection of category tags, , ; The core layer is a text sentence representation processing layer, and the core layer adopts LSTM, BERT or Roberta network structure; Using multi-label correlation coefficient matrix The implicit label information is used to calculate the inner product attention between the text embedding and the label embedding, thereby incorporating the relation matrix information into the text embedding vector X for re-representation, as follows: S501. Calculate the inner product of the text embedding vector X and the label embedding Y: ; in, ; ; ; ; S502. Calculate the weight allocation coefficients in the attention mechanism, using the following formula: ； in, This is a multi-label correlation coefficient matrix; The dimension is The softmax function performs normalization on the embedding dimension; S503, Using weight allocation coefficients The integration effect re-represents the text embedding vector X as , The dimension is The formula is as follows: 。 2. The multi-label classification method based on label dependency measurement according to claim 1, characterized in that, The text embedding vector X is input into the Maxpooling layer and the linear classification layer. The forward cross-entropy loss is calculated, and the gradient of all parameters is updated using the stochastic gradient descent algorithm as follows: S601. Embedding text into a vector The input is fed into the Maxpooling layer to extract semantic information with more obvious features in the sentence length dimension. semantic information Output dimension is The formula is: ； S602, the step S601 Information is integrated, and an MLP network layer is selected for linear classification to obtain the classification tensor. Classification tensors Output dimension is The formula is: ； S603, Utilizing Classification Tensors With category tags Calculate the cross-entropy loss to obtain the overall loss, using the following formula: 。 3. The multi-label classification method based on label dependency measurement according to claim 1 or 2, characterized in that, For sets Embed each category label and traverse the multi-label correlation matrix. The loss function for measuring the pairwise label dependency is calculated as follows: S701. Transforming Classification Label Embedding into a Probability Distribution: Input Y into the softmax function, which operates on the last dimension. The formula is: ； S702, Based on the set of classification labels to which the current sample belongs. Based on the multi-label correlation matrix, filter out the set of label relationships. and Specifically: ； ； in, ;gather Indicates the classification label of the current sample. All tags that have a relationship; a set Indicates the classification label of the current sample. All tags that have an exclusionary relationship or no relationship; S703. Based on steps S701 and S702, calculate the label dependency metric loss function, as shown in the formula: ； in, Probability distribution derived from step S701 ; The distance based on KL divergence is given by the following formula: ； in, For probability distribution Based on probability distribution The KL divergence is used to measure the probability distribution. With probability distribution The degree of difference between them; Satisfying the three elements of distance, specifically refers to and The distance between them is nonnegative. and The distance between them satisfies symmetry and and The distance between them satisfies the triangle inequality.

4. A multi-label classification system based on label dependency measurement, characterized in that, The system includes, The definition module is used to build a pre-classification label system and define the relationships between labels; The acquisition module is used to input the natural language text model into the word embedding layer and the core layer to obtain text embedding vectors; A module is established to combine the multi-label correlation coefficient matrix to build a representation system of text embedding vectors and label embeddings, and to perform forward calculation of loss values, and then update all parameters once through the backpropagation algorithm; The update module is used to perform a secondary update on the label embedding vector parameters by building a label dependency metric loss function based on the multi-label correlation coefficient matrix. The training module is used during the training phase to take input text and classification labels, calculate cross-entropy loss and label dependency metric loss respectively, and update the parameters twice. The prediction module is used in the prediction phase to take text as input and output the multi-class label prediction results of the text. The definition module includes, The processing submodule is used to process the given text. Specifically, it sets the maximum character length in the text to L, pads any insufficient characters with "padding", and generates a text tensor T with dimension L using a dictionary index mapping table. Where R represents the real number space and L is a positive integer; Define a submodule to define the correlation matrix between multiple tags. At the same time, define the category tag set as ,common There are several categories, and the set of category tags to which the current text belongs is set to 1. , for set A subset of; in which, Indicates the first Categories ,and and It is a positive integer; The acquisition module sequentially inputs the text tensor T into the embedding layer and the core layer to obtain the text embedding vector X, whose dimension is... Where E represents the embedding dimension and is a positive integer; The establishment module includes, The initialization submodule is used to utilize the category label set. The number of categories is given, and the initial label embedding Y has the following dimensions. ; The calculation submodule is used to utilize the multi-label correlation coefficient matrix. The implicit label information is used to calculate the inner product attention between the text embedding and the label embedding, thereby incorporating the relation matrix information into the text embedding vector X for re-representation; The update submodule is used to input the re-represented text embedding vector into the Maxpooling layer and the linear classification layer, calculate the forward cross-entropy loss, and use the stochastic gradient descent algorithm to update the gradient of all parameters. Relevance matrix between multiple tags Specifically as follows: ； in, and Relevance includes linear correlation and inclusion; other situations include... and Exclusive or non-exclusive relationships; , , For a collection of category tags, , ; The specific working process of the calculation submodule is as follows: (a) Calculate the inner product of the text embedding vector X and the label embedding Y: ; in, ; ; ; ; (ii) Calculate the weight allocation coefficients in the attention mechanism, using the following formula: ； in, This is a multi-label correlation coefficient matrix; The dimension is The softmax function performs normalization on the embedding dimension; (III) Using weight allocation coefficients The integration effect re-represents the text embedding vector X as , The dimension is The formula is as follows: 。 5. The multi-label classification system based on label dependency measurement according to claim 4, characterized in that, The specific working process of the update submodule is as follows: (1) Embedding text into vectors The input is fed into the Maxpooling layer to extract semantic information with more obvious features in the sentence length dimension. semantic information Output dimension is The formula is: ； (2) Transfer the voice information from step S601 Information is integrated, and an MLP network layer is selected for linear classification to obtain the classification tensor. Classification tensors Output dimension is The formula is: ； (3) Using classification tensors With category tags Calculate the cross-entropy loss to obtain the overall loss, using the following formula: ； The calculation process of the label dependency metric loss function is as follows: ① Transforming category label embedding into a probability distribution: Input Y into the softmax function, which operates on the last dimension. The formula is: ； ② Based on the set of classification labels to which the current sample belongs Based on the multi-label correlation matrix, filter out the set of label relationships. and Specifically: ； ； in, ;gather Indicates the classification label of the current sample. All tags that have a relationship; set Indicates the classification label of the current sample. All tags that have an exclusionary relationship or no relationship; ③ Based on steps S701 and S702, calculate the label dependency metric loss function, as shown in the formula: ； in, Probability distribution derived from step S701 ; The distance based on KL divergence is given by the following formula: ； in, For probability distribution Based on probability distribution The KL divergence is used to measure the probability distribution. With probability distribution The degree of difference between them; Satisfying the three elements of distance, specifically refers to and The distance between them is nonnegative. and The distance between them satisfies symmetry and The distance between them satisfies the triangle inequality.

6. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the multi-label classification method based on label dependency metric as described in any one of claims 1 to 3.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the multi-label classification method based on label dependency metric as described in any one of claims 1 to 3.

Images