A multi-label text classification method based on positive and negative label learning and label correlation

Abstract

The application relates to a multi-label text classification method based on positive and negative label learning and label correlation and belongs to the field of multi-label classification. The method comprises the following steps: constructing a double-hidden-layer feedforward neural network model; initializing model components; reading feature and label information of samples in a training set to generate a feature matrix and a label matrix; randomly initializing a weight matrix and a bias parameter of the model; inputting the feature matrix into an input layer of the model, and calculating neuron outputs layer by layer; calculating the gradient of the weight matrix and the bias parameter between layers in the model by using a composite error function, and dynamically adjusting the gradient by using an Adam optimization algorithm to update the weight matrix and the bias parameter; terminating the training when the error change amplitude is lower than a threshold value or a preset iteration number is reached; and predicting a test set after the model converges to form a final multi-label classification result. The application breaks through the limitations of traditional text classification by using a deep learning technology, and realizes high-precision and high-efficiency classification of complex text data.

Description

Technical Field

[0001] This invention relates to a multi-label text classification method based on positive and negative label learning and label correlation, belonging to the field of multi-label classification technology. Background Technology

[0002] With the rapid development of information technology, massive amounts of text information are growing exponentially, covering diverse fields such as news, social media, medical records, legal documents, and scientific research literature. As an important extension of text classification tasks, multi-label text classification involves mapping text instances to multiple labels. It has been widely used to learn from real-world text objects with rich semantics. Compared to the "either / or" paradigm of traditional single-label classification, multi-label classification is more in line with the intertwined semantic characteristics of text in the real world. The current evolution path of multi-label classification technology mainly presents two branches: the traditional strategy system based on problem transformation and algorithm adaptation, and the modern method framework based on deep learning. In the traditional method system, the binary association method achieves label decoupling by constructing independent binary classifiers, while the label power set method attempts to transform the multi-label problem into a multi-class classification task. Although these methods are feasible in basic scenarios, their inherent defects are mainly reflected in two aspects: first, the linear modeling paradigm is difficult to characterize the complex nonlinear relationship between the text feature space and the label space; second, the potential correlation between labels is systematically ignored.

[0003] It is worth noting that in real-world scenarios, the relationship between text features and labels is often extremely complex and non-linear, which significantly limits the semantic capture accuracy and model generalization ability of traditional methods. Advanced methods, represented by deep neural networks, offer new insights into multi-label text classification technology thanks to their advantages in modeling multi-layer non-linear transformations. However, existing deep learning solutions still face the following problems: high model complexity leads to poor adaptability to small sample or unstructured text data scenarios; a lack of explicit modeling of label correlations makes it difficult to capture label co-occurrence dependencies; and classification thresholds rely on manual experience, failing to handle class imbalance issues.

[0004] Therefore, this invention provides a multi-label text classification method based on positive and negative label learning and label correlation. This invention is supported by the Yunnan Provincial Basic Research Program (Grant No. 202501AT070299). Summary of the Invention

[0005] The technical problem solved by this invention is that it provides a multi-label text classification method based on positive and negative label learning and label correlation to address the issues of insufficient label correlation modeling, imbalanced categories in multi-label data, and poor classification performance due to high model complexity in the field of text classification. This invention constructs a double-hidden-layer feedforward neural network model and implements a novel composite error function to fully mine high-order semantic features in text data, while establishing the correlation between text labels, thus achieving high-precision and high-efficiency classification of complex text data.

[0006] The technical solution of this invention is: a multi-label text classification method based on positive and negative label learning and label correlation, the method comprising:

[0007] Step 1: Data preprocessing and dividing the dataset into training and test sets;

[0008] Step 2: Construct a feedforward neural network model with two hidden layers, including an input layer, a first hidden layer, a second hidden layer, and an output layer;

[0009] Step 3: Input the training set and initialize the double hidden layer feedforward neural network model components; read the feature and label information of the samples in the training set and generate the feature matrix and label matrix; randomly initialize the weight matrix and bias parameters of the double hidden layer feedforward neural network model.

[0010] Step 4, Forward Propagation: Input the feature matrix into the input layer of the double hidden layer feedforward neural network model, and calculate the neuron output layer by layer;

[0011] Step 5, Backpropagation: The gradients of the weight matrix and bias parameters between each layer in the double hidden layer feedforward neural network model are calculated using the composite error function, and the gradients are dynamically adjusted using the Adam optimization algorithm to minimize the composite error function, thereby updating the weight matrix and bias parameters.

[0012] Step 6: Set an error threshold and a maximum number of iterations as training termination conditions. Terminate training when the error change is below the threshold or reaches the preset number of iterations; otherwise, repeat steps 4-5 until the convergence criterion is met.

[0013] Step 7: After the model converges, make predictions on the test set and binarize the prediction output using a fixed threshold to form the final multi-label classification result.

[0014] Further, step 1 includes:

[0015] Step 1.1, Data Preprocessing: Segment the text data, remove stop words, and extract stems; generate TF-IDF matrix and Word2Vec embedding matrix respectively, and concatenate them to form text feature vector;

[0016] Step 1.2: Obtain the text dataset. The text dataset is divided into a training set G and a test set T using the five-fold cross-validation method.

[0017] Further, step 2 includes:

[0018] Step 2.1: Construct a two-hidden-layer feedforward neural network model:

[0019] Input layer: Configured with neurons that match the dimension d of the text feature vector;

[0020] First hidden layer: N hidden neurons are set up for initial feature extraction and transmission;

[0021] The second hidden layer: configured with M hidden neurons, for deep feature extraction and abstract representation;

[0022] Output layer: Set up Q neurons, each neuron corresponds to a candidate class label, and outputs the predicted score;

[0023] Step 2.2, the weight matrix and bias parameters are defined as follows:

[0024] The input layer and the first hidden layer are connected via a fully connected method. The weight matrix from the input layer to the first hidden layer is: Z = [z qp (1≤q≤d, 1≤p≤N), The bias parameter is δ p (1≤p≤N),

[0025] The first hidden layer and the second hidden layer are connected using a fully connected method. The weight matrix from the first hidden layer to the second hidden layer is: V = [v ps (1≤p≤N, 1≤s≤M), The bias parameter is γ s (1≤s≤M),

[0026]

[0027] The second hidden layer and the output layer are fully connected. The weight matrix from the second hidden layer to the output layer is: W = [w sj (1≤s≤M, 1≤j≤Q), The bias parameter is θ j (1≤j≤Q),

[0028] Furthermore, in step 3, the initialization of the dual hidden layer feedforward neural network model components includes initializing the following components: classifier f(X) and nonlinear mapper g(·);

[0029] The classifier f(X) is used to extract sample features and output a classification prediction value C. k (x), C k (x) represents the predicted output of the k-th class label of sample x in the dataset;

[0030] The network architecture of the classifier f(X) is as follows: input layer, two hidden layers, and output layer; all layers are fully connected, and the weight matrix and bias parameters of each layer are (Z,V,W) and (δ), respectively. p ,γ s ,θ j );

[0031] The nonlinear mapper g(·) is located in the hidden layer and the output layer. The activation function of the first hidden layer is a linear activation function, and the activation functions of the second hidden layer and the output layer are "Sigmoid" functions.

[0032] Further, step 4 includes:

[0033] feature matrix The input is a double-hidden-layer feedforward neural network model. The neuron outputs are calculated layer by layer, and then weighted and processed by an activation function to obtain the final neuron output. Here, d is the dimension of the text feature vector, n is the number of samples, and x... n For the nth sample; repeat the above steps until the output layer, the specific steps are as follows:

[0034] Step 4.1, the predicted output for the p-th neuron in the first hidden layer is: b p =f(netb) p +δ p ), where δ p It is the bias parameter of the p-th hidden neuron, netb p The input signal of the p-th hidden neuron is: a q The input feature x i The q-th component, Z qp It is the weight matrix that connects the q-th input neuron to the p-th hidden neuron, and the activation function is set to a linear function;

[0035] Step 4.2: The predicted output of the s-th neuron in the second hidden layer is: h s =f(neth) s +γ s ), where γ sIt is the bias parameter of the s-th hidden neuron, neth s The input signal of the p-th hidden neuron is: v ps It is the weight matrix connecting the p-th hidden neuron and the s-th hidden neuron, and the activation function is set to the "sigmoid" function;

[0036] Step 4.3: The predicted output for the j-th neuron in the output layer is: C j =f(netC) j +θ j ), where θ j It is the bias parameter of the j-th output neuron, netC j The input signal of the j-th output neuron is: w sj It is the weight matrix connecting the s-th hidden neuron and the j-th hidden neuron, and the activation function is set to the "sigmoid" function.

[0037] Furthermore, step 5 specifically includes the following steps:

[0038] Step 5.1: Design a composite error function E i By introducing dynamic weighting coefficients to adaptively adjust the error term in the composite error function E i Embedded tag relevance constraints;

[0039] Step 5.2, based on the composite error function E i Calculation error;

[0040] Step 5.3: Calculate the weight matrix (Z, V, W) and the bias parameter (δ). p ,γ s ,θ j The gradient of ).

[0041] Step 5.4: Update the weight matrix and bias parameters: Use the Adam optimizer to update the weight matrix and bias parameters.

[0042] Further, in step 5.1, the composite error function is defined as a weighted sum of the following three parts, specifically including:

[0043] (1) The composite error function integrates positive and negative label learning terms, and introduces two dynamic weight coefficients λ1 and λ2 into the composite error function; the specific form of the positive and negative label learning terms is as follows:

[0044]

[0045] in, Indicates sample x iThe predicted output Y within the k-th label class i It is a sample x i A set of positive labels, and It is from sample x i The set of negative labels, exp is an exponential function used to amplify the penalty for extreme errors, and Q is the number of sample labels, which is also the number of neurons in the output layer of the double hidden layer feedforward neural network model;

[0046] (2) Embed a label correlation constraint term in the composite error function. The label correlation constraint term is constructed based on the label co-occurrence matrix. The specific form of the label correlation constraint term is as follows:

[0047]

[0048] in, It measures the similarity between different category labels, where β is the weighting coefficient of the label relevance term, and Ri is the weighting coefficient of the label relevance term. kl Let R be the label co-occurrence matrix. kl The calculation formula is:

[0049]

[0050] in, Let Y be the label matrix, Q be the number of sample labels, n be the number of samples, and Y be the number of samples. .,k This represents the label vector of the k-th column;

[0051] (3) The final composite error function is:

[0052]

[0053] Among them, E i It is the i-th error term, representing the network's error relative to sample x. i The error.

[0054] Further, step 5.2 includes:

[0055] (1) The error of the j-th neuron in the output layer is defined as:

[0056] (2) Considering that the activation function of the output layer is "sigmoid", then the sigmoid activation function of the output layer is f′(netC). j +θ j )=(1-C j (C) j ),get:

[0057]

[0058] (3) Similarly, the error of the s-th neuron in the second hidden layer is defined as:

[0059] (4) Since the activation function of the second hidden layer is also "sigmoid", the sigmoid activation function of the second hidden layer is f′(neth s +γ s )=(1-h s (h) s ), thus arriving at

[0060] (5) The error of the p-th hidden neuron in the first hidden layer is similarly defined as:

[0061] (6) Since the activation function of the first hidden layer is "linear", the activation function of the first hidden layer f′(netb) p +δ p ) = 1, therefore:

[0062] Further, in step 5.3, the gradient of the weight matrix (Z,V,W) is:

[0063]

[0064] Bias parameter (δ) p ,γ s ,θ j The gradient of ) is:

[0065] in, Let W, V, and Z be the gradients of the weight matrices. For the bias parameter θ j γ s δ p The gradient.

[0066] Further, step 5.4 includes:

[0067] Step 5.4.1: Update the moment estimates of the weight matrix and bias parameters:

[0068] First, we introduce the partial first-order moment estimate and perform an exponentially weighted average of the gradient of the weight matrix, denoted as m. t The calculation formula is:

[0069]

[0070] in, These are the partial first moments estimates of the weight matrices W, V, and Z;

[0071] Update the partial first-order moment estimates of the gradient of the bias parameters for each layer:

[0072]

[0073] Where t is the current iteration number, and β1 is a hyperparameter used to control the degree of influence of the gradient; For the bias parameter θ j γ s δ p The partial first moment estimate;

[0074] The learning rate of the weight matrix and bias parameters is dynamically adjusted based on the partial second-order moment estimation to avoid large parameter update step sizes or oscillations in the early stages of training. The partial second-order moment estimation is an exponentially weighted average of the squares of the gradients, denoted as v. t The calculation formula is:

[0075]

[0076] Where β2 is a hyperparameter used to control the influence of the squared gradient; where, These are the partial second moments estimates of the weight matrices W, V, and Z; For the bias parameter θ j γ s δ p The partial second moment estimate;

[0077] Because there is a partial first moment estimate m t And the partial second moment estimate v t The estimates are all started from zero, which may cause the underestimation to be biased towards 0 in the first few time steps, resulting in a bias in the estimate; therefore, bias correction is required, which involves the estimation of the scaled moving average to compensate for the bias introduced by zero initialization.

[0078] For m t and v t Perform bias correction, which involves calculating the first-order moment estimate of the bias correction between the weight matrix and the bias parameters. and second moment estimates

[0079] Calculate the corrected first moment estimate of the deviation between the weight matrix and the bias parameters for each layer:

[0080]

[0081] in, The first-order moment estimates for the bias correction of the weight matrices W, V, and Z. For the bias parameter θ jγ s δ p The first-order moment estimate of the deviation correction; Let these be the hyperparameters after the t-th iteration;

[0082] Calculate the corrected second-order primitive moment estimates of the deviation between the weight matrix and the bias parameters for each layer:

[0083]

[0084] in, The bias-corrected second moment estimates for the weight matrices W, V, and Z. For the bias parameter θ j γ s δ p The bias-corrected second moment estimate; Let these be the hyperparameters after the t-th iteration;

[0085] Step 5.4.2: Update the weight matrix and bias parameters:

[0086]

[0087] Where α is the learning rate, and ∈ is set to 10. -8 This is used to ensure that no division by zero error is encountered.

[0088] Furthermore, in step 6, the training termination condition is:

[0089] The preset range for the number of iterations is: 50 ≤ epochs ≤ 100;

[0090] Based on the composite error function E i The error is calculated using the mean square error form, where the amount of error is calculated as follows:

[0091]

[0092] When the error change is less than Training may terminate when a preset number of iterations is reached;

[0093] If the above conditions are not met, repeat steps 4-5 until convergence.

[0094] Furthermore, step 7 includes:

[0095] To make predictions on the test set T, the feature matrix X of the test set T is input into the classifier f(X) to obtain the classification prediction value C. k (x), the label is determined based on the threshold, and the threshold selector is defined as:

[0096] h(x) = {k∈Y:C} k(x)>t(x)}

[0097] If the fixed threshold t(x) is set to 0.5, and C k If (x)>0.5, then sample x is determined to belong to the k-th label class.

[0098] The beneficial effects of this invention are:

[0099] 1. The multi-label text classification method proposed in this invention, based on positive and negative label learning and label correlation, significantly improves the accuracy and efficiency of multi-label text classification;

[0100] 2. The method of the present invention effectively solves the problems of weak nonlinear modeling ability, sensitivity to class imbalance and insufficient utilization of label relevance in multi-label classification methods by constructing a double hidden layer feedforward neural network model and combining a composite error function and an adaptive optimization strategy.

[0101] 3. Compared with existing advanced multi-label text classification methods, this invention demonstrates superior stability and accuracy on various existing text datasets. Experimental results show that the key indicators such as average accuracy, 1-error rate, and ranking loss of this invention are significantly improved compared with traditional methods, fully verifying the superior performance of this model in complex application scenarios. Attached Figure Description

[0102] Figure 1 This is a flowchart of the training process in this invention;

[0103] Figure 2 This is a diagram of the dual-hidden-layer feedforward neural network model architecture in this invention. Detailed Implementation

[0104] Example 1: As Figures 1-2 As shown, a multi-label text classification method based on positive and negative label learning and label correlation is presented, the method comprising:

[0105] Step 1: Data preprocessing and dividing the dataset into training and test sets;

[0106] Further, step 1 includes:

[0107] Step 1.1, Data Preprocessing: Segment the text data, remove stop words, and extract stems; generate TF-IDF matrix and Word2Vec embedding matrix respectively, and concatenate them to form text feature vector;

[0108] Step 1.2: Obtain the text dataset. The text dataset is divided into a training set G and a test set T using the five-fold cross-validation method.

[0109] Step 2: Construct a feedforward neural network model with two hidden layers, including an input layer, a first hidden layer, a second hidden layer, and an output layer;

[0110] Further, step 2 includes:

[0111] Step 2.1: Construct a two-hidden-layer feedforward neural network model:

[0112] Input layer: Configured with neurons that match the dimension d of the text feature vector;

[0113] First hidden layer: N hidden neurons are set up for initial feature extraction and transmission;

[0114] The second hidden layer: configured with M hidden neurons, for deep feature extraction and abstract representation;

[0115] Output layer: Set up Q neurons, each neuron corresponds to a candidate class label, and outputs the predicted score;

[0116] Step 2.2, the weight matrix and bias parameters are defined as follows:

[0117] The input layer and the first hidden layer are connected via a fully connected method. The weight matrix from the input layer to the first hidden layer is: Z = [z qp (1≤q≤d, 1≤p≤N), The bias parameter is δ p (1≤p≤N),

[0118] The first hidden layer and the second hidden layer are connected using a fully connected method. The weight matrix from the first hidden layer to the second hidden layer is: V = [v ps (1≤p≤N, 1≤s≤M), The bias parameter is γ s (1≤s≤M),

[0119] The second hidden layer and the output layer are fully connected. The weight matrix from the second hidden layer to the output layer is: W = [w sj (1≤s≤M, 1≤j≤Q), The bias parameter is θ j (1≤j≤Q),

[0120] Step 3: Input the training set and initialize the double hidden layer feedforward neural network model components; read the feature and label information of the samples in the training set and generate the feature matrix and label matrix; randomly initialize the weight matrix and bias parameters of the double hidden layer feedforward neural network model.

[0121] Feature matrix Tag matrix Q is the number of sample labels, d is the dimension of the text feature vector, n is the number of samples, and x is the number of samples. n For the nth sample;

[0122] Furthermore, in step 3, the initialization of the dual hidden layer feedforward neural network model components includes initializing the following components: classifier f(X) and nonlinear mapper g(·);

[0123] The classifier f(X) is used to extract sample features and output a classification prediction value C. k (x), C k (x) represents the predicted output of the k-th class label of sample x in the dataset;

[0124] The network architecture of the classifier f(X) (e.g.) Figure 2 The structure (shown) consists of: an input layer (dimension matched to the feature vector), two hidden layers (containing N and M neurons respectively), and an output layer (containing Q label neurons); all layers are fully connected, and the weight matrices and bias parameters between layers are (Z, V, W) and (δ), respectively. p ,γ s ,θ j );

[0125] The nonlinear mapper g(·) is located in the hidden layer and the output layer. The activation function of the first hidden layer is a linear activation function, and the activation functions of the second hidden layer and the output layer are "Sigmoid" functions.

[0126] Step 4, Forward Propagation: Input the feature matrix into the input layer of the double hidden layer feedforward neural network model, and calculate the neuron output layer by layer;

[0127] Further, step 4 includes:

[0128] feature matrix The input is a double-hidden-layer feedforward neural network model. The neuron outputs are calculated layer by layer, and then weighted and processed by an activation function to obtain the final neuron output. Here, d is the dimension of the text feature vector, n is the number of samples, and x... n For the nth sample; repeat the above steps until the output layer, the specific steps are as follows:

[0129] Step 4.1, the predicted output for the p-th neuron in the first hidden layer is: b p =f(netb) p +δ p ), where δ p It is the bias parameter of the p-th hidden neuron, netb p The input signal of the p-th hidden neuron is: a qThe input feature x i The q-th component, Z qp It is the weight matrix that connects the q-th input neuron to the p-th hidden neuron, and the activation function is set to a linear function;

[0130] Step 4.2: The predicted output of the s-th neuron in the second hidden layer is: h s =f(neth) s +γ s ), where γ s It is the bias parameter of the s-th hidden neuron, neth s The input signal of the p-th hidden neuron is: v ps It is the weight matrix connecting the p-th hidden neuron and the s-th hidden neuron, and the activation function is set to the "sigmoid" function;

[0131] Step 4.3: The predicted output for the j-th neuron in the output layer is: C j =f(netC) j +θ j ), where θ j It is the bias parameter of the j-th output neuron, netC j The input signal of the j-th output neuron is: w sj It is the weight matrix connecting the s-th hidden neuron and the j-th hidden neuron, and the activation function is set to the "sigmoid" function.

[0132] Step 5, Backpropagation: A novel composite error function is used to calculate the gradients of the weight matrix and bias parameters between layers in the double hidden layer feedforward neural network model. The Adam optimization algorithm is then used to dynamically adjust the gradients, aiming to minimize the composite error function, and to update the weight matrix and bias parameters.

[0133] Furthermore, step 5 specifically includes the following steps:

[0134] Step 5.1: To measure the difference between the predicted label and the true label, a composite error function E is designed. i To address the class imbalance problem, dynamic weighting coefficients are introduced to adaptively adjust the error term. Furthermore, to capture higher-order correlations between labels, a composite error function E... i The network embeds a label relevance constraint term, which is constructed based on the label co-occurrence matrix and is used to force the network to learn the dependencies between labels during training.

[0135] Step 5.2, based on the composite error function E i Calculation error;

[0136] Step 5.3: Calculate the weight matrix (Z, V, W) and the bias parameter (δ). p ,γ s ,θ j The gradient of ).

[0137] Step 5.4: Update the weight matrix and bias parameters: Use the Adam optimizer to update the weight matrix and bias parameters.

[0138] Further, in step 5.1, the composite error function is defined as a weighted sum of the following three parts, specifically including:

[0139] (1) The composite error function integrates positive and negative label learning terms. To alleviate the imbalance between positive and negative labels in multi-label datasets, two dynamic weight coefficients, λ1 and λ2, are introduced into the composite error function to balance the outputs of the positive and negative classes in the two error terms, maintaining consistency in their error outputs. The specific form of the positive and negative label learning terms is as follows:

[0140]

[0141] in, Indicates sample x i The predicted output Y within the k-th label class i It is a sample x i A set of positive labels, and It is from sample x i The set of negative labels, exp is an exponential function used to amplify the penalty for extreme errors, and Q is the number of sample labels, which is also the number of neurons in the output layer of the double hidden layer feedforward neural network model;

[0142] (2) To capture the higher-order correlation characteristics between labels, a label correlation constraint term is embedded in the composite error function. This label correlation constraint term is constructed based on the label co-occurrence matrix and is used to force the network to learn the dependencies between labels during training. The specific form of the label correlation constraint term is as follows:

[0143]

[0144] in, It measures the similarity between different category labels, where β is the weighting coefficient of the label relevance term, and Ri is the weighting coefficient of the label relevance term. kl Let R be the label co-occurrence matrix. kl The calculation formula is:

[0145]

[0146] in, Let Y be the label matrix, Q be the number of sample labels, n be the number of samples, and Y be the number of samples..,k This represents the label vector of the k-th column;

[0147] (3) The final composite error function is:

[0148]

[0149] Among them, E i It is the i-th error term, representing the network's error relative to sample x. i The error.

[0150] This composite loss function quantifies the difference between the predicted label matrix and the initial label matrix. Specifically, for a positive label of 1, if the model's predicted output is less than 0.5, it means that the model has incorrectly classified it as a negative label. In this case, the exponential function in the positive label error term significantly amplifies the error value. This error value is then passed back to each layer of the model through the backpropagation algorithm to guide the update of the model parameters, minimizing the composite error function to ensure the consistency between the predicted label matrix and the initial label matrix.

[0151] Further, step 5.2 includes:

[0152] (1) The error of the j-th neuron in the output layer is defined as:

[0153] (2) Considering that the activation function of the output layer is "sigmoid", then the sigmoid activation function of the output layer is f′(netC). j +θ j )=(1-C j (C) j ),get:

[0154]

[0155] (3) Similarly, the error of the s-th neuron in the second hidden layer is defined as:

[0156] (4) Since the activation function of the second hidden layer is also "sigmoid", the sigmoid activation function of the second hidden layer is f′(neth s +γ s )=(1-h s (h) s Substituting this equation into equation (3), we obtain...

[0157] (5) The error of the p-th hidden neuron in the first hidden layer is similarly defined as:

[0158] (6) Since the activation function of the first hidden layer is "linear", the activation function of the first hidden layer f′(netb) p +δ p ) = 1, therefore:

[0159] Further, in step 5.3, the gradient of the weight matrix (Z,V,W) is:

[0160]

[0161] Bias parameter (δ) p ,γ s ,θ j The gradient of ) is:

[0162] in, Let W, V, and Z be the gradients of the weight matrices. For the bias parameter θ j γ s δ p The gradient.

[0163] When optimizing the objective function, the model is prone to getting trapped in local minima and cannot guarantee convergence to the global optimum. In addition, the gradient descent method is extremely sensitive to the configuration of the learning rate. If the learning rate is set too high, the algorithm may linger around the optimal solution and find it difficult to achieve stable convergence. Therefore, in this invention, the Adam optimizer is used to update the parameters.

[0164] Further, step 5.4 includes:

[0165] Step 5.4.1: Update the moment estimates of the weight matrix and bias parameters:

[0166] First, we introduce the partial first-order moment estimate and perform an exponentially weighted average of the gradient of the weight matrix, denoted as m. t The calculation formula is:

[0167]

[0168] in, These are the partial first moments estimates of the weight matrices W, V, and Z;

[0169] Update the partial first-order moment estimates of the gradient of the bias parameters for each layer:

[0170]

[0171] Where t is the current iteration number, and β1 is a hyperparameter (set to 0.9) used to control the degree of influence of the gradient; For the bias parameter θ j γs δ p The partial first moment estimate;

[0172] The learning rate of the weight matrix and bias parameters is dynamically adjusted based on the partial second-order moment estimation to avoid large parameter update step sizes or oscillations in the early stages of training. The partial second-order moment estimation is an exponentially weighted average of the squares of the gradients, denoted as v. t The calculation formula is:

[0173]

[0174] Where β2 is a hyperparameter (set to 0.999) used to control the influence of the squared gradient; where, These are the partial second moments estimates of the weight matrices W, V, and Z; For the bias parameter θ j γ s δ p The partial second moment estimate;

[0175] Due to the partial first moment m t And partial second moment v t The estimates all start from zero, and β1 and β2 are both close to 1, m t and v t The values ​​are biased towards 0 in the first few time steps, which may result in a partial first-order moment estimate m in the first few time steps. t And the partial second moment estimate v t There is a bias; therefore, bias correction is required, which involves estimating the scaled moving average to compensate for the bias introduced by zero initialization.

[0176] For m t and v t Perform bias correction, which involves calculating the first-order moment estimate of the bias correction between the weight matrix and the bias parameters. and second moment estimates

[0177] Calculate the corrected first moment estimate of the deviation between the weight matrix and the bias parameters for each layer:

[0178]

[0179] in, The first-order moment estimates for the bias correction of the weight matrices W, V, and Z. For the bias parameter θ j γ s δ p The first-order moment estimate of the deviation correction; Let these be the hyperparameters after the t-th iteration;

[0180] Calculate the corrected second-order primitive moment estimates of the deviation between the weight matrix and the bias parameters for each layer:

[0181]

[0182] in, The bias-corrected second moment estimates for the weight matrices W, V, and Z. For the bias parameter θ j γ s δ p The bias-corrected second moment estimate; Let these be the hyperparameters after the t-th iteration;

[0183] Step 5.4.2: Update the weight matrix and bias parameters:

[0184]

[0185] Where α is the learning rate, set to 0.01, and ∈ is set to 10. -8 This is used to ensure that no division by zero error is encountered.

[0186] Step 6: Set an error threshold and a maximum number of iterations as training termination conditions. Terminate training when the error change is below the threshold or reaches the preset number of iterations; otherwise, repeat steps 4-5 until the convergence criterion is met.

[0187] Furthermore, in step 6, the training termination condition is:

[0188] The preset range for the number of iterations is: 50 ≤ epochs ≤ 100;

[0189] Based on the composite error function E i The error is calculated using the mean square error form, where the amount of error is calculated as follows:

[0190]

[0191] When the error change is less than Training may terminate when a preset number of iterations is reached;

[0192] If the above conditions are not met, repeat steps 4-5 until convergence.

[0193] Step 7: After the model converges, make predictions on the test set and binarize the prediction output using a fixed threshold to form the final multi-label classification result.

[0194] Furthermore, step 7 includes:

[0195] To make predictions on the test set T, the feature matrix X of the test set T is input into the classifier f(X) to obtain the classification prediction value C. k (x), the label is determined based on the threshold, and the threshold selector is defined as:

[0196] h(x) = {k∈Y:C} k (x)>t(x)}

[0197] If the fixed threshold t(x) is set to 0.5, and C k If (x) > 0.5, then sample x is determined to belong to the k-th label class.

[0198] This invention proposes a multi-label text classification method based on positive and negative label learning and label correlation. It aims to overcome the limitations of traditional text classification through deep learning technology, achieving high-precision and high-efficiency classification of complex text data. Its innovation lies in constructing a composite error function. This composite error function possesses unique properties: First, it employs a dual-constraint strategy, strictly defining the output thresholds for positive and negative labels, abandoning empirical or inapplicable threshold setting methods, effectively improving the prediction accuracy of individual labels. Second, it introduces two weighting factors, assigning different importance to the error terms of positive and negative labels respectively, achieving error term balance and enhancing prediction performance on imbalanced multi-label datasets. Third, it incorporates label correlation constraints, helping the network to discover co-occurrence relationships between label categories. In terms of network training, the Adam algorithm is used, dynamically adjusting the learning rate based on historical parameter gradient values, and combined with backpropagation to update parameters, effectively improving multi-label classification performance.

[0199] To verify the effectiveness of the method of this invention, the following experiments were conducted: This invention uses 1-error rate, ranking loss, and average precision to measure the performance of multi-label classification methods. A higher average precision value indicates better model performance. The 1-error rate is the opposite of the ranking loss; a lower value indicates better model performance. Methods used in the comparative experiments included lazy learning-based methods (ML-KNN), neural network-based methods (LSIC-PS, BP-MLL, CLIF), and methods based on high-order label correlation (HOMI).

[0200] On five multi-label text datasets—Computers, Yelp, Education, Reference, and Recreation—a comparative experiment was conducted using five-fold cross-validation to evaluate MLNN-SOT (the implementation method) against five benchmark methods. Average precision, 1-error rate, and ranking loss were used as evaluation metrics. The experimental results are shown in Tables 1-3. The MLNN-SOT method, based on positive and negative label learning and label relevance, demonstrated significant advantages on all five datasets. It achieved the highest average precision (0.895 on Yelp), the lowest 1-error rate (0.145 on Yelp), and the best ranking loss (only 0.061 on Reference), comprehensively surpassing benchmark methods such as ML-KNN and BP-MLL. MLNN-SOT alleviates the class imbalance problem by adjusting the positive and negative label error weights (λ1, λ2), enhances co-occurrence dependency modeling by combining label cosine similarity constraints, and utilizes a four-layer deep network to extract nonlinear features from high-dimensional data, enabling accurate classification in complex scenes. In contrast, traditional methods (such as ML-KNN) are limited by the linear assumption, high-order labeling methods (such as HOMI) lack deep feature learning, and existing deep models (such as CLIF, BP-MLL, and LSIC-PS) neglect error adjustment for different categories. The stability of this method on multiple indicators and multiple datasets verifies the effectiveness and generalization ability of its innovative design.

[0201] Table 1 shows the experimental results of the six models in terms of average accuracy.

[0202] Computers 0.660 0.684 0.679 0.659 0.608 0.701 Education 0.524 0.579 0.595 0.560 0.529 0.626 Yelp 0.852 0.853 0.823 0.888 0.743 0.895 Reference 0.626 0.685 0.642 0.678 0.569 0.719 Recreation 0.499 0.588 0.592 0.607 0.593 0.654

[0203] Table 2 shows the experimental results of the six models in terms of 1-error rate.

[0204] Computers 0.413 0.398 0.382 0.391 0.471 0.374 Education 0.629 0.549 0.519 0.513 0.648 0.484 Yelp 0.210 0.215 0.208 0.155 0.341 0.145 Reference 0.473 0.396 0.440 0.380 0.526 0.366 Recreation 0.645 0.532 0.513 0.472 0.523 0.446

[0205] Table 3 shows the experimental results of the six models in terms of ranking loss.

[0206] Computers 0.081 0.091 0.096 0.143 0.098 0.068 Education 0.099 0.109 0.093 0.196 0.097 0.083 Yelp 0.145 0.139 0.107 0.111 0.173 0.103 Reference 0.086 0.092 0.095 0.072 0.105 0.061 Recreation 0.175 0.158 0.149 0.187 0.141 0.116

[0207] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A multi-label text classification method based on positive and negative label learning and label correlation, characterized in that: The method includes: Step 1: Data preprocessing and dividing the dataset into training and test sets; Step 2: Construct a feedforward neural network model with two hidden layers, including an input layer, a first hidden layer, a second hidden layer, and an output layer; Step 3: Input the training set and initialize the double hidden layer feedforward neural network model components; read the feature and label information of the samples in the training set and generate the feature matrix and label matrix; randomly initialize the weight matrix and bias parameters of the double hidden layer feedforward neural network model. Step 4, Forward Propagation: Input the feature matrix into the input layer of the double hidden layer feedforward neural network model, and calculate the neuron output layer by layer; Step 5, Backpropagation: The gradients of the weight matrix and bias parameters between each layer in the double hidden layer feedforward neural network model are calculated using the composite error function, and the gradients are dynamically adjusted using the Adam optimization algorithm to minimize the composite error function, thereby updating the weight matrix and bias parameters. Step 6: Set an error threshold and a maximum number of iterations as training termination conditions. When the error change is lower than the threshold or the preset number of iterations is reached, training is terminated. Otherwise, repeat steps 4-5 until the convergence criterion is met. Step 7: After the model converges, make predictions on the test set and binarize the prediction output using a fixed threshold to form the final multi-label classification result. The specific steps of step 5 include: Step 5.1: Design a composite error function By introducing dynamic weighting coefficients to adaptively adjust the error term in the composite error function, Embedded tag relevance constraints; Step 5.2: Based on the composite error function Calculation error; Step 5.3: Calculate the weight matrix With bias parameters The gradient; Step 5.4: Update the weight matrix and bias parameters: Use the Adam optimizer to update the weight matrix and bias parameters; In step 5.1, the composite error function is defined as a weighted sum of the following three parts, specifically including: (1) The composite error function integrates positive and negative label learning terms and introduces two dynamic weight coefficients into the composite error function. and The specific form of the positive and negative label learning items is as follows: ； in, Indicates sample The predicted output within the k-th label class, It is a sample-only A set of positive labels, and It is a sample The set of negative labels, exp is an exponential function used to amplify the penalty for extreme errors, and Q is the number of sample labels, which is also the number of neurons in the output layer of the double hidden layer feedforward neural network model; (2) Embed a label correlation constraint term in the composite error function. The label correlation constraint term is constructed based on the label co-occurrence matrix. The specific form of the label correlation constraint term is as follows: ； in, It measures the similarity between different category labels. The weighting coefficients for the label relevance items. This is the label co-occurrence matrix. The calculation formula is: ； ； in, Here is the label matrix, and n is the number of samples. This represents the label vector of the k-th column; (3) The final composite error function is: ； in, It is the i-th error term, representing the network's error relative to the sample. The error.

2. The multi-label text classification method based on positive and negative label learning and label correlation according to claim 1, characterized in that: Step 1 includes: Step 1.1, Data Preprocessing: Segment the text data, remove stop words, and extract stems; generate TF-IDF matrix and Word2Vec embedding matrix respectively, and concatenate them to form text feature vector; Step 1.2: Obtain the text dataset. The text dataset is divided into a training set G and a test set T using the five-fold cross-validation method.

3. The multi-label text classification method based on positive and negative label learning and label correlation according to claim 1, characterized in that: Step 2 includes: Step 2.1: Construct a two-hidden-layer feedforward neural network model: Input layer: Configured with neurons that match the dimension d of the text feature vector; First hidden layer: N hidden neurons are set up for initial feature extraction and transmission; The second hidden layer: configured with M hidden neurons, for deep feature extraction and abstract representation; Output layer: Set up Q neurons, each neuron corresponds to a candidate class label, and outputs the predicted score; Step 2.2, the weight matrix and bias parameters are defined as follows: The input layer and the first hidden layer are connected using a fully connected method. The weight matrix from the input layer to the first hidden layer is as follows: , , The bias parameter is , ; It is the weight matrix that connects the q-th input neuron to the p-th hidden neuron; The first hidden layer and the second hidden layer are connected using a fully connected method. The weight matrix from the first hidden layer to the second hidden layer is as follows: , , The bias parameter is , ; It is the weight matrix connecting the p-th hidden neuron and the s-th hidden neuron; The second hidden layer and the output layer are connected using a fully connected method. The weight matrix from the second hidden layer to the output layer is as follows: , , The bias parameter is , , It is the weight matrix connecting the s-th hidden neuron and the j-th hidden neuron.

4. The multi-label text classification method based on positive and negative label learning and label correlation according to claim 1, characterized in that: In step 3, initializing the dual-hidden-layer feedforward neural network model component includes initializing the following components: classifier and nonlinear mapper ; The classifier Used to extract sample features and output classification prediction values. , This represents the predicted label of the k-th class for sample x in the dataset; The classifier The network architecture consists of an input layer, two hidden layers, and an output layer; all layers are fully connected, and the weight matrices and bias parameters of each layer are as follows: and ; The nonlinear mapper Located in the hidden layer and the output layer, the activation function of the first hidden layer is a linear activation function, while the activation functions of the second hidden layer and the output layer are "sigmoid" functions.

5. The multi-label text classification method based on positive and negative label learning and label correlation according to claim 1, characterized in that: Step 4 includes: feature matrix The input is a two-hidden-layer feedforward neural network model. The neuron outputs are calculated layer by layer, and then weighted and processed by an activation function to obtain the final neuron output. Here, d is the dimension of the text feature vector, and n is the number of samples. For the nth sample; repeat the above steps until the output layer, the specific steps are as follows: Step 4.1: The predicted output for the p-th neuron in the first hidden layer is: ,in, These are the bias parameters of the p-th hidden neuron. The input signal of the p-th hidden neuron is: , Input features The q-th component, It is the weight matrix that connects the q-th input neuron to the p-th hidden neuron, and the activation function is set to a linear function; Step 4.2: The predicted output of the s-th neuron in the second hidden layer is: ,in, It is the bias parameter of the s-th hidden neuron. The input signal of the s-th hidden neuron is: , It is the weight matrix connecting the p-th hidden neuron and the s-th hidden neuron, with the activation function set to the "sigmoid" function, and N is the number of hidden neurons in the first hidden layer; Step 4.3: The predicted output for the j-th neuron in the output layer is: ,in, These are the bias parameters of the j-th output neuron. The input signal of the j-th output neuron is: , It is the weight matrix connecting the s-th hidden neuron and the j-th hidden neuron, with the activation function set to the "sigmoid" function, and M is the number of hidden neurons in the second hidden layer.

6. The multi-label text classification method based on positive and negative label learning and label correlation according to claim 5, characterized in that: Step 5.2 includes: (1) The error of the j-th neuron in the output layer is defined as: ; (2) Considering that the activation function of the output layer is "sigmoid", then the sigmoid activation function of the output layer... ,get: ； (3) Similarly, the error of the s-th neuron in the second hidden layer is defined as: ; (4) Since the activation function of the second hidden layer is also "sigmoid", then the sigmoid activation function of the second hidden layer... , and thus ; (5) The error of the p-th hidden neuron in the first hidden layer is similarly defined as: ; (6) Since the activation function of the first hidden layer is "linear", the activation function of the first hidden layer... Therefore, we can conclude that: .

7. The multi-label text classification method based on positive and negative label learning and label correlation according to claim 6, characterized in that: In step 5.3, the weight matrix The gradient is: ； ； ； bias parameters The gradient is: ; in, , , Weight matrix , , gradient, , , For bias parameters , , The gradient.

8. A multi-label text classification method based on positive and negative label learning and label correlation according to claim 5, characterized in that: Step 5.4 includes: Step 5.4.1: Update the moment estimates of the weight matrix and bias parameters: First, the partial first-order moment estimate is introduced to perform an exponentially weighted average of the gradient of the weight matrix, denoted as . The calculation formula is: ； in, , , Weight matrix , , The partial first moment estimate; Update the partial first-order moment estimates of the gradient of the bias parameters for each layer: ； Where t is the current iteration number, This is a hyperparameter used to control the degree of influence of the gradient; , , For bias parameters , , The partial first moment estimate; The learning rate of the weight matrix and bias parameters is dynamically adjusted based on the partial second-order moment estimation to avoid large parameter update step sizes or oscillations in the early stages of training. The partial second-order moment estimation is an exponentially weighted average of the squared gradients, denoted as... The calculation formula is: ； ； in, Here, is a hyperparameter used to control the influence of the squared gradient; where, , , Weight matrix , , The partial second moment estimate; , , For bias parameters , , The partial second moment estimate; Because of the partial first moment estimate And partial second moment estimates The estimates are all started from zero, which may cause the underestimation to be biased towards 0 in the first few time steps, resulting in a bias in the estimate; therefore, bias correction is required, which involves the estimation of the scaled moving average to compensate for the bias introduced by zero initialization. right and Perform bias correction, which involves calculating the first-order moment estimate of the bias correction between the weight matrix and the bias parameters. and second moment estimates : Calculate the corrected first moment estimate of the deviation between the weight matrix and the bias parameters for each layer: ； in, , , Weight matrix , , The bias-corrected first-order moment estimate, , , For bias parameters , , The first-order moment estimate of the deviation correction; Let these be the hyperparameters after the t-th iteration; Calculate the corrected second-order primitive moment estimates of the deviation between the weight matrix and the bias parameters for each layer: ； in, , , Weight matrix , , The bias-corrected second moment estimate, , , For bias parameters , , The bias-corrected second moment estimate; Let these be the hyperparameters after the t-th iteration; Step 5.4.2: Update the weight matrix and bias parameters: ； Where α is the learning rate. Set as This is used to ensure that no division by zero error is encountered.

Images