Multi-label text classification method and device based on neural network

Abstract

The invention discloses a multi-label text classification method and device based on a neural network. The method comprises the following steps: collecting multi-label text data to form a training sample; performing word embedding and label embedding on the training sample to obtain a word vector and a label vector respectively; the method comprises the following steps of: establishing an initial CNN (Convolutional Neural Network)-BiLSTM (BiLSTM)-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model comprises a CNN (Convolutional Neural Network) layer, a BiLSTM (BiLSTM) layer and an ATTENTION layer; the method comprises the following steps: training a CNN (convolutional neural network)-BiLSTM-ATTENTION (BiLSTM-ATTENTION) neural network model until the CNN-BiLSTM-ATTENTION neural network model is converged; receiving text data to be classified, inputting the text data to the converged CNN-BiLSTM-ATTENTION neural network model, and outputting a classification result; the method is suitable for a scene of multi-dimensional label vector classification.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a multi-label text classification method and device based on a neural network. Background Art

[0002] Text classification is an important branch of Natural Language Processing (NLP). In traditional text classification problems, each sample has only one category label, which is called single-label text classification. With the increasing richness of text information and the increasing degree of classification refinement, a sample is associated with multiple category labels, and there may be certain dependencies between category labels. This is called multi-label text classification. In multi-label classification, each text instance can be assigned multiple labels at the same time. These labels may be independent of each other, or they may overlap or have a hierarchical structure. For example, a text may belong to both the "technology" category and the "security" category, which can be further subdivided into "network security" or "software security".

[0003] Existing multi-label classification mainly includes: label binarization method, which treats each label as an independent binary classification problem and trains a separate classifier for each label; label combination method, which takes the prediction result of one label as the input of the next label prediction to form a sequential relationship between labels; neural network-based models, mainly deep learning methods are widely used in multi-label classification problems. These models handle multiple label tasks by sharing parameters. For example, patent document CN110490221B provides a multi-label classification method, including: establishing a label library, wherein the label library stores different labels corresponding to different objects and the label order; obtaining image data of a picture; constructing a classification model; training the classification model, using the image data of a known picture and the labels corresponding to the objects contained in the image and the label sequence formed according to the label order in the label library as a data set, a set number of image data of positive samples containing the label and negative samples not containing the label for each label as a training set, and the label sequence of the positive sample and the negative sample as a verification set; inputting the image data of the picture into the trained classification model, obtaining the label corresponding to each object in the picture, thereby obtaining the label sequence of the picture, wherein the label sequence is a binary sequence, and according to the order of the labels in the label library, 1 represents an object corresponding to the label in the picture, and 0 represents an object without the label in the picture. Although the prior art has made significant progress in multi-label classification, the following problems still exist: 1. Most methods regard labels as independent classification problems, ignoring the potential dependencies between labels; 2. When the number of labels is large, the labels in the training data are often unevenly distributed, making it difficult for the model to learn effective feature representations on sparse data. Summary of the invention

[0004] The present invention provides a multi-label text classification method and device based on a neural network, which is suitable for multi-level multi-label text classification and can learn effective feature representation in data to improve the accuracy of multi-label text classification.

[0005] A multi-label text classification method based on neural network, comprising:

[0006] Collect multi-label text data to form training samples;

[0007] Performing word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively;

[0008] Establishing an initial CNN-BiLSTM-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer, and an ATTENTION layer;

[0009] The CNN-BiLSTM-ATTENTION neural network model is trained: the word vector is input into the CNN layer for local feature extraction, the word vector and the extracted local feature are input into the BiLSTM layer for processing to obtain mixed features, the mixed features and the label vector are input into the ATTENTION layer for processing, and the classification probability is output; the loss value is calculated according to the classification probability and the parameters of the CNN-BiLSTM-ATTENTION neural network model are updated, and the training steps are repeated until the CNN-BiLSTM-ATTENTION neural network model converges;

[0010] Receive the text data to be classified and input it into the converged CNN-BiLSTM-ATTENTION neural network model, and output the classification result.

[0011] Further, word embedding and label embedding are performed on the training samples to obtain word vectors and label vectors respectively, including:

[0012] The BERT model is used to encode and segment the text in the training sample to obtain a word vector;

[0013] Determine whether the label in the training sample has Chinese semantics. If the label does not have Chinese semantics, use a random generator to generate a random vector according to the dimension of word embedding, and normalize the random vector to obtain a label vector;

[0014] If the label has Chinese semantics, a data set including text and corresponding labels is established according to the training sample, a mapping relationship is established between the text and the label in the data set, a subset including the target label is generated, and the TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label.

[0015] Furthermore, the TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label, including:

[0016] For each word in the text in the subset, the word frequency and the inverse document frequency are calculated, the weight of each word is calculated according to the word frequency and the inverse document frequency, the weights are sorted, and a preset number of important words that are most relevant to the target tag are selected according to the sorted weights;

[0017] Embed the selected important words to obtain multiple important word embedding vectors;

[0018] The label vector of the target label is obtained by weighted averaging the multiple important word embedding vectors.

[0019] Furthermore, the word vector is input into the CNN layer for local feature extraction, including:

[0020] Input the word vector into the convolution layer to capture local patterns and generate a feature map;

[0021] Performing nonlinear activation processing on the feature map through an activation layer;

[0022] The feature map processed by the activation layer is sent to the pooling layer for downsampling. Based on the maximum pooling strategy, the feature map is divided into multiple domains, and the maximum value outputted by each domain is used as the local feature.

[0023] Furthermore, the BiLSTM layer includes a forward LSTM unit, a backward LSTM unit, a concatenation unit and a Highway layer; the local features include local context information;

[0024] The word vector and the extracted local features are input into the BiLSTM layer for processing to obtain mixed features, including:

[0025] Input the word vector and the local feature into the forward LSTM unit and the backward LSTM unit;

[0026] The forward LSTM unit captures the word vector and the local features from left to right to obtain a first hidden state vector containing the semantic information of the previous context, and the backward LSTM unit captures the word vector and the local features from right to left to obtain a second hidden state vector containing the semantic information of the following context;

[0027] The concatenation unit concatenates the first hidden state vector and the second hidden state vector at each time step to obtain a global fusion feature;

[0028] The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature.

[0029] Furthermore, the Highway layer includes a first input layer, a second input layer, a linear transformation layer and a weighted fusion module;

[0030] The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature, including:

[0031] The first input layer receives the global fusion feature and sends the fusion feature to the linear transformation layer;

[0032] The linear transformation layer transforms the global fusion feature using trainable weights to obtain the global transformation feature and sends it to the weighted fusion module;

[0033] The second input layer is used to receive the word vector and send it to the weighted fusion module;

[0034] The weighted fusion module generates a weight matrix using a trainable parameter matrix, and performs weighted fusion on the global transformation features and word vectors according to the weight matrix to obtain the mixed features and output them.

[0035] Furthermore, the mixed features and label vectors are input into the ATTENTION layer for processing, and the classification probability is output, including:

[0036] Performing linear transformation on the mixed feature and the label vector respectively to obtain text transformation feature and label embedding feature;

[0037] Calculate and obtain a query vector, a key vector and a value vector according to the text transformation feature and the label embedding feature;

[0038] A softmax activation function is used to perform a dot product operation on the query vector and the key vector to obtain an attention weight;

[0039] Perform a weighted summation of the attention weight and the value vector to obtain a context vector corresponding to the label;

[0040] According to the context vector, a sigmoid activation function is used to output a classification probability vector to obtain the classification probability.

[0041] Furthermore, the attention weight is calculated by the following formula:

[0042]

[0043] The context vector corresponding to the label is calculated using the following formula:

[0044]

[0045] Among them, q j represents the query vector, k i represents the key vector, n represents the number of text transformation features, a ij is the attention weight, which indicates the importance of the i-th text transformation feature to the j-th label, e j represents the context vector corresponding to the jth label, v i Represents a vector of values.

[0046] A multi-label text classification device based on a neural network, comprising:

[0047] The acquisition module is used to collect multi-label text data to form training samples;

[0048] A preprocessing module, used for performing word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively;

[0049] A model building module, used to build an initial CNN-BiLSTM-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer, and an ATTENTION layer;

[0050] A training module, used for training the CNN-BiLSTM-ATTENTION neural network model: inputting the word vector into the CNN layer for local feature extraction, inputting the word vector and the extracted local feature into the BiLSTM layer for processing to obtain mixed features, inputting the mixed features and the label vector into the ATTENTION layer for processing, and outputting the classification probability; calculating the loss value according to the classification probability and updating the parameters of the CNN-BiLSTM-ATTENTION neural network model, and repeating the training steps until the CNN-BiLSTM-ATTENTION neural network model converges;

[0051] The classification module is used to receive the text data to be classified and input it into the converged CNN-BiLSTM-ATTENTION neural network model, and output the classification results.

[0052] Furthermore, the preprocessing module performs word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively, including:

[0053] The BERT model is used to encode and segment the text in the training sample to obtain a word vector;

[0054] Determine whether the label in the training sample has Chinese semantics. If the label does not have Chinese semantics, use a random generator to generate a random vector according to the dimension of word embedding, and normalize the random vector to obtain a label vector;

[0055] If the label has Chinese semantics, a data set including text and corresponding labels is established according to the training sample, a mapping relationship is established between the text and the label in the data set, a subset including the target label is generated, and the TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label.

[0056] Furthermore, the preprocessing module uses the TF-IDF algorithm to calculate the subset to obtain the label vector of the target label, including:

[0057] For each word in the text in the subset, the word frequency and the inverse document frequency are calculated, the weight of each word is calculated according to the word frequency and the inverse document frequency, the weights are sorted, and a preset number of important words that are most relevant to the target tag are selected according to the sorted weights;

[0058] Embed the selected important words to obtain multiple important word embedding vectors;

[0059] The label vector of the target label is obtained by weighted averaging the multiple important word embedding vectors.

[0060] Furthermore, the training module inputs the word vector into the CNN layer for local feature extraction, including:

[0061] Input the word vector into the convolution layer to capture local patterns and generate a feature map;

[0062] Performing nonlinear activation processing on the feature map through an activation layer;

[0063] The feature map processed by the activation layer is sent to the pooling layer for downsampling. Based on the maximum pooling strategy, the feature map is divided into multiple domains, and the maximum value outputted by each domain is used as the local feature.

[0064] Furthermore, the BiLSTM layer includes a forward LSTM unit, a backward LSTM unit, a concatenation unit and a Highway layer; the local features include local context information;

[0065] The training module inputs the word vector and the extracted local features into the BiLSTM layer for processing to obtain mixed features, including:

[0066] Input the word vector and the local feature into the forward LSTM unit and the backward LSTM unit;

[0067] The forward LSTM unit captures the word vector and the local features from left to right to obtain a first hidden state vector containing the semantic information of the previous context, and the backward LSTM unit captures the word vector and the local features from right to left to obtain a second hidden state vector containing the semantic information of the following context;

[0068] The concatenation unit concatenates the first hidden state vector and the second hidden state vector at each time step to obtain a global fusion feature;

[0069] The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature.

[0070] Furthermore, the Highway layer includes a first input layer, a second input layer, a linear transformation layer and a weighted fusion module;

[0071] The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature, including:

[0072] The first input layer receives the global fusion feature and sends the fusion feature to the linear transformation layer;

[0073] The linear transformation layer transforms the global fusion feature using trainable weights to obtain the global transformation feature and sends it to the weighted fusion module;

[0074] The second input layer is used to receive the word vector and send it to the weighted fusion module;

[0075] The weighted fusion module generates a weight matrix using a trainable parameter matrix, and performs weighted fusion on the global transformation features and word vectors according to the weight matrix to obtain the mixed features and output them.

[0076] Furthermore, the training module inputs the mixed features and label vectors into the ATTENTION layer for processing and outputs classification probabilities, including:

[0077] Performing linear transformation on the mixed feature and the label vector respectively to obtain text transformation feature and label embedding feature;

[0078] Calculate and obtain a query vector, a key vector and a value vector according to the text transformation feature and the label embedding feature;

[0079] A softmax activation function is used to perform a dot product operation on the query vector and the key vector to obtain an attention weight;

[0080] Perform a weighted summation of the attention weight and the value vector to obtain a context vector corresponding to the label;

[0081] According to the context vector, a sigmoid activation function is used to output a classification probability vector to obtain the classification probability.

[0082] Furthermore, the attention weight is calculated by the following formula:

[0083]

[0084] The context vector corresponding to the label is calculated using the following formula:

[0085]

[0086] Among them, q j represents the query vector, k i represents the key vector, n represents the number of text transformation features, a ij is the attention weight, which indicates the importance of the i-th text transformation feature to the j-th label, e j represents the context vector corresponding to the jth label, v i Represents a vector of values.

[0087] An electronic device comprises a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the above method.

[0088] The neural network-based multi-label text classification method and device provided by the present invention have at least the following beneficial effects:

[0089] (1) CNN convolutional neural network and BiLSTM bidirectional long short-term memory network are used to extract local and global semantic features and allocate label weights of multi-head attention mechanism to improve the accuracy of multi-label text classification, which is suitable for multi-dimensional label vector classification scenarios;

[0090] (2) By analyzing the relationship between the text to be classified and other categories and taking into full consideration the contextual semantic environment of the label, the accuracy of multi-label text classification and the utilization rate of data are effectively improved, and the classification accuracy can be improved under limited data conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 The present invention provides a flowchart of an embodiment of a multi-label text classification method based on a neural network.

[0092] Figure 2 The present invention provides a flowchart of an embodiment of preprocessing training samples in the multi-label text classification method based on neural network.

[0093] Figure 3 A flowchart of an embodiment of local feature extraction by a CNN layer in model training in a multi-label text classification method based on a neural network provided by the present invention.

[0094] Figure 4 A schematic diagram of an embodiment of the process of local feature extraction by a CNN layer in model training in the neural network-based multi-label text classification method provided by the present invention.

[0095] Figure 5 A schematic diagram of the structure of an embodiment of the BiLSTM layer in the neural network-based multi-label text classification method provided by the present invention.

[0096] Figure 6 A flowchart of an embodiment of obtaining mixed features by BiLSTM layer processing in the multi-label text classification method based on neural network provided by the present invention.

[0097] Figure 7 A schematic diagram of the structure of an embodiment of the Highway layer in the neural network-based multi-label text classification method provided by the present invention.

[0098] Figure 8 A flowchart of an embodiment of performing refined processing on the Highway layer in the multi-label text classification method based on a neural network provided by the present invention.

[0099] Fig. 9 A flowchart of an embodiment of ATTENTION layer processing in the neural network-based multi-label text classification method provided by the present invention.

[0100] Fig.10 A flowchart of an embodiment of a multi-label text classification device based on a neural network provided by the present invention. DETAILED DESCRIPTION

[0101] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0102] refer to Figure 1 In some embodiments, a multi-label text classification method based on a neural network is provided, comprising:

[0103] S1, collect multi-label text data to form training samples;

[0104] S2, performing word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively;

[0105] S3. Establish an initial CNN-BiLSTM-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer, and an ATTENTION layer;

[0106] S4, training the CNN-BiLSTM-ATTENTION neural network model: inputting the word vector into the CNN layer for local feature extraction, inputting the word vector and the extracted local feature into the BiLSTM layer for processing to obtain mixed features, inputting the mixed features and the label vector into the ATTENTION layer for processing, and outputting the classification probability; calculating the loss value according to the classification probability and updating the parameters of the CNN-BiLSTM-ATTENTION neural network model, and repeating the training steps until the CNN-BiLSTM-ATTENTION neural network model converges;

[0107] S5. Receive the text data to be classified and input it into the converged CNN-BiLSTM-ATTENTION neural network model, and output the classification result.

[0108] Specifically, in step S1, firstly, multi-label text data with known labels are obtained and formed into training samples, in which each text data can be assigned multiple labels at the same time. These labels may be independent of each other, or may have overlapping or hierarchical structures.

[0109] Further, refer to Figure 2 In step S2, word embedding and label embedding are performed on the training sample to obtain word vectors and label vectors respectively, including:

[0110] S21, using the BERT model to encode and segment the text in the training sample to obtain a word vector;

[0111] S22, determining whether the label in the training sample has Chinese semantics, if the label does not have Chinese semantics, using a random generator to generate a random vector according to the dimension of word embedding, and normalizing the random vector to obtain a label vector;

[0112] S23. If the label has Chinese semantics, a data set including text and corresponding labels is established according to the training sample, a mapping relationship is established between the text and the label in the data set, a subset including the target label is generated, and the TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label.

[0113] Specifically, in step S21, for the text in the training sample, the BERT model encodes and segments the input text through its pre-trained Transformer structure, and serializes the segmented characters or words to obtain word vectors.

[0114] Furthermore, in step S22, it is determined whether the label in the training sample has Chinese semantics. If the label does not have Chinese semantics, that is, the label is in English or other character types, a random generator is used to generate a random vector according to the dimension of word embedding, and the random vector is normalized to obtain a label vector.

[0115] Furthermore, in step S23, if the label has Chinese semantics, a data set containing text and corresponding labels is established based on the training sample, and a mapping relationship is established between the text and the label in the data set, that is, the corresponding relationship between the text and the label, to generate a subset containing the target label, and the target label is the label that needs to be embedded. The TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label:

[0116] S231, for each word in the text in the subset, calculating the word frequency and the inverse document frequency, calculating the weight of each word according to the word frequency and the inverse document frequency, sorting the weights, and selecting a preset number of important words that are most relevant to the target tag according to the sorted weights;

[0117] S232, embedding the selected important words to obtain multiple important word embedding vectors;

[0118] S233. Obtain a label vector of the target label by performing weighted averaging on the multiple important word embedding vectors.

[0119] Specifically, in step S231, the term frequency (TF), i.e., the frequency of a word appearing in a text, can be obtained by simply calculating the number of times a word appears in a text divided by the total number of words in the text. The inverse document frequency (IDF) indicates the rarity of a word in a subset. The greater the inverse document frequency, the rarer the word. The following TF-IDF formula is then used to calculate the weight of each word:

[0120] TF_IDF=TF(w,S)×IDF(w,D); (1)

[0121] Among them, TF_IDF represents the weight of word w, TF(w,S) represents the frequency of word w in text S, that is, word frequency, IDF(w,D) represents the rarity of word w in subset D, that is, inverse document frequency, and the weight of a word is the product of its word frequency and inverse document frequency.

[0122] Then, the words are sorted according to the weights, and a preset number of important words that are most relevant to the target label are selected according to the sorted weights. For example, the words are sorted in descending order of weights, and the first preset number of words are taken as important words. The selected important words are embedded to obtain multiple important word embedding vectors, and the label vector of the target label is obtained after weighted averaging of the multiple important word embedding vectors.

[0123] Furthermore, in step S3, an initial CNN-BiLSTM-ATTENTION neural network model is established, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer and an ATTENTION layer.

[0124] Further, refer to Figure 3 and Figure 4 In step S4, the CNN-BiLSTM-ATTENTION neural network model is trained. First, the word vector is input into the CNN layer for local feature extraction, which specifically includes:

[0125] S401, inputting the word vector into a convolutional layer to capture local patterns and generate a feature map;

[0126] S402, performing nonlinear activation processing on the feature map through an activation layer;

[0127] S403, sending the feature map processed by the activation layer to the pooling layer for down-sampling, dividing the feature map into multiple domains based on the maximum pooling strategy, and taking the maximum value outputted by each domain as the local feature.

[0128] Specifically, in the established CNN-BiLSTM-ATTENTION neural network model, the CNN layer includes a convolution layer, an activation layer and a pooling layer. In step S401, the convolution layer captures the local pattern (n-gram information) of the input word vector, generates a feature map and sends it to the activation layer. In step S402, the feature map is processed by a nonlinear activation function (ReLU activation function) to increase the expressive power and nonlinear features of the model, thereby better capturing the complex relationships in the text. Further, in step S403, the feature map processed by the activation layer is sent to the pooling layer for downsampling. Based on the maximum pooling strategy, the feature map is divided into multiple domains, and the maximum value output by each domain is used as a local feature. The pooling layer reduces the spatial size of the data, reduces the number of parameters and the amount of calculation, thereby reducing the risk of overfitting. Finally, after convolution, activation and pooling operations, the CNN layer outputs the processed local features, which are used as inputs to the subsequent BiLSTM layer for further sequence feature learning.

[0129] Further, refer to Figure 5 The BiLSTM layer includes a forward LSTM unit, a backward LSTM unit, a concatenation unit and a Highway layer; the local features output by the CNN layer include local context information.

[0130] refer to Figure 6 In step S4, the word vector and the extracted local features are input into the BiLSTM layer for processing to obtain mixed features, including:

[0131] S404, inputting the word vector and the local feature into the forward LSTM unit and the backward LSTM unit;

[0132] S405, the forward LSTM unit performs feature capture on the word vector and the local features from left to right to obtain a first hidden state vector containing the semantic information of the previous context, and the backward LSTM unit performs feature capture on the word vector and the local features from right to left to obtain a second hidden state vector containing the semantic information of the following context;

[0133] S406, a concatenation unit concatenates the first hidden state vector and the second hidden state vector at each time step to obtain a global fusion feature;

[0134] S407: Input the global fusion feature and the word vector into the Highway layer for refinement processing to obtain the mixed feature.

[0135] Specifically, in step S404, the word vector and the local features output by CNN are simultaneously input into the forward LSTM unit and the backward LSTM unit. The local features contain local context information, and the original word vector contains the semantic information of the word. In step S405, the forward LSTM unit captures the features of the word vector and the local features from left to right to obtain the first hidden state vector containing the semantic information of the previous context. At the same time, the backward LSTM unit captures the features of the word vector and the local features from right to left to obtain the second hidden state vector containing the semantic information of the following context. In step S406, the concatenation unit concatenates the first hidden state vector and the second hidden state vector at each time step, fuses the local semantic features and the contextual semantic features, and generates a bidirectional global fusion feature. Next, in step S407, the global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature.

[0136] Specifically, in step S407, refer to Figure 7 and Figure 8 ,The Highway layer includes the first input layer, the second input layer, the linear transformation layer and the weighted fusion module.

[0137] In step S407, the global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature, including:

[0138] S4071. The first input layer receives the global fusion feature and sends the fusion feature to the linear transformation layer;

[0139] S4072, the linear transformation layer transforms the global fusion feature using the trainable weights to obtain the global transformation feature and sends it to the weighted fusion module;

[0140] S4073, the second input layer is used to receive the word vector and send it to the weighted fusion module;

[0141] S4074. The weighted fusion module generates a weight matrix using a trainable parameter matrix, and performs weighted fusion on the global transformation features and word vectors according to the weight matrix to obtain the mixed features and output them.

[0142] Specifically, the Highway layer includes two inputs. The first input is the global fusion feature output by the splicing unit, and the second input is the original word vector. The first input layer receives the global fusion feature and sends the fusion feature to the linear transformation layer. The linear transformation layer transforms the global fusion feature using trainable weights to obtain a new global transformation feature. The linear transformation layer is designed to extract more refined global semantic information. Next, the weighted fusion module generates a weight matrix using a trainable parameter matrix. This weight matrix is ​​used to measure the importance of different features and determine how to highlight or suppress specific global features in the mixed features. Through the weight matrix, the original word vector is weightedly fused with the global transformation feature output by the linear transformation layer to obtain a mixed feature. The mixed feature contains local features, global features, and semantic feature representations optimized by the weight matrix.

[0143] Further, refer to Fig. 9 In step S4, the mixed feature and label vector are input into the ATTENTION layer for processing, and the classification probability is output, including:

[0144] S408, performing linear transformation on the mixed feature and the label vector respectively to obtain a text transformation feature and a label embedding feature;

[0145] S409, calculating and obtaining a query vector, a key vector and a value vector according to the text transformation feature and the label embedding feature;

[0146] S410, performing a dot product operation on the query vector and the key vector using a softmax activation function to obtain an attention weight;

[0147] S411, performing weighted summation on the attention weight and the value vector to obtain a context vector corresponding to the label;

[0148] S412. According to the context vector, a sigmoid activation function is used to output a classification probability vector to obtain a classification probability.

[0149] Specifically, in step S408, the mixed feature and the label vector are linearly transformed respectively to ensure that they have the same dimensional size, thereby obtaining a text transformation feature and a label embedding feature.

[0150] Further, in step S409, it is assumed that H and L are the matrix of the converted text transformation features and the matrix of the label embedding, respectively, where h i Represented as the i-th text transformation feature (i-th row of H), l jDenoted as the jth label embedding (jth row in L). Then the weight matrix W with the same number of trainable parameters is used. q , W k , W v , generate query vector q j , key vector k i , value vector v i :

[0151]

[0152] Furthermore, in step S410, a softmax activation function is used to perform a dot product operation on the query vector and the key vector to obtain an attention weight, which is calculated by the following formula:

[0153]

[0154] Among them, q j represents the query vector, k i represents the key vector, n represents the number of text transformation features, a ij is the attention weight, which indicates the importance of the i-th text transformation feature to the j-th label.

[0155] Furthermore, in step S411, the attention weight and the value vector are weighted and summed to obtain a context vector corresponding to the label, that is:

[0156]

[0157] Among them, a ij is the attention weight, which indicates the importance of the i-th text transformation feature to the j-th label, e j represents the context vector corresponding to the jth label, v i Represents a vector of values.

[0158] Furthermore, in step S412, according to the context vector, a sigmoid activation function is used to output a classification probability vector to obtain a classification probability. The sigmoid activation function is specifically:

[0159]

[0160] Among them, x is the context vector and f(x) is the output classification probability.

[0161] Furthermore, in step S4, based on the loss function, according to the classification probability of the label category of each text and the known label category of each text, the loss value of the CNN-BiLSTM-ATTENTION neural network model is calculated. When the loss value is not less than a preset threshold, the parameters of the CNN-BiLSTM-ATTENTION neural network model are updated according to the loss value, and the updated CNN-BiLSTM-ATTENTION neural network model is used to continue to obtain the prediction results of the label category of each text until the calculated loss value of the network model is less than the preset threshold, thereby obtaining the trained CNN-BiLSTM-ATTENTION neural network model.

[0162] In some embodiments, the trained CNN-BiLSTM-ATTENTION neural network model is finally tested according to the test set, and the text to be classified is classified using the tested network model.

[0163] Furthermore, in some embodiments, the loss function of the CNN-BiLSTM-ATTENTION neural network model may use a standard binary cross entropy loss function as the loss function, and the specific expression is:

[0164]

[0165] Among them, Loss l represents the loss value, It is expressed as the probability that the label is predicted as l, N represents the number of data points in the training sample, where the i-th data point consists of text and indicator If the text tag is l, then the indicator is 1, otherwise it is set to 0.

[0166] Further, refer to Fig.10 In some embodiments, a multi-label text classification device based on a neural network is also provided, comprising:

[0167] A collection module 201 is used to collect multi-label text data to form training samples;

[0168] A preprocessing module 202 is used to perform word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively;

[0169] A model building module 203 is used to build an initial CNN-BiLSTM-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer, and an ATTENTION layer;

[0170] The training module 204 is used to train the CNN-BiLSTM-ATTENTION neural network model: input the word vector into the CNN layer to extract local features, input the word vector and the extracted local features into the BiLSTM layer for processing to obtain mixed features, input the mixed features and the label vector into the ATTENTION layer for processing, and output the classification probability; calculate the loss value according to the classification probability and update the parameters of the CNN-BiLSTM-ATTENTION neural network model, and repeat the training steps until the CNN-BiLSTM-ATTENTION neural network model converges;

[0171] The classification module 205 is used to receive the text data to be classified and input it into the converged CNN-BiLSTM-ATTENTION neural network model, and output the classification result.

[0172] Furthermore, the preprocessing module 202 performs word embedding and label embedding on the training sample to obtain a word vector and a label vector respectively, including:

[0173] The BERT model is used to encode and segment the text in the training sample to obtain a word vector;

[0174] Determine whether the label in the training sample has Chinese semantics. If the label does not have Chinese semantics, use a random generator to generate a random vector according to the dimension of word embedding, and normalize the random vector to obtain a label vector;

[0175] If the label has Chinese semantics, a data set including text and corresponding labels is established according to the training sample, a mapping relationship is established between the text and the label in the data set, a subset including the target label is generated, and the TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label.

[0176] Furthermore, the preprocessing module 202 uses the TF-IDF algorithm to calculate the subset to obtain the label vector of the target label, including:

[0177] For each word in the text in the subset, the word frequency and the inverse document frequency are calculated, the weight of each word is calculated according to the word frequency and the inverse document frequency, the weights are sorted, and a preset number of important words that are most relevant to the target tag are selected according to the sorted weights;

[0178] Embed the selected important words to obtain multiple important word embedding vectors;

[0179] The label vector of the target label is obtained by weighted averaging the multiple important word embedding vectors.

[0180] Furthermore, the training module 204 inputs the word vector into the CNN layer for local feature extraction, including:

[0181] Input the word vector into the convolution layer to capture local patterns and generate a feature map;

[0182] Performing nonlinear activation processing on the feature map through an activation layer;

[0183] The feature map processed by the activation layer is sent to the pooling layer for downsampling. Based on the maximum pooling strategy, the feature map is divided into multiple domains, and the maximum value outputted by each domain is used as the local feature.

[0184] Further, the BiLSTM layer includes a forward LSTM unit, a backward LSTM unit and a Highway layer; the local features include local context information;

[0185] The training module 204 inputs the word vector and the extracted local features into the BiLSTM layer for processing to obtain mixed features, including:

[0186] Input the word vector and the local feature into the forward LSTM unit and the backward LSTM unit;

[0187] The forward LSTM unit captures the word vector and the local features from left to right to obtain a first hidden state vector containing the semantic information of the previous context, and the backward LSTM unit captures the word vector and the local features from right to left to obtain a second hidden state vector containing the semantic information of the following context;

[0188] Concatenate the first hidden state vector and the second hidden state vector at each time step to obtain a global fusion feature;

[0189] The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature.

[0190] Furthermore, the Highway layer includes a first input layer, a second input layer, a linear transformation layer and a weighted fusion module;

[0191] The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature, including:

[0192] The first input layer receives the global fusion feature and sends the fusion feature to the linear transformation layer;

[0193] The linear transformation layer transforms the global fusion feature using trainable weights to obtain the global transformation feature and sends it to the weighted fusion module;

[0194] The second input layer is used to receive the word vector and send it to the weighted fusion module;

[0195] The weighted fusion module generates a weight matrix using a trainable parameter matrix, and performs weighted fusion on the global transformation features and word vectors according to the weight matrix to obtain the mixed features and output them.

[0196] Furthermore, the training module 204 inputs the mixed features and label vectors into the ATTENTION layer for processing and outputs classification probabilities, including:

[0197] Performing linear transformation on the mixed feature and the label vector respectively to obtain text transformation feature and label embedding feature;

[0198] Calculate and obtain a query vector, a key vector and a value vector according to the text transformation feature and the label embedding feature;

[0199] A softmax activation function is used to perform a dot product operation on the query vector and the key vector to obtain an attention weight;

[0200] Perform a weighted summation of the attention weight and the value vector to obtain a context vector corresponding to the label;

[0201] According to the context vector, a sigmoid activation function is used to output a classification probability vector to obtain the classification probability.

[0202] Furthermore, the attention weight is calculated by the following formula:

[0203]

[0204] The context vector corresponding to the label is calculated using the following formula:

[0205]

[0206] Among them, q j represents the query vector, k i represents the key vector, n represents the number of text transformation features, a ij is the attention weight, which indicates the importance of the i-th text transformation feature to the j-th label, e j represents the context vector corresponding to the jth label, v i Represents a vector of values.

[0207] In some embodiments, an electronic device is provided, including a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is configured to read the instructions and execute the above method.

[0208] (1) CNN convolutional neural network and BiLSTM bidirectional long short-term memory network are used to extract local and global semantic features and allocate label weights of multi-head attention mechanism to improve the accuracy of multi-label text classification, which is suitable for multi-dimensional label vector classification scenarios;

[0209] (2) By analyzing the relationship between the text to be classified and other categories and taking into full consideration the contextual semantic environment of the label, the accuracy of multi-label text classification and the utilization rate of data are effectively improved, and the classification accuracy can be improved under limited data conditions.

[0210] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-label text classification method based on neural network, characterized in that: include: Collect multi-label text data to form training samples; Performing word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively; Establishing an initial CNN-BiLSTM-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer, and an ATTENTION layer; The CNN-BiLSTM-ATTENTION neural network model is trained: the word vector is input into the CNN layer for local feature extraction, the word vector and the extracted local feature are input into the BiLSTM layer for processing to obtain mixed features, the mixed features and the label vector are input into the ATTENTION layer for processing, and the classification probability is output; the loss value is calculated according to the classification probability and the parameters of the CNN-BiLSTM-ATTENTION neural network model are updated, and the training steps are repeated until the CNN-BiLSTM-ATTENTION neural network model converges; Receive the text data to be classified and input it into the converged CNN-BiLSTM-ATTENTION neural network model, and output the classification result.

2. The method according to claim 1, characterized in that Performing word embedding and label embedding on the training sample to obtain word vectors and label vectors respectively includes: The BERT model is used to encode and segment the text in the training sample to obtain a word vector; Determine whether the label in the training sample has Chinese semantics. If the label does not have Chinese semantics, use a random generator to generate a random vector according to the dimension of word embedding, and normalize the random vector to obtain a label vector; If the label has Chinese semantics, a data set including text and corresponding labels is established according to the training sample, a mapping relationship is established between the text and the label in the data set, a subset including the target label is generated, and the TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label.

3. The method according to claim 2, characterized in that The TF-IDF algorithm is used to calculate the subset to obtain the label vector of the target label, including: For each word in the text in the subset, the word frequency and the inverse document frequency are calculated, the weight of each word is calculated according to the word frequency and the inverse document frequency, the weights are sorted, and a preset number of important words that are most relevant to the target tag are selected according to the sorted weights; Embed the selected important words to obtain multiple important word embedding vectors; The label vector of the target label is obtained by weighted averaging the multiple important word embedding vectors.

4. The method according to claim 1, characterized in that: Inputting the word vector into the CNN layer for local feature extraction includes: Input the word vector into the convolution layer to capture local patterns and generate a feature map; Performing nonlinear activation processing on the feature map through an activation layer; The feature map processed by the activation layer is sent to the pooling layer for downsampling. Based on the maximum pooling strategy, the feature map is divided into multiple domains, and the maximum value outputted by each domain is used as the local feature.

5. The method according to claim 1, characterized in that The BiLSTM layer includes a forward LSTM unit, a backward LSTM unit, a concatenation unit and a Highway layer; the local features include local context information; The word vector and the extracted local features are input into the BiLSTM layer for processing to obtain mixed features, including: Input the word vector and the local feature into the forward LSTM unit and the backward LSTM unit; The forward LSTM unit captures the word vector and the local features from left to right to obtain a first hidden state vector containing the semantic information of the previous context, and the backward LSTM unit captures the word vector and the local features from right to left to obtain a second hidden state vector containing the semantic information of the following context; The concatenation unit concatenates the first hidden state vector and the second hidden state vector at each time step to obtain a global fusion feature; The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature.

6. The method according to claim 5, characterized in that The Highway layer includes a first input layer, a second input layer, a linear transformation layer and a weighted fusion module; The global fusion feature and the word vector are input into the Highway layer for refinement processing to obtain the mixed feature, including: The first input layer receives the global fusion feature and sends the fusion feature to the linear transformation layer; The linear transformation layer transforms the global fusion feature using trainable weights to obtain the global transformation feature and sends it to the weighted fusion module; The second input layer is used to receive the word vector and send it to the weighted fusion module; The weighted fusion module generates a weight matrix using a trainable parameter matrix, and performs weighted fusion on the global transformation features and word vectors according to the weight matrix to obtain the mixed features and output them.

7. The method according to claim 1, characterized in that The mixed features and label vectors are input into the ATTENTION layer for processing, and the classification probability is output, including: Performing linear transformation on the mixed feature and the label vector respectively to obtain text transformation feature and label embedding feature; Calculate and obtain a query vector, a key vector and a value vector according to the text transformation feature and the label embedding feature; A softmax activation function is used to perform a dot product operation on the query vector and the key vector to obtain an attention weight; Perform a weighted summation of the attention weight and the value vector to obtain a context vector corresponding to the label; According to the context vector, a sigmoid activation function is used to output a classification probability vector to obtain the classification probability.

8. The method according to claim 7, characterized in that The attention weight is calculated by the following formula: The context vector corresponding to the label is calculated using the following formula: Among them, q j represents the query vector, k i represents the key vector, n represents the number of text transformation features, a ij is the attention weight, which indicates the importance of the i-th text transformation feature to the j-th label, e j represents the context vector corresponding to the jth label, v i Represents a vector of values.

9. A multi-label text classification device based on a neural network, characterized in that: include: The acquisition module is used to collect multi-label text data to form training samples; A preprocessing module, used for performing word embedding and label embedding on the training samples to obtain word vectors and label vectors respectively; A model building module, used to build an initial CNN-BiLSTM-ATTENTION neural network model, wherein the CNN-BiLSTM-ATTENTION neural network model includes a CNN layer, a BiLSTM layer, and an ATTENTION layer; A training module, used for training the CNN-BiLSTM-ATTENTION neural network model: inputting the word vector into the CNN layer for local feature extraction, inputting the word vector and the extracted local feature into the BiLSTM layer for processing to obtain mixed features, inputting the mixed features and the label vector into the ATTENTION layer for processing, and outputting the classification probability; calculating the loss value according to the classification probability and updating the parameters of the CNN-BiLSTM-ATTENTION neural network model, and repeating the training steps until the CNN-BiLSTM-ATTENTION neural network model converges; The classification module is used to receive the text data to be classified and input it into the converged CNN-BiLSTM-ATTENTION neural network model, and output the classification results.

10. An electronic device, characterized in that: The method comprises a processor and a storage device, wherein the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1 to 8.

Images