A hierarchical multi-label text classification method based on hybrid attention

Abstract

The present invention provides a hierarchical multi-label text classification method based on hybrid attention. The method uses pre-trained word vectors as word embeddings, uses Bi-GRU to perform preliminary feature extraction on the input word embeddings; uses a graph convolutional neural network to model a hierarchical label structure system, and generates a label representation containing label associations; uses multiple convolutional neural networks with different convolution kernel sizes to further perform local feature extraction of different granularities on the output of Bi-GRU, performs maximum pooling and splices them into a text feature, and further extracts features from the text feature using attention based on label representation; at the same time, uses a self-attention mechanism to perform global feature extraction on the output of Bi-GRU, adaptively fuses the text features based on label representation and the text features based on self-attention mechanism representation, obtains a text representation based on hybrid attention, finally extracts information between labels through a relational network, and obtains the final classification result through a multi-layer perceptron.

Description

Technical Field

[0001] The present invention relates to the fields of computer information technology and natural language processing, and more specifically, to a hierarchical multi-label text classification method based on hybrid attention. Background Art

[0002] The advent of the Internet era has enabled people to access all kinds of information more conveniently. At the same time, various media data are also generated continuously, which provides the basic conditions for mining valuable data on the Internet. If there is a lack of efficient management methods and means of acquiring knowledge for massive amounts of data, it is undoubtedly a waste. In data mining, text classification is one of the core issues.

[0003] The task of multi-label text classification is to select the subset that is most relevant to the text content in a given set of labels. In actual scenarios, a lot of data is related to multiple labels in the label set. These labels can concisely show the specific content of the data, allowing people to manage massive data more conveniently and effectively, and further analyze the data. Hierarchical multi-label text classification is a special case of multi-label text classification. Its label system has a hierarchical structure. The general multi-label text classification algorithm does not consider the impact of the hierarchical label structure on the classification effect, and does not fully utilize the correlation information between text labels, resulting in inaccurate identification of the category to which it belongs during classification, especially for data with long-tail distribution. There is still a lot of room for improvement in the classification effect. At the same time, most existing models focus on the local features or global features of the text, lack of comprehensive consideration of local and global features, so that the important features involved in classification are not captured enough. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a hierarchical multi-label text classification method based on hybrid attention, which improves the performance of hierarchical multi-label text classification by utilizing the label hierarchy for label semantic representation and making full use of the global and local semantic information of the text.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: a hierarchical multi-label text classification method based on hybrid attention, the method comprising the following steps:

[0006] S1, multi-label text data preprocessing; text data is used to train the model. It consists of text content and its corresponding label set. There is a tree diagram with a hierarchical relationship between all label categories in the entire data set. For the tree diagram, it is composed of many nodes, each node represents a label category, and the label corresponding to each sample text in the data set comes from the node on this label tree diagram;

[0007] S2, for the text label, obtaining the prior level information of the hierarchical classification system, wherein the prior level information refers to the prior probability of mutual dependence between labels, which can be obtained by calculating the transition probability between the parent label and the child label;

[0008] S3, build a deep learning hierarchical multi-label text classification model;

[0009] The deep learning multi-label text classification model includes a word embedding module, a text encoding module, a label encoding module, a text representation module based on a label attention mechanism, a text representation module based on a self-attention mechanism, a feature fusion module, a vector regression layer, a relational network module, and a label probability prediction layer;

[0010] S4, input the text data after data set preprocessing into the model training; after the model training is completed, use the trained model to classify the multi-label text.

[0011] In the above technical solution, step S1 includes data preprocessing of samples in data set D, specifically including the following steps: step 1.1, word segmentation, removal of stop words, and removal of punctuation marks; step 1.2, statistically calculating the word frequency word_frequence in the text in data set D, deleting words with a number of occurrences less than X1, recording the filtered words, and constructing a word list. After data set D is preprocessed, data set D is divided into a training set, a validation set, and a test set according to a certain ratio.

[0012] In the above technical solution, step S2 includes: for the data of the training set in the data set D, assuming that the parent node v i and child node v j There are hierarchical paths between i,j , then the feature f(e i,j ) is given by the prior probability p(U j |U i ) and p(U i |U j )express:

[0013]

[0014] f(e i,j ) represents the relationship between two nodes. This relationship is described by the transition probability or co-occurrence probability of the two nodes. The transition probability of the two nodes includes the transition probability p(U j |U i ), the transition probability from child node to parent node p(U i |U j), a parent label node may contain multiple child label nodes, and the sum of the transfer probabilities from the parent label node to each child label node is 1. If there is only one child node under the parent label node, then the value is 1 at this time; if there are multiple child labels, then this value is less than 1 at this time, but their sum is 1; where U j and U i Respectively indicate that the text sample is marked as v j Node labels and are marked as v i Node label, p(U j |U i ) is a given v i The case is marked as v j The conditional probability of the node label. P(U j ∩U i ) is {v j ,v i} are marked at the same time. j and N i They represent v in the training set respectively. j Node label and v i The number of node labels.

[0015] In the above technical solution, step 3 also includes performing word embedding processing on the input text and its labels through a word embedding module, and the word embedding processing method is specifically as follows:

[0016] Step 2.1: Get the preprocessed text sequence and embed the words {x1,x2,...,x n} is converted to a word vector representation {w1,w2,...,w n}, n refers to the number of words in the preprocessed text.

[0017] Step 2.2: Obtain the label set {l1,l2,...,l n}, through the Kaiming encoding method, the label set is converted into a label embedding set {c1,c2,...,c n}.

[0018] In the above technical solution, step S3 also includes encoding the word vector representation {w1,w2,...,w n} to perform encoding processing, the encoding processing method is specifically as follows:

[0019] Use Bi-GRU network to represent the word vector of the text {w1,w2,...,w n} to generate implicit representations {h1,h2,...,h n}. Then the implicit representation {h1,h2,...,h n}Furthermore, three convolutions with different kernel sizes are fed into the convolutional layer, and three semantic vectors with different receptive fields are obtained. Finally, the three semantic vectors are concatenated into a new semantic representation vector S = {s1, s2, ..., s n}.

[0020] In the above technical solution, step S3 also includes encoding the label vector {c1, c2, ..., c n} for encoding processing, the label encoding processing method is as follows:

[0021] Use a single-layer GCN to represent the label vector {c1,c2,...,c n} to generate an implicit representation M={m1,m2,...,m n}. The implementation process is as follows:

[0022] Hierarchical GCN aggregates data flows within top-down, bottom-up, and self-loop edges. In hierarchical GCN, each directed edge represents a pair of label-related features, and these data flows are node-transformed using linear transformations along the edges.

[0023] In order to achieve node transformation, the present invention uses a weighted adjacency matrix to represent this linear transformation, and the initial value of the weighted adjacency matrix comes from the prior level information of the hierarchical classification system in step 2. Formally, the hierarchical GCN encodes the hidden state of node k according to its related neighborhood, where neighborhood N(k) = {n k ,child(k),parent(k)},n k Refers to the kth label node in the hierarchical label tree, child(k) refers to the child label node of the kth node, parent(k) refers to the parent label node of the kth node, and the hidden state of node k is calculated as follows:

[0024]

[0025] In the above formula, v j , v k are trainable parameters. and is a trainable bias parameter; for u k,j and k,j For example, u k,j Understand the information between nodes k and j, g k,j Understand it as the gate value, control u k,j Last pair

[0026] The influence of node k; σ refers to the activation function in deep learning, which can be taken as the sigmoid function. b l ∈R N×dim , and b g ∈R N , dim is the dimension of the vector, which is a predefined hyperparameter. d(j, k) represents the hierarchical direction from node j to node k, including top-down, bottom-up, and self-loop edges. k,j ∈R represents the hierarchical probability f d(k,j) (e kj ), f d(k,j) (e kj ) refers to the transition probability from the kth node to the jth label node. It is obtained by f(e_(i,j)) above, and the self-loop edge uses a k,k = 1, top-down edges use The bottom-up edges use f p (e j,k )=1. The characteristic matrix F of the above edge is 0,0 ,a 0,1 ,…,a c-1,c-1} represents the weighted adjacency matrix of the text label directed hierarchical graph. Finally, the output hidden state h of node k k Indicates that it corresponds to a label representation of hierarchical information.

[0027] In the above technical solution, step S3 also includes a text representation module based on a label attention mechanism: representing the text from the text encoding layer and the label representation from the label encoding layer d c It represents the dimension size of the text encoding vector, which is a predetermined fixed value. The text representation based on label attention is calculated by the following formula:

[0028]

[0029] Among them, α kj Represents the amount of information of the j-th text feature vector for the k-th label. k That is, text representation based on label attention.

[0030] In the above technical solution, step S3 also includes a text representation module based on a self-attention mechanism: representing the hidden text output from the Bi-GRU in the text encoding layer The text representation based on the self-attention mechanism is calculated by the following formula:

[0031]

[0032] Among them, w1, w2 are parameters, h is the text representation, α kt is the weight of the tth vector in the text representation, u k It is a text representation based on self-attention mechanism.

[0033] In the above technical solution, step S3 also includes a feature fusion module: adaptively fusing the text features based on the label attention mechanism and the text features based on the self-attention mechanism to obtain the final text feature d ik-fusio , calculated as follows:

[0034]

[0035] Among them w1, w2 are parameters, v k is the text representation based on label attention, u k is the text representation based on self-attention, β k v k The weight it occupies.

[0036] In the above technical solution, step S3 also includes a relational network module to further mine the associated information between tags: the mining method is to combine the text features d generated by the feature fusion module ik-fusion Input to the fully connected layer to get the logits vector corresponding to each label O = {o1,o2,...,o n}, and then input the vector O into the relationship network module to obtain the prediction vector y = {y1, y2, ..., y n}, and finally input the prediction vector y into the multi-layer perceptron to get the label prediction probability, where the relational network is essentially a residual network.

[0037] In the above technical solution, step S4 includes using a cross entropy loss function during training, using the Adam optimizer for training, and the cross entropy loss function for multi-label text classification is as follows:

[0038]

[0039] Among them, y ij is the actual probability of the i-th sample for the j-th label, is the predicted probability of the i-th sample for the j-th label, and finally the trained deep learning multi-label text classification model is obtained. L refers to the number of label categories, and N refers to the number of sample texts.

[0040] The advantages and beneficial effects of the present invention are as follows:

[0041] The present invention uses Bi-GRU combined with CNN to extract the semantic representation of text, and can obtain the local semantic information of text more fully. The present invention uses a graph neural network to represent the hierarchical information of hierarchical multi-label classification, and can obtain a label representation with hierarchical association information. The present invention uses a self-attention mechanism to extract the semantic representation of text, and can obtain a semantic representation of the global association of text. The present invention uses an adaptive fusion of text features based on label representation and text features based on self-attention representation, and can obtain a text representation that connects global, local text, and label information. The present invention uses a relational network in the last layer of the model, which can enable the original label prediction vector to further obtain label association.

[0042] The present invention includes four aspects: first, a graph convolutional neural network is used to extract label representations that contain hierarchical relationships; second, multiple convolutions of different granularities are used to extract local features; third, a label-based attention mechanism and a self-attention mechanism are used to further extract text features, and adaptively fuse them (FA). Fourth, a relational network is used to further extract label relevance. The present invention provides a hierarchical multi-label classification method based on hybrid attention, which extracts text features from the input text to be classified, and then classifies it through a multi-layer perceptron, and can mark the text with one or more labels. It can be widely used in e-commerce, news, scientific papers and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the hierarchical multi-label text classification method based on hybrid attention of the present invention;

[0044] Figure 2 This is a network structure diagram of a hierarchical multi-label text classification model based on hybrid attention in the present invention;

[0045] Figure 3 This is a schematic diagram of the hierarchical structure of hierarchical multi-label text classification labels of the present invention;

[0046] Figure 4 It is a schematic diagram of the calculation of the hierarchical multi-label text classification graph convolutional neural network of the present invention;

[0047] Figure 5 Schematic diagram of the relationship network of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] In order to achieve the above object, the technical solution adopted by the present invention is: a hierarchical multi-label text classification method based on hybrid attention, the method comprising the following steps:

[0050] Step S1, preprocessing the multi-label text data in the data set D;

[0051] Step S2, for the text label, obtaining the priori hierarchical information of the hierarchical classification system, wherein the priori hierarchical information refers to the priori probability of mutual dependence between labels, which can be obtained by calculating the transition probability between the parent label and the child label;

[0052] Step S3, constructing a deep learning hierarchical multi-label text classification model;

[0053] The deep learning multi-label text classification model includes a word embedding module, a text encoding module, a label encoding module, a text representation module based on a label attention mechanism, a text representation module based on a self-attention mechanism, a feature fusion module, a vector regression layer, a relational network module and a label probability prediction layer;

[0054] Step S4, input the text data after data set preprocessing into the model training. After the model training is completed, the trained model is used to classify the multi-label text.

[0055] Preferably, step S1 comprises the following steps:

[0056] Data preprocessing is performed on the samples in the data set D, which specifically includes the following steps:

[0057] Step 1-1, segment the text in data set D, remove stop words, and remove punctuation marks;

[0058] Step 1-2, count the word frequency word_frequence in the text in the statistical data set D, delete the words whose occurrence times are less than X1, record the filtered words, and build a word list.

[0059] In step 1-3, after preprocessing, the dataset D is divided into a training set, a validation set, and a test set according to a ratio of 3:1:1.

[0060] Preferably, step S2 comprises the following steps:

[0061] For the data of the training set in dataset D, assume that the parent node v i and child node v j There are hierarchical paths between i,j Then the feature f(e) of the edge formed by the parent-child node path i,j ) is given by the prior probability p(U j |U i ) and p(Ui |U j )express:

[0062]

[0063] f(e i,j ) represents the relationship between two nodes. This relationship is described by the transition probability or co-occurrence probability of the two nodes. The transition probability of the two nodes includes the transition probability p(U j |U i ), the transition probability from child node to parent node p(U i |U j ), where U j and U i Respectively indicate that the text data is marked as v j Node labels and are marked as ν i Node label, p(U j |U i ) is a given v i The case is marked as v j The conditional probability of the node label, P(U j ∩U i ) is {v j ,v i} is marked at the same time, N j and N i They represent v in the training set respectively. j Node labels and ν i The number of node labels.

[0064] Preferably, step S3 further comprises the following steps:

[0065] The word embedding module is used to embed the input text and its labels. The word embedding processing method is as follows:

[0066] Step 2-1: Get the preprocessed text sequence and embed the words {x1, x2, ..., x in the text by querying the word embedding table (Glove-300d). n} is converted to a word vector representation {w1,w2,...,w n}.

[0067] Step 2-2: Obtain the label set {l1, l2, ..., l n}, through the Kaiming encoding method, the label set is converted into a label embedding set {c1,c2,...,c n}, n refers to the number of words in the preprocessed text.

[0068] Preferably, step S3 further comprises the following steps:

[0069] The word vector {w1,w2,...,w n} to perform encoding processing, the encoding processing method is specifically as follows:

[0070] Use Bi-GRU network to represent the word vector of the text {w1,w2,...,w n} to generate implicit representations {h1,h2,...,h n}. Then the implicit representation {h1,h2,...,h n}Furthermore, three convolution kernels with sizes of 2, 3, and 4 and a hidden layer number of 100 are fed into the convolution, and three semantic vectors with different receptive fields are obtained. After the maximum pooling, the three semantic vectors are concatenated into a new semantic representation vector S = {s1, s2, ..., s n}.

[0071] Preferably, step S3 further comprises the following steps:

[0072] The label vector {c1,c2,...,c n} is encoded. The label encoding method is to use a single-layer GCN to represent the label vector {c1,c2,...,c n} to generate an implicit representation M={m1,m2,...,m n}. The implementation process is as follows:

[0073] Hierarchical GCN aggregates data flows within top-down, bottom-up, and self-loop edges. In hierarchical GCN, each directed edge represents a pair of label-related features, and these data flows are node-transformed using linear transformations along the edges.

[0074] In order to achieve node transformation, the present invention uses a weighted adjacency matrix to represent this linear transformation, and the initial value of the weighted adjacency matrix comes from the prior level information of the hierarchical classification system in step S2. Formally, the hierarchical GCN encodes the hidden state of node k according to its related neighborhood, where neighborhood N(k) = {n k ,child(k),parent(k)},n k Refers to the kth label node in the hierarchical label tree, child(k) refers to the child label node of the kth node, parent(k) refers to the parent label node of the kth node, and the hidden state of node k is calculated as follows:

[0075]

[0076] In the above formula, v j , v k are trainable parameters. and is a trainable bias parameter; for u k,j and k,j For example, u k,j Understand the information between nodes k and j, g k,j Understand it as the gate value, control u k,j Finally, the impact on node k; σ refers to the activation function in deep learning, which can be taken as the sigmoid function. b l ∈R N×dim , and b g ∈R N , dim is the dimension of the vector, which is a predefined hyperparameter; d(j, k) represents the hierarchical direction from node j to node k, including top-down, bottom-up and self-loop edges; where a k,j ∈R represents the hierarchical probability f d(k,j) (e kj ), f d(k,j) (e kj ) refers to the transition probability from the kth node to the jth label node, which is obtained by f(e i,j ) to obtain the self-loop edge using a k,k = 1, top-down edges use The bottom-up edges use f p (e j,k )=1;The characteristic matrix F of the above edge={a 0,0 ,a 0,1 ,…,a c-1,c-1} represents the weighted adjacency matrix of the text label directed hierarchical graph. Finally, the output hidden state h of node k k Indicates that it corresponds to a label representation of hierarchical information.

[0077] Preferably, step S3 further comprises the following steps:

[0078] The extraction method of the text representation module based on the label attention mechanism is: the text representation from the text encoding layer and the label representation from the label encoding layer The text representation based on label attention is calculated by the following formula:

[0079]

[0080] Among them, α kj Represents the amount of information of the j-th text feature vector for the k-th label.k That is, text representation based on label attention.

[0081] Preferably, step S3 further comprises the following steps:

[0082] The extraction method of the text representation module based on the self-attention mechanism is as follows: the hidden text representation from the Bi-GRU output in the text encoding layer The text representation based on the self-attention mechanism is calculated by the following formula:

[0083]

[0084] Among them, w1, w2 are parameters, h is the text representation, α kt is the weight of the tth vector in the text representation, u k It is a text representation based on self-attention mechanism.

[0085] Preferably, step S3 further comprises the following steps:

[0086] The feature fusion module is: adaptively fuse the text features based on the label attention mechanism and the text features based on the self-attention mechanism to obtain the final text feature d ik-fusion , calculated as follows:

[0087]

[0088] Among them w1, w2 are parameters, v k is the text representation based on label attention, u k is the text representation based on self-attention, β k v k The weight it occupies.

[0089] Preferably, step S3 further comprises the following steps:

[0090] The relational network module is used to further mine the association information between tags: the mining method is to combine the text features generated by the feature fusion module ik-fusion Input to the fully connected layer to get the logits vector corresponding to each label O = {o1,o2,...,o n}, then input the vector O into the relationship network module to obtain the prediction vector y = {y1, y2, ..., y n}, and finally input the prediction vector y into the multi-layer perceptron to get the label prediction probability, where the relational network is essentially a residual network.

[0091] Preferably, step S4 comprises the following steps:

[0092] The cross entropy loss function needs to be used during training, and the Adam optimizer is used for training. The cross entropy loss function for multi-label text classification is as follows:

[0093]

[0094] Among them, y ij is the actual probability of the i-th sample for the j-th label, is the predicted probability of the i-th sample for the j-th label, and finally we get the trained deep learning multi-label text classification model. L refers to the number of label categories, and N refers to the number of text samples.

[0095] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A hierarchical multi-label text classification method based on hybrid attention, It is characterized in that The method comprises the following steps: Step S1, preprocessing the multi-label text data in the data set D; Step S2, for the text label, obtaining the priori hierarchical information of the hierarchical classification system, wherein the priori hierarchical information refers to the priori probability of mutual dependence between labels, which can be obtained by calculating the transition probability between the parent label and the child label; Step S3, constructing a deep learning hierarchical multi-label text classification model; The deep learning hierarchical multi-label text classification model includes a word embedding module, a text encoding module, a label encoding module, a text representation module based on a label attention mechanism, a text representation module based on a self-attention mechanism, a feature fusion module, a vector regression layer, a relational network module and a label probability prediction layer; The extraction method of the label attention mechanism text representation module in step S3 is: the text representation from the text encoding layer and the label representation from the label encoding layer d c It represents the dimension size of the text encoding vector, which is a predetermined fixed value. The text representation based on label attention is calculated by the following formula: Among them, α kj represents the information content of the j-th text feature vector for the k-th label, v k That is, text representation based on label attention; The extraction method of the self-attention mechanism text representation module in step S3 is as follows: the hidden text representation output from the Bi-GRU in the text encoding layer The text representation based on the self-attention mechanism is calculated by the following formula: where w 1 , w 2 is a parameter, h is the text representation, α kt is the weight of the tth vector in the text representation, u k It is a text representation based on self-attention mechanism; The feature fusion module in step S3 is: adaptively fuse the text features based on the label attention mechanism and the text features based on the self-attention mechanism to obtain the final text feature d ik - fusion , calculated as follows: where w 1 , w 2 is the parameter, v k is the text representation based on label attention, u k is the text representation based on self-attention, β k v k The weight it occupies; The relational network module in step S3 further mines the association information between tags: the mining method is to combine the text features d generated by the feature fusion module ik - fusion Input to the fully connected layer to get the logits vector O corresponding to each label = {o 1 ,o 2 ,...,o n }, and then input the vector O into the relationship network module to obtain the prediction vector y = {y 1 ,y 2 ,...,y n }, finally, the prediction vector y is input into the multi-layer perceptron to obtain the label prediction probability, where the relational network is essentially a residual network; Step S4, input the text data after data set preprocessing into the model training. After the model training is completed, the trained model is used to classify the multi-label text.

2. The hierarchical multi-label text classification method based on label attention according to claim 1, Features: In step S1, the text data in the data set D is preprocessed, specifically including the following steps: Step 1.1, perform word segmentation, remove stop words, and remove punctuation marks; Step 1.2, calculate the word frequency word_frequence in the text in the statistical data set D, delete the words whose occurrence times are less than X1, record the filtered words, and build a word list; After preprocessing, the dataset D is divided into training set, validation set, and test set according to a certain ratio.

3. The hierarchical multi-label text classification method based on label attention according to claim 1, Features: The specific implementation of step S2 includes: For the data in dataset D, assume that the parent node v i and child node v j There are hierarchical paths i,j , then the feature f(e i,j ) is given by the prior probability p(U j |U i ) and p(U i |U j )express: f(e i,j ) represents the relationship between two nodes. This relationship is described by the transition probability or co-occurrence probability of the two nodes. The transition probability of the two nodes includes the transition probability p(U j |U i ), the transition probability from child node to parent node p(U i |U j ), where U j and U i Respectively indicate that the text data is marked as v j Node labels and are marked as ν o Node label, p(U j |U i ) is a given v i The case is marked as v j The conditional probability of the node label, P(U j ∩U i ) is {v j ,v i } is marked at the same time, N j and N i They represent the training set v j Node labels and ν i The number of node labels.

4. The hierarchical multi-label text classification method based on label attention according to claim 3, Features: In step S3, the word embedding module performs word embedding processing on the input text and its labels. The specific word embedding processing method is as follows: Step 2.1: Get the preprocessed text sequence and embed the words {x 1 ,x 2 ,...,x n } is converted to a word vector representation {w 1 ,w 2 ,...,w n }; Step 2.2: Obtain the label set for hierarchical multi-label text classification {l 1 ,l 2 ,...,l n }, through the Kaiming encoding method, the label set is converted into a label embedding set with a dimension of d {c 1 ,c 2 ,...,c n }, n refers to the number of words in the preprocessed text.

5. The hierarchical multi-label text classification method based on label attention according to claim 4, Features: In step S3, the word vector representation {w 1 ,w 2 ,...,w n } to perform encoding processing, the encoding processing method is specifically as follows: Use Bi-GRU network to represent the word vector of the text {w 1 ,w 2 ,...,w n } to generate an implicit representation {h 1 ,h 2 ,...,h n }, then the implicit representation {h 1 ,h 2 ,...,h m } is further fed into three convolutions with different kernel sizes, and three semantic vectors with different receptive fields are obtained. Finally, the three semantic vectors are concatenated into a new semantic representation vector S = {s 1 ,s 2 ,…,s m }; In step S3, the label vector representation {c 1 ,c 2 ,...,c n } for encoding processing, the label encoding processing method is as follows: Use a single-layer GCN to represent the label vector {c 1 ,c 2 ,...,c n } to generate an implicit representation M with label-level association information. 1 ,m 2 ,...,m n }, the implementation process is as follows: Hierarchical GCN aggregates data flows within top-down, bottom-up, and self-loop edges. In hierarchical GCN, each directed edge represents a pair of label-related features, and these data flows are transformed using linear transformations along the edges. In order to achieve node transformation, a weighted adjacency matrix is ​​used to represent this linear transformation, and the initial value of the weighted adjacency matrix comes from the prior level information of the hierarchical classification system in step S2. Formally, the hierarchical GCN encodes the hidden state of node k according to its related neighborhood, where neighborhood N(k) = {n k ,child(k),parent(k)},n k Refers to the kth label node in the hierarchical label tree, child(k) refers to the child label node of the kth node, parent(k) refers to the parent label node of the kth node, and the hidden state of node k is calculated as follows: In the above formula, v j and v k are trainable parameters. and is a trainable bias parameter; For u k,j and k,j For example, u k,j Understand the information between nodes k and j, g k,j Understand it as the gate value, control u k,j Finally, the impact on node k; σ refers to the activation function in deep learning, which can be taken as the sigmoid function. b l ∈R N×dim , and b g ∈R N , dim is the dimension of the vector, which is a predefined hyperparameter; d(j, k) represents the hierarchical direction from node j to node k, including top-down, bottom-up and self-loop edges; where a k,j ∈R represents the hierarchical probability f d(k,j) (e kj ), f d(k,j) (e kj ) refers to the transition probability from the kth node to the jth label node, which is obtained by f(e i,j ) to obtain the self-loop edge using a k,k = 1, top-down edges use The bottom-up edges use f p (e j,k )=1;The characteristic matrix F of the above edge={a 0,0 ,a 0,1 ,…,a c-1,c-1 } represents the weighted adjacency matrix of the text label directed hierarchical graph. Finally, the output hidden state h of node k k Indicates that it corresponds to a label representation of hierarchical information.

6. The hierarchical multi-label text classification method based on label attention according to claim 1, Features: In step S4, the cross entropy loss function is used in the training process, and the Adam optimizer is used for training. The cross entropy loss function for multi-label text classification is as follows: Among them, y ij is the actual probability of the i-th sample for the j-th label, is the predicted probability of the i-th sample for the j-th label, and finally the trained deep learning multi-label text classification model is obtained. L refers to the number of label categories, and N refers to the number of text samples.

Images