A text classification method based on label semantic learning and attention adjustment mechanism

Abstract

The application discloses a text classification method based on label semantic learning and attention adjustment mechanism, and mainly comprises the following steps: preprocessing text data, extracting text semantic features, text label graph embedding, using a multi-head adjustment attention mechanism to measure the semantic relationship between words and labels, then multi semantic integration and network training, thereby realizing multi-label text classification, training the model, and then using the trained model to predict the category of a text. The application proposes a multi-head adjustment attention hybrid BERT model for a multi-label text classification framework, which can effectively extract useful features from text content, establish semantic connection between labels and words, obtain label-specific word representation, and thus improve the performance of multi-label text classification.

Description

TECHNICAL FIELD

[0001] The present application relates to a text multi-label classification method, and in particular to a text classification method based on label semantic learning and attention adjustment mechanism. BACKGROUND

[0002] The rapid development of the Internet and social media has made WeChat, Weibo and blogs a part of people's lives, conveying various emotions and emotional states in people's hearts, and recording the ups and downs of people's lives. A large amount of text data containing various information has accumulated on the network, and classifying these text data not only has important practical significance, but also has high commercial value. Therefore, how to effectively analyze these large-scale social text data to help people more effectively analyze the information they need has become a hot issue. On social media, simple keyword extraction cannot meet the needs of users. Because of the inherent characteristics of social media data, the results of feature word extraction often contain a lot of meaningless words, such as junk information or information that does not match the theme. According to surveys, about 40% of Twitter data contains junk information, and a text processing technology can solve this problem - text classification technology. Text classification technology can effectively distinguish junk information and can distinguish information according to different themes.

[0003] Text classification has always been an important technical means for summarizing and organizing text and mining text information. Traditional text classification models are mostly single-label text classification, but in recent years, as the types of network text have become more and more complex, the text categories have also become multi-label, such as an article can have both "computer" and "medical" labels, in which case the traditional single-label text classification model cannot cope with this classification task. Therefore, text multi-label classification has become one of the hotspots in the field of natural language processing. Multi-label classification algorithm assigns multiple labels to a sample, and is widely used in recommendation systems, public opinion analysis and sentiment classification, etc. There is usually a correlation between different labels of a sample, and how to learn the correlation between labels in the modeling process is a great challenge. In news text classification, the relationship between labels is hierarchical. At the same time, the relevance of each label to different parts of the news text is not the same, and the fusion of labels and text features needs to be modeled specifically.

[0004] At present, many research institutions take the text data in social media as the research object, and the purpose is to mine useful information from massive data, such as public behavior prediction, medical care monitoring, stock market analysis and prediction, etc. However, the text data in social media has the defects of short length, non-standard external format and content, and a large number of garbage information, which brings new challenges to the classification technology of text. In order to overcome these problems, more representative external format features are extracted on the basis of text classification technology, and deep learning model is applied to social media text semantic analysis and classification, which overcomes many defects of traditional manual extraction of semantic features, so the classification effect is better.

[0005] The task of multi-label text classification is to select the most relevant subset of labels from a given set of labels to the content of the text. After decades of development, there are already a large number of researches in the field of multi-label text classification, but there are still some problems that have not been solved well. For example, from the perspective of text data, different parts of the text are related to different labels to different degrees, for example, a news related to "economy" and "literature and art" at the same time, the first half of the news may be more inclined to economy, and the second half is more inclined to literature and art, and their length is inconsistent. However, few algorithms are specifically modeled in this direction. From the perspective of labels, there is a connection between labels, and there is a clear logical connection between the labels of multi-label classification. Therefore, the dependence relationship between the labels is modeled, the connection between the labels is fully utilized to improve the multi-label text classification effect, and the application value is high. SUMMARY

[0006] The application provides a multi-label text classification method based on label semantic learning and attention adjustment mechanism, based on the diversity of text labels, a new attention adjustment mechanism is designed for the semantic connection between labels and text, and a new label atlas construction method is proposed to obtain the internal structure and correlation of labels, so as to establish the semantic connection between labels and words, and combine the context environment perception features and label features to classify the text with multiple labels.

[0007] The application provides a text classification method based on label semantic learning and attention adjustment mechanism, characterized in that the text classification method comprises the following steps:

[0008] 1) data preprocessing, using WordPiece tokenizer to divide the text and make all characters lowercase, and expressing each text as a plurality of tokens; all characters of the label are made lowercase;

[0009] 2) Extracting text semantic features, encoding the multi-label text data respectively to obtain the corresponding feature representation, for the text in the data, using a word embedding module, embedding the original word into a low-dimensional vector, the semantic features of the text can be obtained; using the pre-trained BERT model to calculate the context-aware representation of each word, different embeddings are generated for words in different contexts; BERT accepts input in the form of a sequence and outputs the representation form of the sequence, obtaining the semantic feature representation of the final text;

[0010] 3) Constructing text label graph embedding, for each label corresponding to the text, using the label graph to reflect the label structure, regarding each label as a node, collecting the features of all neighbors to form its representation, each edge reflects the semantic correlation between nodes, if the labels coexist, there is an edge, finally obtaining a low-dimensional latent space representing the label graph;

[0011] 4) Using a multi-head attention adjustment mechanism to measure the semantic relationship between words and labels, obtaining word representations specific to labels, and weighting important fine-grained semantic information in the document;

[0012] 5) Multi-semantics integration and network training, using a bidirectional long short-term memory network (Bi-LSTM) as a document encoder, combining the mixed representation of context-aware features and label-specific word features into the document encoder for classification, and starting to train the entire model.

[0013] The data preprocessing step is specifically: word embedding is performed on the given text content, a word is split into words, the meaning of the word itself and the prefix and suffix are separated, so that the final word table becomes concise, obtaining an input document containing k words, denoted as [w1, w2,..., w i ,...,w k ], where w i represents the i-th word in the text.

[0014] The step of extracting text semantic features is specifically:

[0015] Let d be an input text containing k words, denoted as [w1, w2,..., w i ,...,w k ], where w i represents the i-th word in the text, input the text d into BERT to obtain the context representation of each input word H = [h1, h1,..., h k ].

[0016] The step of label graph embedding is specifically:

[0017] Define the label graph as g = (V, E), its nodes vi representative label λ i , edge (v i , v j ) ∈ E, A ∈ R C×C is the adjacency matrix between nodes, D ii = ∑ j A ij is the degree of the node, the adjacency matrix represents the non-negative weight between any two nodes, and this adjacency matrix is established in a data-driven manner; first, the label annotation of the sample in the training set is used to calculate the occurrence frequency of all label pairs, and the matrix C ∈ R C×C is obtained; by using this label co-occurrence matrix, the adjacency matrix A is obtained according to the following formula:

[0018]

[0019]

[0020] A = A + I

[0021] where C ij represents the simultaneous occurrence of labels λ i and λ j , and I is the unit matrix, indicating that each node is connected to itself; a word-label adjacency matrix B is constructed, and the construction method is as follows:

[0022]

[0023]

[0024] B = B + I

[0025] where B ij is the relationship between w i and λ j ; C ij of B represents the co-occurrence of w i and λ j in the sample; label embedding is determined by label co-occurrence graph and captures label semantic information defined by graph structure;

[0026] propagate information through graph neural networks, learn contextual label embeddings, and aggregate the values of all neighboring nodes to update the current node; each convolutional layer only processes first-order neighborhood information; represent labels in a low-dimensional latent space, so that two adjacent labels in the graph have similar representations; for each node v i ∈ V, initialize with a one-hot vector , and the label embedding can be represented as:

[0027]

[0028]

[0029] is a standardized symmetric adjacency matrix, Θ∈R C×C is a trainable weight, N(i) represents the number of neighbor nodes, ρ is a ReLU activation function, k is set to 2, and the label embedding set E = [e1, e2, …, e c ].

[0030] The step of adjusting the attention is specifically:

[0031] The word space is projected into the label space, and a fully connected layer φ is used to re-encode the word representation, and the formula is:

[0032] H * = φ(H)

[0033] H * ∈R K×C , the attention operation is used to calculate the attention score between the target word t and each label, and the dot product between I and E is calculated, and the formula is:

[0034]

[0035] The softmax function is used to regularize I t , and the formula is:

[0036] a t = softmax(I t ) (8)

[0037] The attention adjustment is divided into two stages; the task of the first stage is to judge the correlation between the word and the label, and the sigmoid function is used to judge; if the correlation score is less than the threshold τ, it is considered that the word is irrelevant to these labels;

[0038] In the second stage, the attention score is calculated by using the above softmax to normalize the probability distribution; the operation is as follows:

[0039] a t = sigmoid(I t )

[0040]

[0041]

[0042] Then, the label embedding amount of the word t is weighted and averaged by using the adjusted attention amount, and the formula is:

[0043]

[0044] where, is the word representation of a specific label, and indicates that different labels have distinguishable intrinsic characteristics; finally, the label-specific word sequence can be represented as

[0045] The steps of the multi-semantics integration fusion and network training are specifically:

[0046] The embedded H and H l By level joint, the information of two aspects is aggregated, and the formula is as follows:

[0047]

[0048] where is the final mixed word embedding, which is then provided as input to the document encoder.

[0049] The document encoder is a bidirectional long short-term memory network (Bi-LSTM); the Bi-LSTM learns the word embedding of each input text by forward and reverse learning, and at time t, the hidden state can be represented as:

[0050]

[0051]

[0052]

[0053] The final hidden state h k is used to represent the entire document; h k is input to the classifier to predict the confidence score of each label in the document; the classifier consists of a fully connected layer and a sigmoid function, and the formula is as follows:

[0054]

[0055] where is the trainable parameter of the fully connected layer; D is the dimension of the word vector.

[0056] The binary classification loss is used as the loss function of the multi-label text classification task, and the formula is as follows:

[0057]

[0058] In addition, the embedding of the label graph is further limited, so that similar labels are closer in the label semantic space, and non-adjacent labels are mutually exclusive. The loss of the label graph embedding can be represented as:

[0059]

[0060] As described above, the label embedding module and the attention module are regarded as a clustering process, which requires the special word representation of the label to be closer to the center of its class; one method to encode this property is to make the cosine similarity Φ(e i ,e j ) close to the edge weight A ij of all i,j pairs, therefore, another loss function is designed to measure the result of clustering, which can be expressed as:

[0061]

[0062] Finally, the loss function is defined as:

[0063] L=L g +L c +L e .

[0064] The model is trained by backpropagation using the loss function described above.

[0065] The beneficial effects of the technical solutions provided by the application are:

[0066] 1. The technical solution of the application can classify text data composed of multiple types of labels, while traditional text classification methods cannot meet multi-label classification. The classification method proposed by the application can effectively extract the features of the semantics of the text itself and the text labels, thereby improving the performance of multi-label classification of text.

[0067] 2. The application designs a new attention adjustment mechanism to measure the semantic relationship between the text and the label. The attention adjustment value learned by the attention adjustment mechanism from the text and the label is a further reinforcement representation and learning of important fine-grained semantic information in the document to capture useful label-specific information and suppress noise, thereby improving the accurate classification of multi-label text.

[0068] 3. The application makes full use of the amount of data to obtain accurate information gain estimation and improve the performance of the classification model by utilizing the semantic relationship between the label and the text. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a flowchart of a multi-label text classification method based on label semantic learning and attention adjustment mechanism;

[0070] Figure 2 is a detailed description diagram of a multi-label text classification method based on label semantic learning and attention adjustment mechanism;

[0071] Figure 3 is a flowchart of extracting text semantic features using BERT model and Bi-LSTM. DETAILED DESCRIPTION

[0072] The technical solutions of the present application will be described in detail below in conjunction with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and are not intended to limit the present application. It should be noted that in the drawings or description, similar or identical parts are denoted by the same reference numerals. The implementation not shown or described in the drawings is known to those skilled in the art. In addition, some parameters are given a value range, and some parameters are given an exact value, but it should be understood that these exact values can be approximated to the corresponding values within the acceptable error tolerance or design constraints.

[0073] Example 1

[0074] Multi-label text classification is a basic but challenging task in natural language processing. It has been widely used in the fields of sentiment analysis, intent recognition, recommendation system, etc. Previous studies usually treat labels as symbols without semantics, ignoring the relationship between labels, resulting in the loss of information. Since modeling the semantics of labels can improve the efficiency of multi-label text classification, the present application proposes a multi-label text classification method based on label semantic learning and attention adjustment mechanism, which simultaneously utilizes label semantics and fine-grained text information. The method includes the following steps:

[0075] 101: Data preprocessing, i.e. for any text and its labels, using WordPiece tokenizer to tokenize the text and making all characters lowercase, while representing each text as a number of tokens. This step aims to better represent the content of the text.

[0076] 102: Extracting text semantic features: For the text in the data, first use the word embedding module to embed the original words into low-dimensional vectors, which can obtain the text semantic features. However, the challenge of polysemy makes it difficult for context-independent word embeddings to perform well in classification tasks. Therefore, the present application uses a pre-trained BERT model to calculate the context-aware representation of each word. This model is based on a multi-layer bidirectional Transformer model, which generates different embedded words for words in different contexts. BERT accepts input in the form of a sequence and outputs the representation of the sequence, obtaining the semantic feature representation of the final text.

[0077] 103: Constructing text label graph embedding: For each label corresponding to the text, use the label graph to reflect the label structure, which is a flexible method to capture the topology in the label space. The specific method is to regard each label as a node, and collect the features of all neighbors to form its representation. Each edge reflects the semantic correlation between nodes. If the labels coexist, there is an edge, and finally a low-dimensional latent space is obtained to represent the label graph.

[0078] 104: Multi-head attention adjustment mechanism to measure the semantic relationship between words and labels: a multi-head adjustment attention mechanism is used to measure the semantic relationship between words and labels, obtain label-specific word representations, and weight important fine-grained semantic information in the document.

[0079] 105: The steps of multi-semantics integration fusion and network training are as follows: a bidirectional long short-term memory network (Bi-LSTM) is used as a document encoder, then the mixed representation combining the context-aware features and the label-specific lexical features is sent to the document encoder for classification, the entire model is trained, and multi-label text classification is realized.

[0080] Embodiment 2

[0081] The scheme in Embodiment 1 will be described in detail below in combination with specific calculation formulas and examples, as described below:

[0082] 201: Data preprocessing: given a text content, the text content needs to be word embedded. Since there are many changes in time, singular and plural, etc. in words, the word table will be very large, which will affect the training speed, and even a very large word table cannot handle out-of-vocabulary words. Therefore, a word needs to be split into characters, separating the meaning of the word itself and the prefix and suffix, so that the final word table becomes concise, and an input document containing k words is obtained, denoted as [w1, w2, …, w i ,...,w k ], where w i represents the i-th word in the text.

[0083] 202: Text semantic feature extraction: let d be an input text containing k words, denoted as [w1, w2, …, w i ,...,w k ], where w i represents the i-th word in the text. The text d is input into BERT to obtain the context representation H = [h1, h1, …, h k ] of each input word.

[0084] 203: The steps of label graph embedding are as follows: define the label graph as g = (V, E), where the node v i represents the label λ i , the edge (v i , v j ) ∈ E, A ∈ R C×C is the adjacency matrix between nodes, and D ii = ∑ j A ij is the degree of the node. The adjacency matrix represents the non-negative weight between any two nodes. This adjacency matrix is established in a data-driven way. First, the occurrence of all label pairs is calculated by using the label annotations of samples in the training set, and the matrix C∈R C×C By using this label co-occurrence matrix, the adjacency matrix A can be obtained according to the following formula:

[0085]

[0086]

[0087] A = A + I

[0088] where C ij represents the co-occurrence of labels λ i and λ j , and I is the identity matrix, meaning that each node is connected to itself. A word-label adjacency matrix B is constructed, and the construction method is as follows:

[0089]

[0090]

[0091] B = B + I

[0092] where B ij is the relationship between w i and λ j . The C ij of B represents the co-occurrence of w i and λ j in the sample. The label embedding is determined by the label co-occurrence graph and captures the label semantic information defined by the graph structure.

[0093] GCN is introduced to propagate information through the graph neural network to learn the contextual label embedding. GCN aggregates the values of all adjacent nodes to update the current node. Each convolutional layer only processes first-order neighborhood information. By stacking multiple convolutional layers, multi-order neighborhood information can be obtained. The goal is to represent labels in a low-dimensional latent space, so that two adjacent labels in the graph have similar representations, while non-adjacent nodes are mutually exclusive. For each node v i ∈V, a one-hot vector is initialized. Then, the label embedding can be represented as:

[0094]

[0095]

[0096] is the normalized symmetric adjacency matrix, and Θ∈R C×Cis a trainable weight, N(i) denotes the number of neighbor nodes, and ρ is the ReLU activation function, the present application considers two layers of GCN for label embedding, which means that k is set to 2, and then the label embedding set E = [e1, e2, …, eK] can be realized. c ]。

[0097] 204: Multi-head attention adjustment mechanism: In order to make the semantic relationship between the labels and the word-level information of the document be explicitly modeled, the guidance of the fine-grained document information that provides the model with explicit classification clues. The present application proposes an innovative attention mechanism, a multi-head attention adjustment mechanism, which explicitly calculates the semantic relationship between words and labels, and the adjustment attention learned from word labels is to weight the important fine-grained semantic information in the document.

[0098] First, the word space is projected into the label space. A fully connected layer φ is used to re-encode the word representation, and the formula is:

[0099] H * = φ(H)

[0100] H * ∈ R K×C , the attention score between the target word t and each label is calculated by using the attention operation. A simple method is to calculate the dot product between and E, and the formula is:

[0101]

[0102] The softmax function is used to regularize I t , and the formula is:

[0103] a t = softmax(I t )

[0104] For those documents with fewer labels, other labels can be considered as redundant information, and in this case, filtering out unnecessary information plays a relatively important role. In order to refine the classification clues and reduce the irrelevance and redundancy of the document content, the present application proposes an attention adjustment mechanism based on multi-head attention. The model dynamically allocates the weight of the label to the word by adjusting the attention.

[0105] Since the degree of association between word tokens and category labels will affect their attention scores, adjusting the attention can be divided into two stages. The task of the first stage is to judge the relevance between words and labels, and this task is regarded as a binary classification task, so the sigmoid function is used. If some relevance scores are less than the threshold τ, it is considered that the word is irrelevant to these labels.

[0106] ​In the second stage, the attention score is calculated using the softmax to normalize the probability distribution. Thus, the weight of irrelevant labels is reduced and the weight of relevant labels is increased.

[0107] The overall operation is shown in the following formula:

[0108] a t = sigmoid(I t )

[0109]

[0110]

[0111] Then, the label embedding of word t is weighted and averaged using the adjusted attention amount.

[0112]

[0113] where, is the word representation of a specific label, which considers that different labels have distinguishable intrinsic characteristics. Finally, the label-specific word sequence can be represented as

[0114] The label graph embedding module encodes the label graph through label embedding. The combination of the attention module and the label graph embedding module can be regarded as a process of clustering and aggregation. The purpose is to learn the prototype representation of each class, and then generate a label-specific word representation based on it, which will aggregate the label semantics.

[0115] 205: Multi-Semantic Integration Fusion: After the above steps, two forms of word representation can be obtained: H and H l The former focuses on the meaning of words in context, while the latter focuses on the semantic relationship between words and labels. For simplicity, the embedded H and H l Aggregate information from both aspects through level joint. The formula is as follows:

[0116]

[0117] where is the final hybrid word embedding, which is then provided as input to the document encoder.

[0118] 206: Network Training: Use a bidirectional long short-term memory network (Bi-LSTM) as a document encoder to generate document representations. Bi-LSTM can learn word embeddings for each input text through forward and backward learning. At time t, the hidden state can be represented as:

[0119]

[0120]

[0121]

[0122] Use the final hidden state h k This represents the entire document. Finally, input h into the classifier. k To predict the confidence score for each tag in a document. The classifier consists of a fully connected layer and a sigmoid function:

[0123]

[0124] in These are the trainable parameters of the fully connected layer. D is the dimension of the word vector.

[0125] Similar to previous studies, binary classification loss is used as the loss function for the MLTC task, and its formula is:

[0126]

[0127] Furthermore, restricting the embedding of label graphs ensures that similar labels are closer in the label semantic space, rather than adjacent labels being mutually exclusive. The loss of label graph embedding can be expressed as:

[0128]

[0129] As mentioned above, viewing the label embedding module and attention module as a clustering process requires that the word representations specific to the label be closer to the center of its category. One way to encode this property is to make the cosine similarity Φ(e i ,e j Approximates the edge weights A corresponding to all i,j. ij Therefore, another loss function was designed to measure the clustering results, which can be expressed as:

[0130]

[0131] Finally, the loss function is defined as:

[0132] L = L g +L c +L e

[0133] The model is trained using backpropagation of the loss described above.

[0134] To sum up, the multi-label text classification method is realized by the steps 201-206, and the label semantics and fine-grained text information are utilized. The pre-trained BERT model is used to calculate the context-aware representation of the document, the label graph is modeled based on the adjacent similarity, then the GCN is used to encode the label graph, the structural information and rich semantic association between labels are captured, the multi-head adjustment attention mechanism is used, the semantic relationship between words and labels is calculated, the important fine-grained semantic information in the document is weighted, and the feature concatenator is used to fuse the features of the words in the context and the features between the words and the labels. Thus, the multi-label text classification can be realized. Based on the above, the model is trained, and then the trained model is used to predict the category of a text. The multi-head adjustment attention hybrid BERT model is proposed for the multi-label text classification framework, which can effectively extract useful features from the text content, establish the semantic relationship between the labels and the words, obtain the word representation specific to the label, and thus improve the performance of the multi-label text classification.

[0135] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the protection scope of the present application. It should be understood by those skilled in the art that any modification or equivalent replacement of the technical solutions of the present application shall fall within the essential protection scope of the technical solutions of the present application.

Claims

1. A text classification method based on label semantic learning and attention adjustment mechanism, the text being multi-label text, characterized in that, The text classification method comprises the following steps: 1) data preprocessing, using WordPiece tokenizer to segment the text and making all characters lowercase, and representing each text as a plurality of tokens; making all characters of the label lowercase; 2) extracting text semantic features, encoding the multi-label text data respectively to obtain corresponding feature representations, for the text in the data, using a word embedding module to embed the original word into a low-dimensional vector, the semantic features of the text can be obtained; using a pre-trained BERT model to calculate the context-aware representation of each word, different embeddings are generated for words in different contexts; BERT accepts input in the form of a sequence and outputs the representation of the sequence, obtaining the semantic feature representation of the final text; 3) text label graph embedding, for each label corresponding to a text, the label graph is used to reflect the label structure, each label is regarded as a node, and the features of all neighbors are collected to form the representation of each node, and each edge reflects the semantic correlation between nodes, if the labels coexist, there is an edge, and finally a low-dimensional latent space is obtained to represent the label graph; 4) using a multi-head attention adjustment mechanism to measure the semantic relationship between words and labels, obtaining word representations specific to labels, and weighting important fine-grained semantic information in the document; 5) multi-semantics integration and network training, using a bidirectional long short-term memory network (Bi-LSTM) as a document encoder, combining the mixed representation of context-aware features and label-specific word features into the document encoder for classification, starting training the entire model, and realizing multi-label text classification.

2. The text classification method based on label semantic learning and attention adjustment mechanism according to claim 1, characterized in that, The data preprocessing step specifically comprises: For a given text content, word embeddings are performed, a sentence is divided into several words, and words are split into subwords, separating the meaning of the word itself from prefixes and suffixes, so that the final vocabulary becomes compact, obtaining an input document containing k words, denoted as [w1, w2,..., w i ,...,w k ], where w i is the i-th word in the text.

3. The text classification method based on label semantic learning and attention adjustment mechanism according to claim 1 or 2, characterized in that, The text semantic feature extraction step specifically comprises: Let d be an input text containing k words, denoted as [w1, w2, ..., wk]. i ,...,w k ], where w i Let d be the i-th word in the text. Input the text d into BERT to obtain the context representation H = [h1, h2, ..., hn] for each input word. k ].

4. The text classification method based on label semantic learning and attention adjustment mechanism according to claim 1 or 2, characterized in that, The label graph embedding step specifically comprises: Definition of label graph as g = (V, E), whose nodes v i represent labels λ i , edges (v i , v j ) ∈ E, A ∈ R C×C is the adjacency matrix between nodes, D ii = ∑ j A ij is the degree of the node, and the adjacency matrix represents the non-negative weight between any two nodes, and this adjacency matrix is established in a data-driven manner; first, the occurrence times of all label pairs are calculated by using the label annotations of samples in the training set, obtaining matrix C ∈ R C×C , and the adjacency matrix A is obtained by using this label co-occurrence matrix according to the following formula: A = A + I where C ij represents the label λ i and λ j appear simultaneously, I is the identity matrix, which means that each node is connected to itself; a word-label adjacency matrix B is also constructed as follows: B = B + I where B ij is w i and λ j ; B's C ij represents the co-occurrence of w i and λ j in the sample, the label embeddings are determined by the label co-occurrence graph and capture the label semantic information defined by the graph structure; By propagating information through the graph neural network, learning the contextual label embedding, aggregating the values of all neighboring nodes to update the current node; each convolutional layer only processes first-order neighborhood information; representing labels in a low-dimensional latent space so that two adjacent labels in the graph have similar representations; for each node v i ∈V, a one-hot vector Initialization, the label embedding can be represented as: is a standardized symmetric adjacency matrix, Θ ∈ R C×C is a trainable weight, N(i) represents the number of neighbor nodes, ρ is the ReLU activation function, k is set to 2, and the label embedding set E = [e1, e2, …, e c ].

5. The text classification method based on label semantic learning and attention adjustment mechanism according to any one of claims 1-2, characterized in that, The multi-head attention adjustment mechanism specifically comprises: Project the word space into the label space using a fully connected layer φ to re-encode the word representation, and the formula is: H * = φ(H) H * ∈R K×C , the attention operation is adopted to calculate the attention score between the target word t and each label, that is, the dot product between t and E is calculated, and the formula is: and E, and the formula is: The softmax function is used to regularize I t with the formula: a t = softmax(I t ) The attention adjustment is divided into two stages; the task of the first stage is to judge the correlation between words and labels, and a sigmoid function is used for judgment; if the correlation score is less than the threshold τ, it is considered that the word is irrelevant to the labels; In the second stage, the attention score is calculated using the softmax to normalize the probability distribution; the operation is as shown in the following formula: a t = sigmoid(I t ) Then, the label embedding amount of the word t is weighted and averaged using the adjusted attention amount, and the formula is: wherein, is a word representation of a particular tag, indicating that different tags have intrinsic characteristics that can be distinguished; finally, a sequence of words specific to a tag can be represented as 6. The text classification method based on label semantic learning and attention adjustment mechanism according to claim 5, characterized in that, The multi-semantics integration and network training step specifically comprises: The embedded H and H l By level-join, the two aspects of information are aggregated, and the formula is shown as follows: wherein is the final mixed word embedding, which is then provided as input to the document encoder.

7. The text classification method based on label semantic learning and attention adjustment mechanism according to claim 5, characterized in that, The document encoder is a bidirectional long short-term memory network (Bi-LSTM); the Bi-LSTM learns the word embedding of each input text through forward and backward learning, and at time t, the hidden state can be represented as: Using the final hidden state h k to represent the entire document; inputting h k to a classifier to predict the confidence score of each label in the document; the classifier consists of a fully connected layer and a sigmoid function, as shown in the following formula: wherein are trainable parameters of the fully connected layer; D is the dimension of the word vector.

8. The text classification method based on label semantic learning and attention adjustment mechanism according to claim 7, characterized in that, The binary classification loss is used as the loss function of the multi-label text classification task, and the formula is: The loss of the label graph embedding can be represented as: The label embedding module and the attention module are regarded as a clustering process, which requires the special word representation of the label to be closer to the center of its class; one method to encode this property is to make the cosine similarity Φ(e i ,e j ) close to the edge weight A ij of all i, j pairs, and a loss function is designed to measure the clustering results, which is represented as: Finally, the loss function is defined as: L = L g + L c + L e The model is trained using the above loss backpropagation.

Images