A Text Multi-Label Classification Method Based on a Knowledge-Enhanced Attention Network

Abstract

The present invention relates to the technical field of natural language processing. Specifically, it relates to a text multi-label classification method based on a knowledge-enhanced attention network. The text multi-label classification method based on the knowledge-enhanced attention network includes the following steps: Input text data: Obtain the text data to be processed; Data preprocessing: Clean and standardize the text data; Feature extraction: In order to retain the information in the text that is meaningful for the classification task while reducing redundancy, use the BERT pre-trained model to convert the text into a vector representation; Create a text classification model and construct a knowledge-enhanced attention mechanism model; Model training and verification. On the one hand, the present invention enhances the text semantic representation by introducing an external knowledge base. On the other hand, it uses an improved attention mechanism to improve the multi-label classification accuracy, and further combines the dependency relationship between labels to optimize the classification result. At the same time, the present invention has the technical effects of improving the classification performance and adapting to complex multi-label scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to a text multi-label classification method based on a knowledge-enhanced attention network. Background Art

[0002] With the advent of the big data era, multi-label classification tasks have been widely applied in fields such as text classification, recommendation systems, and sentiment analysis. However, existing methods face the following problems when dealing with multi-label classification:

[0003] 1. Insufficient capture of context semantics: Traditional methods mostly rely on simple word embeddings or shallow neural network models, making it difficult to fully mine the deep semantic information of the text.

[0004] 2. Inadequate modeling of label dependencies: There are often complex semantic associations between labels, and it is difficult for existing methods to effectively capture such relationships.

[0005] 3. Limited utilization of external knowledge: Most methods only rely on the text itself, ignoring the important role of external knowledge in enhancing semantic understanding.

[0006] Therefore, how to combine external knowledge, optimize semantic capture, and label dependency modeling has become an important research direction in the field of multi-label classification. Summary of the Invention

[0007] The purpose of the present invention is to provide a text multi-label classification method based on a knowledge-enhanced attention network to solve the problems of insufficient capture of text context semantics and inadequate modeling of label dependencies in existing text multi-label classification methods as mentioned in the above background art.

[0008] To achieve the above object, the present invention provides a text multi-label classification method based on a knowledge-enhanced attention network, including the following steps:

[0009] S1. Input text data: Obtain the text data to be processed, providing the basic input for subsequent steps;

[0010] S2. Data preprocessing: Clean and standardize the text data to improve the accuracy of subsequent analysis;

[0011] S3. Feature extraction: To retain the information meaningful for the classification task in the text while reducing redundancy, use the BERT pre-trained model to convert the text into a vector representation;

[0012] S4. Create a text classification model and construct a knowledge-enhanced attention mechanism model;

[0013] S5. Model training and validation;

[0014] S6, Result Output: Output the classification result and evaluate the model performance.

[0015] As a further improvement of this technical solution, the data preprocessing in step S2 specifically includes the following steps:

[0016] S21, Text Cleaning: Remove irrelevant characters (such as punctuation marks and special symbols) and stop words;

[0017] S22, Text Normalization: Convert case and perform lemmatization (such as stemming or part-of-speech reduction);

[0018] S23, Word Segmentation: Split the continuous text string into a sequence of meaningful words, using common word segmentation algorithms;

[0019] S24, Retrieve external knowledge based on entity linking technology, semantically expand the key entities in the text, enhance the richness of the text representation, and provide more abundant information for the input text data.

[0020] As a further improvement of this technical solution, the specific operation steps of step S24 are as follows:

[0021] S241, Text Entity Recognition: Extract key entities from the input text as the basis for associating with the knowledge base, and use a pre-trained model for entity recognition to extract entities ;

[0022] S242, Link the entities. For each recognized entity, crawl the text in its online knowledge base and select the one with the highest confidence. If there is more than one entity in this text, retrieve the text in their knowledge bases for each entity and combine them in the order of the entities in the input text. These combined texts are regarded as the external knowledge base of this text, align the extracted entities with the nodes in the external knowledge base, and use the entity linking algorithm to find the knowledge base node corresponding to the entity , the expression is as follows: ,

[0023] , where represents the external knowledge base; ]>

[0024] S243, Knowledge Expansion: Extract the relationships and neighbor nodes related to the entities from the external knowledge base to construct a knowledge subgraph , let be a certain entity node in the external knowledge base , and its neighbor node set is:

[0025] , then the knowledge expansion is expressed as:

[0026] ,in, represents the extracted knowledge subgraph, for The node's neighbor node set;

[0027] S244, knowledge representation embedding, using the knowledge graph embedding method to embed the expanded knowledge subgraph Convert to a low-dimensional embedding vector and obtain the entity set through the training process Knowledge Representation ;

[0028] S245, external knowledge enhanced text representation, the external knowledge representation is combined with the input text embedding to generate knowledge enhanced text representation, let the input text embedding be I, the external knowledge embedding be , the combination method can be weighted splicing: ,in, is an aggregation operation for embedding external knowledge, and is a hyperparameter used to balance the contribution of input text and external knowledge, and the final knowledge-enhanced text representation It can be directly used for the following multi-label classification tasks.

[0029] As a further improvement of the present technical solution, the feature extraction in step S3 specifically includes the following steps:

[0030] S31. Using BERT pre-trained language model to enhance text representation of knowledge To vectorize: ,

[0031] in, Indicates the final knowledge-enhanced text content of the input;

[0032] S32. Vectorized content As input, bidirectional LSTM is used for unified encoding to obtain a unified context representation of the input text content and the acquired external knowledge, and the output dimension is a 2H-dimensional vector. The unified representation of the text content and the acquired external knowledge are expressed as ,in and Represent the length of the input text content and the acquired external knowledge respectively. The specific encoding process is as follows: , h t Represents the state value at time t, t represents the current processing position in the sequence, and T represents the total number of time steps of the input sequence.

[0033] As a further improvement of this technical solution, the specific operation steps of step S4 are:

[0034] S41. Obtain the text feature representation related to the label by using the dot - product attention mechanism. The dot - product attention mechanism is as follows: ,

[0035] In the label - to - text attention mechanism, use the label embedding as the query vector , and the text embedding as the key vector and the value vector . Finally, the text embedding related to the label is as follows: , where , represents the transpose of; k is the dimension of the label embedding H, which is used to prevent the dot - product result from being too large and causing gradient instability;

[0036] S42. represents the text representation carrying label information, represents the label representation carrying text information. A fusion mechanism is used to combine these two parts of information. The fusion feature representation method is as follows: ,

[0037] where , represents the concatenation operation.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows: On the one hand, the present invention enhances the text semantic representation by introducing an external knowledge base. On the other hand, it uses an improved attention mechanism to improve the multi - label classification accuracy, further optimizes the classification result by combining the label - to - label dependency relationship. At the same time, the present invention has the technical effects of improving the classification performance and adapting to complex multi - label scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is the overall flowchart of the text multi - label classification method based on the knowledge - enhanced attention network of the present invention.

[0040] Figure 2 is the overall framework structure schematic diagram of the text multi - label classification method based on the knowledge - enhanced attention network of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0042] In a specific embodiment, such as Figure 1 shown, the present invention provides a text multi-label classification method based on a knowledge-enhanced attention network, and the method includes the following steps:

[0043] Step (1): Input text data: Obtain the text data to be processed, providing the basic input for subsequent steps.

[0044] Step (2): Data preprocessing: Clean and standardize the text data to improve the accuracy of subsequent analysis.

[0045] Step (2.1): Include text cleaning: Remove irrelevant characters (such as punctuation marks and special symbols) and stop words.

[0046] Step (2.2): Text normalization: Convert case, perform lemmatization (such as stemming or part-of-speech reduction).

[0047] Step (2.3): Word segmentation processing: Split the continuous text string into a sequence of meaningful words, using common word segmentation algorithms.

[0048] Step (2.4): External knowledge enhancement module, retrieve external knowledge based on entity linking technology, semantically expand the key entities in the text, enhance the richness of the text representation, and provide more abundant information for the input text data.

[0049] Step (2.4.1): Text entity recognition, extract the key entities from the input text as the basis for associating with the knowledge base. Use a pre-trained model for entity recognition and extract entities .

[0050] For example, for the input text , the result of entity recognition is:

[0051]

[0052] Among them, represents the input text, is the set of extracted entities.

[0053] Step (2.1.2): Link the entities. For each recognized entity, crawl the text in its online knowledge base (such as Baidu Encyclopedia, Wikidata) online and select the one with the highest confidence. If there are more than one entity in this text, retrieve the text in their knowledge bases for each entity and combine them in the order of the entities in the input text. These combined texts are regarded as the external knowledge base of this text. Align the extracted entities with the nodes in the external knowledge base, and use the entity linking algorithm to find the knowledge base node corresponding to the entity ​ , the expression is as follows:

[0054]

[0055] Among them, represents the external knowledge base.

[0056] Step (2.1.3): Knowledge expansion, extracting the relationships and neighbor nodes related to the entity from the external knowledge base to construct a knowledge subgraph , the extracted knowledge subgraph includes the entity and its related hyponymy, meronymy, and other relationships. Let be a certain entity node in the external knowledge base , and its neighbor node set is , then the knowledge expansion is expressed as:

[0057]

[0058] Among them, represents the extracted knowledge subgraph, is the neighbor node set of the node.

[0059] Step (2.1.4): Knowledge representation embedding, using knowledge graph embedding methods (such as TransE, DistMult, etc.) to convert the expanded knowledge subgraph into a low-dimensional embedding vector. Through the training process, the knowledge representation of the entity set is obtained. .

[0060] Step (2.1.5): External knowledge enhanced text representation, combining the external knowledge representation with the input text embedding to generate a knowledge-enhanced text representation.

[0061] Let the input text embedding be I, and the external knowledge embedding be , and the combination method can adopt weighted splicing:

[0062]

[0063] Among them, is the aggregation operation on the external knowledge embedding, and are hyperparameters used to balance the contributions of the input text and external knowledge. The final knowledge-enhanced text representation can be directly used for the following multi-label classification task.

[0064] Step (4): As Figure 2 shown, for the knowledge-enhanced text representation Perform feature extraction on the text and labels. To retain the information meaningful for the classification task in the text while reducing redundancy, use the BERT pre-trained model to convert the text into a vector representation.

[0065] Step (4.1): Use the BERT pre-trained language model to represent the knowledge-enhanced text for vectorization:

[0066]

[0067] where represents the input of the final knowledge-enhanced text content.

[0068] Step (4.2): Then, use the vectorized content as the input, and adopt a bidirectional LSTM for unified encoding to obtain a unified context representation of the input text content and the acquired external knowledge, and output a vector of dimension 2H. The unified representations of the text content and the acquired external knowledge are respectively represented as where and represent the lengths of the input text content and the acquired external knowledge respectively. The specific encoding process is as follows:

[0069]

[0070] Step (5): As Figure 2 shown, after feature extraction, construct a text classification model and construct a knowledge-enhanced attention mechanism model. Currently, methods regarding the attention mechanism often explore the fine-grained relationships between individual words in the text, but ignore the potential correlation between the label and the text. In fact, some words in the text have a stronger correlation with the label. To explore the hidden relationship between the text and the label and improve the classification performance of the model, the present invention uses a label-text bidirectional attention mechanism.

[0071] Step (5.1): The label-to-text attention mechanism mainly uses the dot product attention mechanism to obtain the text feature representation related to the label. This mechanism allows discarding the text content irrelevant to the label and only focusing on the text representation of the content related to the label, rather than the entire text content. This mechanism can improve the expression ability of the model in the multi-label classification task.

[0072] First, the dot product attention mechanism is as follows:

[0073]

[0074] In the label-to-text attention mechanism, use the label embedding as the query vector , and the text embedding As the key vector and the value vector . Finally, the text embedding related to the label is as follows:

[0075]

[0076] Wherein, , denotes the transpose of.

[0077] Step (5): As Figure 2 shown, after feature extraction, a text classification model is constructed, and a knowledge-enhanced attention mechanism model is constructed. Currently, methods regarding the attention mechanism often explore the fine-grained relationships between individual words in the text, but ignore the potential correlation between the label and the text. In fact, some words in the text have a stronger correlation with the label. In order to explore the hidden relationship between the text and the label and improve the classification performance of the model, therefore, the present invention uses a label-text bidirectional attention mechanism.

[0078] Step (5.1): The label-to-text attention mechanism mainly obtains the text feature representation related to the label by using the dot product attention mechanism. This mechanism allows discarding the text content irrelevant to the label and only focusing on the text representation of the content related to the label, rather than the entire text content. This mechanism can improve the expressive ability of the model in the multi-label classification task.

[0079] Step (5.2): The text-to-label attention mechanism. Since combining the original label embedding with the text features will also make the final features contain too many irrelevant labels, which will significantly affect the classification performance of the model. Using the text-to-label attention mechanism helps the model better learn the labels related to the text, and by utilizing these labels related to the text, the overall performance of the model is improved. We use the text embedding as the query vector , the label embedding as the key vector and the value vector . Use the dot product attention mechanism to calculate the label embedding related to the text. The label embedding related to the text is as follows:

[0080]

[0081] Wherein, , denotes the transpose of.

[0082] Step (5.3): The fusion mechanism. Represents the text representation carrying label information. Represents the label representation carrying text information. Focusing on the context of the text, a label-to-text attention mechanism is used to learn the relevant text representation for each label. Focusing on label information, a text-to-label attention mechanism is used to highlight the label representation related to the text. Therefore, in order to effectively utilize these two types of feature information, a fusion mechanism is adopted to combine these two parts of information. The fusion feature representation method is shown as follows:

[0083]

[0084] Where , Represents the concatenation operation.

[0085] Step (6): Model training and validation. Generally, the cross-entropy loss is usually used for this operation, and then the calculated loss is backpropagated to fine-tune the parameters of the network. However, this method cannot solve the problem of overfitting caused by the imbalance of samples in the class. In this case, the loss value of one class is much larger than that of the other class. Therefore, it is necessary to adjust the loss values of the two classes according to certain weights. Appropriately scaling the loss can reduce the influence of the class with more samples and promote better training of the model.

[0086] For the scaling of the loss, the following weight parameters can be used:

[0087]

[0088] In the above formula, Corresponds to the total number of negative samples, Is the number of positive samples, Is the scaling parameter. The generated weights and correspond to the assigned weights for negative and positive samples respectively. Using the values 1, 10, and 100, the influence of positive samples is increased by increasing the values of and respectively.

[0089] Therefore, the calculation of the weighted scale binary cross-entropy loss of the samples is as follows:

[0090]

[0091] Here k = {1, 2, 3, …, 𝐾}, where K is the total number of samples, Is the true label (0 or 1) of sample k. In addition, Is the predicted probability of sample k on class 0, and Is the probability of predicting sample k on class 1. So, the final loss function is:

[0092]

[0093] Among them, corresponds to the average value in all instances.

[0094] Step (7): Result output, output the classification result and evaluate the model performance. Considering the imbalance between classes, this method uses Micro-F1 to measure the performance of all methods. Micro-F1 is a metric that takes into account the overall precision and recall of all labels. Let , , respectively represent the true positive, false positive, and false negative of the t label in the label set 𝑆. Micro-F1 is defined as:

[0095]

[0096] Among them,

[0097]

[0098] .

[0099] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art of this industry should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only preferred examples of the present invention and are not used to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A text multi-label classification method based on a knowledge-enhanced attention network, characterized in that, It includes the following steps: S1. Input text data: Obtain the text data to be processed; S2. Data preprocessing: Clean and standardize the text data; Retrieve external knowledge based on entity linking technology and semantically expand the key entities in the text. The specific operation steps are as follows: S241. Text entity recognition: Extract key entities from the input text as the basis for association with the knowledge base. Use a pre-trained model for entity recognition to extract entities. ; S242. Link the entities. For each identified entity, crawl the text in its online knowledge base online and select the one with the highest confidence. If there are more than one entity in the text, retrieve the text in their knowledge bases for each entity and combine them in the entity order of the input text. These combined texts are regarded as the external knowledge base of the text. Align the extracted entities with the nodes in the external knowledge base, and use the entity linking algorithm to find the knowledge base node corresponding to the entity corresponding to , and the expression is as follows: , wherein, represents an external knowledge base; S243. Knowledge expansion: Extract the relationships and neighbor nodes related to the entity from the external knowledge base to construct a knowledge subgraph , assume is a certain entity node in the external knowledge base , and its neighbor node set is: , the knowledge expansion is expressed as: , where represents the extracted knowledge subgraph, is the set of neighbor nodes of the node; S244. Knowledge representation embedding. Using the knowledge graph embedding method, the extended knowledge subgraph is converted into a low-dimensional embedding vector. Through the training process, the knowledge representation of the entity set is obtained. ; S245. External knowledge enhanced text representation. Combine the external knowledge representation with the input text embedding to generate a knowledge-enhanced text representation. Let the input text embedding be I and the external knowledge embedding be . The combination method can adopt weighted splicing: , where is an aggregation operation on the external knowledge embedding, and are hyperparameters used to balance the contributions of the input text and external knowledge. The final knowledge-enhanced text representation can be directly used for the multi-label classification task described below; S3. Feature extraction: In order to retain the information meaningful for the classification task in the text and reduce redundancy at the same time, use the BERT pre-trained model to convert the text into a vector representation; S4. Create a text classification model and construct a knowledge-enhanced attention mechanism model; S5. Model training and validation; S6. Result output: Output the classification result and evaluate the model performance.

2. The text multi-label classification method based on the knowledge-enhanced attention network according to claim 1, characterized in that The specific data preprocessing in step S2 includes the following steps: S21. Text cleaning: Remove irrelevant characters and stop words; S22. Text normalization: Convert case and perform lemmatization; S23. Word segmentation: Split the continuous text string into a sequence of meaningful words using common word segmentation algorithms.

3. The text multi-label classification method based on the knowledge-enhanced attention network according to claim 1, characterized in that The specific feature extraction in step S3 includes the following steps: S31. Use the BERT pre-trained language model to represent the knowledge-enhanced text for vectorization: , Among them, represents the input of the final knowledge-enhanced text content; S32. Take the vectorized content as input, and use a bidirectional LSTM to uniformly encode it to obtain a unified context representation of the input text content and the acquired external knowledge, and output a 2H-dimensional vector. The unified representations of the text content and the acquired external knowledge are respectively denoted as , where and represent the lengths of the input text content and the acquired external knowledge respectively. The specific encoding process is as follows: 0 t denotes the state value at time t, where t represents the current position being processed in the sequence, and T represents the total number of time steps in the input sequence.

4. The text multi-label classification method based on a knowledge-enhanced attention network according to claim 1, wherein, The specific operation steps of step S4 are as follows: S41. Obtain the text feature representation related to the label by using the dot product attention mechanism, and the dot product attention mechanism is as follows: , In the label-to-text attention mechanism, label embeddings are used as query vectors , and text embeddings as key vectors and value vectors . Finally, the text embeddings related to the labels are as follows: , where , denotes the transpose of; k is the dimension of the label embedding H; S42、 Represents the text representation carrying label information, Represents the label representation carrying text information, and a fusion mechanism is used to combine these two parts of information. The fusion feature representation method is as follows: , Among them , represents a connection operation.

Images