Text multi-label classification method based on knowledge-enhanced attention network

Abstract

The invention relates to the technical field of natural language processing, in particular to a text multi-label classification method based on a knowledge-enhanced attention network. The text multi-label classification method based on the knowledge-enhanced attention network comprises the following steps: inputting text data; acquiring to-be-processed text data; data preprocessing: cleaning and standardizing text data; feature extraction: in order to reserve information meaningful to a classification task in the text and reduce redundancy, converting the text into vector representation by using a BERT pre-training model; creating a text classification model, and constructing an attention mechanism model based on knowledge enhancement; and training and verifying the model. According to the method, on one hand, text semantic representation is enhanced by introducing an external knowledge base, on the other hand, the multi-label classification precision is improved by utilizing an improved attention mechanism, the classification result is further optimized in combination with the dependency relationship between the labels, and meanwhile, the method has the technical effects of improving the classification performance and adapting to a complex multi-label scene.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a text multi-label classification method based on a knowledge enhanced attention network. Background Art

[0002] With the advent of the big data era, multi-label classification tasks have been widely used in fields such as text classification, recommendation systems, and sentiment analysis. However, existing methods face the following problems when dealing with multi-label classification: 1. Insufficient contextual semantics capture: Traditional methods mostly rely on simple word embedding or shallow neural network models, which makes it difficult to fully mine the deep semantic information of the text.

[0003] 2. Insufficient label dependency modeling: There are often complex semantic associations between labels, and existing methods find it difficult to effectively capture such relationships.

[0004] 3. Limited utilization of external knowledge: Most methods rely only on the text itself, ignoring the important role of external knowledge in enhancing semantic understanding.

[0005] Therefore, how to combine external knowledge, optimize semantic capture and label dependency modeling has become an important research direction in the field of multi-label classification. Summary of the invention

[0006] The purpose of the present invention is to provide a text multi-label classification method based on knowledge enhanced attention network to solve the problems of insufficient capture of text context semantics and insufficient label dependency modeling in the existing text multi-label classification methods proposed in the above background technology.

[0007] To achieve the above object, the present invention provides a text multi-label classification method based on knowledge enhanced attention network, comprising the following steps: S1. Input text data: obtain the text data to be processed and provide basic input for subsequent steps; S2. Data preprocessing: Clean and standardize text data to improve the accuracy of subsequent analysis; S3. Feature extraction: In order to retain the information in the text that is meaningful to the classification task and reduce redundancy, the BERT pre-trained model is used to convert the text into a vector representation; S4. Create a text classification model and build an attention mechanism model based on knowledge enhancement; S5, model training and verification; S6. Result output: Output classification results and evaluate model performance.

[0008] As a further improvement of the technical solution, the data preprocessing in step S2 specifically includes the following steps: S21, text cleaning: remove irrelevant characters (such as punctuation, special symbols) and stop words; S22, text normalization: case conversion and word form restoration (such as stemming or part of speech restoration); S23, word segmentation: split the continuous text string into meaningful word sequences using a commonly used word segmentation algorithm; S24. Retrieve external knowledge based on entity linking technology, perform semantic expansion on key entities in the text, enhance the richness of text representation, and provide richer information for input text data.

[0009] As a further improvement of the technical solution, the specific operation steps of step S24 are: S241, Text Entity Recognition, extract key entities from the input text as the basis for associating with the knowledge base, use the pre-trained model for entity recognition, extract entities ; S242, linking entities. For each identified entity, crawl the text in its online knowledge base online and select the one with the highest confidence. If there is more than one entity in the text, retrieve the text in their knowledge base for each entity and combine them in the order of the entities in the input text. These combined texts are regarded as the external knowledge base of the text. Align the extracted entities with the nodes in the external knowledge base and use the entity linking algorithm to find the entities that match the entity. Corresponding knowledge base node , the expression is as follows: ,in, Represents an external knowledge base; S243, knowledge expansion, extracting entity-related relationships and neighbor nodes from external knowledge bases to build knowledge subgraphs ,set up Is an external knowledge base For a certain entity node in , its neighbor node set is: , then the knowledge expansion is expressed as: ,in, represents the extracted knowledge subgraph, for The node's neighbor node set; S244, knowledge representation embedding, using the knowledge graph embedding method to embed the expanded knowledge subgraph Convert to a low-dimensional embedding vector and obtain the entity set through the training process Knowledge Representation ; S245, external knowledge enhanced text representation, combining the external knowledge representation with the input text embedding to generate knowledge enhanced text representation, assuming that the input text embedding is I and the external knowledge embedding is , the combination method can be weighted splicing: ,in, is an aggregation operation for embedding external knowledge, and is a hyperparameter used to balance the contribution of input text and external knowledge, and the final knowledge-enhanced text representation It can be directly used for the following multi-label classification tasks.

[0010] As a further improvement of the present technical solution, the feature extraction in step S3 specifically includes the following steps: S31. Using BERT pre-trained language model to enhance text representation To vectorize: , in, Indicates the final knowledge-enhanced text content of the input; S32. Vectorized content As input, bidirectional LSTM is used for unified encoding to obtain a unified contextual representation of the input text content and the acquired external knowledge, and the output dimension is a 2H-dimensional vector. The unified representation of the text content and the acquired external knowledge are expressed as ,in and Respectively represent the length of the input text content and the acquired external knowledge. The specific encoding process is as follows: ,h t Represents the state value at time t, t represents the current processing position in the sequence, and T represents the total number of time steps of the input sequence.

[0011] As a further improvement of the technical solution, the specific operation steps of step S4 are: S41. Obtain the text feature representation related to the label by using the dot product attention mechanism. The dot product attention mechanism is as follows: , In the label-to-text attention mechanism, label embedding is used As a query vector , and text embedding As a key vector Sum value vector ,Finally, the text embedding associated with the tag as follows: ,in, , express The transpose of ; k is the dimension of the label embedding H, which is used to prevent the dot product result from being too large and causing gradient instability; S42, Represents a text representation that carries tag information. Represents the label representation that carries text information. A fusion mechanism is used to combine these two parts of information. The fusion feature representation method is as follows: , in , Represents a join operation.

[0012] Compared with the prior art, the present invention has the following advantages: on the one hand, the present invention enhances the semantic representation of text by introducing an external knowledge base, and on the other hand, utilizes an improved attention mechanism to improve the accuracy of multi-label classification, and further optimizes the classification results by combining the dependency between labels. At the same time, the present invention has the technical effect of improving classification performance and adapting to complex multi-label scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is an overall flow chart of the text multi-label classification method based on knowledge enhanced attention network of the present invention.

[0014] Figure 2 Schematic diagram of the overall framework structure of the text multi-label classification method based on knowledge enhanced attention network of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] In a specific embodiment, Figure 1 As shown, the present invention provides a text multi-label classification method based on knowledge enhanced attention network, which includes the following steps: Step (1): Input text data: Obtain the text data to be processed to provide basic input for subsequent steps.

[0017] Step (2): Data preprocessing: Clean and standardize text data to improve the accuracy of subsequent analysis.

[0018] Step (2.1): includes text cleaning: removing irrelevant characters (such as punctuation, special symbols) and stop words.

[0019] Step (2.2): Text normalization: case conversion, word form restoration (such as stemming or part of speech restoration).

[0020] Step (2.3): Word segmentation: Split the continuous text string into meaningful word sequences using a commonly used word segmentation algorithm.

[0021] Step (2.4): External knowledge enhancement module retrieves external knowledge based on entity linking technology, semantically expands key entities in the text, improves the richness of text representation, and provides richer information for input text data.

[0022] Step (2.4.1): Text entity recognition, extract key entities from the input text as the basis for associating with the knowledge base. The model performs entity recognition and extracts entities .

[0023] For example, for the input text , the result of entity recognition is:

[0024] in, Represents input text, is the collection of entities extracted.

[0025] Step (2.1.2): Link entities. For each identified entity, crawl the text in its online knowledge base (such as Baidu Encyclopedia, Wikidata) and select the one with the highest confidence. If there is more than one entity in the text, retrieve the text in their knowledge base for each entity and combine them in the order of the entities in the input text. These combined texts are regarded as the external knowledge base of the text. Align the extracted entities with the nodes in the external knowledge base and use the entity linking algorithm to find the entities that match the entities. Corresponding knowledge base node , the expression is as follows:

[0026] in, Represents an external knowledge base.

[0027] Step (2.1.3): Knowledge expansion, extracting entity-related relationships and neighbor nodes from the external knowledge base and constructing a knowledge subgraph , the extracted knowledge subgraph It includes entities and their related hierarchical relationships, attribute relationships, etc. Is an external knowledge base For a certain entity node in , its neighbor node set is , then the knowledge expansion is expressed as:

[0028] in, represents the extracted knowledge subgraph, for The set of neighbor nodes of a node.

[0029] Step (2.1.4): Knowledge representation embedding, using knowledge graph embedding methods (such as TransE, DistMult, etc.) to embed the expanded knowledge subgraph Converted into a low-dimensional embedding vector. Through the training process, the entity set is obtained Knowledge Representation .

[0030] Step (2.1.5): External knowledge enhances text representation, combining the external knowledge representation with the input text embedding to generate knowledge-enhanced text representation.

[0031] Let the input text embedding be I and the external knowledge embedding be , the combination method can be weighted splicing:

[0032] in, is an aggregation operation for embedding external knowledge, and is a hyperparameter used to balance the contribution of input text and external knowledge. The final knowledge-enhanced text representation It can be directly used for the following multi-label classification tasks.

[0033] Step (4): If Figure 2 As shown, knowledge-enhanced text representation In order to retain the meaningful information in the text for the classification task and reduce redundancy, the BERT pre-trained model is used to convert the text into a vector representation.

[0034] Step (4.1): Use the BERT pre-trained language model to enhance the text representation of knowledge To vectorize:

[0035] in, Represents the final knowledge-enhanced text content of the input.

[0036] Step (4.2): Next, vectorize the content As input, a bidirectional LSTM is used for unified encoding to obtain a unified contextual representation of the input text content and the acquired external knowledge, and the output dimension is a 2H-dimensional vector. The unified representation of the text content and the acquired external knowledge are respectively expressed as ,in and Respectively represent the length of the input text content and the acquired external knowledge. The specific encoding process is as follows:

[0037] Step (5): If Figure 2 As shown, after feature extraction, a text classification model is constructed, and an attention mechanism model based on knowledge enhancement is constructed. At present, the methods of attention mechanism often explore the fine-grained relationship between individual words in the text, but ignore the potential correlation between the label and the text. In fact, some words in the text are more correlated with the label. In order to explore the hidden relationship between text and label and improve the classification performance of the model, the present invention uses a label-text bidirectional attention mechanism.

[0038] Step (5.1): Label-to-text attention mechanism, mainly by using the dot product attention mechanism to obtain the text feature representation related to the label, this mechanism allows to discard the text content irrelevant to the label and only focus on the text representation of the content related to the label, rather than the entire text content. This mechanism can improve the expressiveness of the model in multi-label classification tasks.

[0039] First, the dot product attention mechanism is as follows:

[0040] In the label-to-text attention mechanism, label embedding is used As a query vector , and text embedding As a key vector Sum value vector Finally, the text embedding associated with the tag as follows:

[0041] in, , express The transpose of .

[0042] Step (5): If Figure 2 As shown, after feature extraction, a text classification model is constructed, and an attention mechanism model based on knowledge enhancement is constructed. At present, the methods of attention mechanism often explore the fine-grained relationship between individual words in the text, but ignore the potential correlation between the label and the text. In fact, some words in the text are more correlated with the label. In order to explore the hidden relationship between text and label and improve the classification performance of the model, the present invention uses a label-text bidirectional attention mechanism.

[0043] Step (5.1): Label-to-text attention mechanism, mainly by using the dot product attention mechanism to obtain the text feature representation related to the label, this mechanism allows to discard the text content irrelevant to the label and only focus on the text representation of the content related to the label, rather than the entire text content. This mechanism can improve the expressiveness of the model in multi-label classification tasks.

[0044] Step (5.2): Text-to-label attention mechanism. Since combining the original label embedding with the text feature will cause the final feature to contain too many irrelevant labels, this will significantly affect the classification performance of the model. Using the text-to-label attention mechanism helps the model better learn the labels related to the text, and by utilizing these text-related labels, the overall performance of the model is improved. We use text embedding As a query vector , label embedding As a key vector Sum value vector . Use the dot product attention mechanism to calculate the tag embedding associated with the text. as follows:

[0045] in, , express The transpose of .

[0046] Step (5.3): Fusion mechanism. Represents a text representation that carries tag information. Represents a tag that carries text information. Focusing on the context of the text, a label-to-text attention mechanism is used to learn the relevant text representation of each label. Focusing on the label information, the text-to-label attention mechanism is used to highlight the label representation related to the text. Therefore, in order to effectively utilize these two types of feature information, a fusion mechanism is used to combine the two parts of information. The fusion feature representation method is expressed as follows:

[0047] in , Represents a join operation.

[0048] Step (6): Model training and verification. The general method usually uses cross entropy loss for this operation, and then back-propagates the calculated loss to fine-tune the parameters of the network. However, this method cannot solve the problem of overfitting caused by the imbalance of samples in the class. In this case, the loss value is much larger than the loss value from the other class. Therefore, it is necessary to adjust the loss values ​​of the two classes according to certain weights. Appropriately scaling the loss can reduce the impact of the class with more samples and promote better training of the model.

[0049] For loss scaling, the following weight parameters can be used:

[0050] In the above formula, Corresponding to the total number of negative samples, is the number of positive samples, is a scaling parameter. The generated weights and correspond to the assigned weights of negative and positive samples, respectively. Use values ​​of 1, 10, and 100 to increase the influence of positive samples by increasing the values ​​of and respectively.

[0051] Therefore, the weighted scaled binary cross entropy loss of a sample is calculated as follows:

[0052] Here k={1,2,3, … , 𝐾}, where K is the total number of samples, is the true label (0 or 1) of sample k. In addition, is the predicted probability of sample k being in class 0, and is the probability of predicting sample k in class 1, so the final loss function is:

[0053] in, corresponds to the average value among all instances.

[0054] Step (7): Output the classification results and evaluate the model performance. Considering the imbalance between classes, this method uses Micro-F1 to measure the performance of all methods. Micro-F1 is a measure of the overall precision and recall rate considering all labels. , , They represent the true positive, false positive, and false negative of the 𝑡 label in the label set 𝑆. Micro-F1 is defined as:

[0055] in,

[0056] .

[0057] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A text multi-label classification method based on knowledge-enhanced attention network, characterized in that: The following steps are involved: S1. Input text data: obtain the text data to be processed; S2. Data preprocessing: cleaning and standardizing text data; S3. Feature extraction: In order to retain the information in the text that is meaningful to the classification task and reduce redundancy, the BERT pre-trained model is used to convert the text into a vector representation; S4. Create a text classification model and build an attention mechanism model based on knowledge enhancement; S5, model training and verification; S6. Result output: Output classification results and evaluate model performance.

2. The text multi-label classification method based on knowledge enhanced attention network according to claim 1 is characterized in that: The data preprocessing in step S2 specifically includes the following steps: S21, text cleaning: remove irrelevant characters and stop words; S22, text normalization: case conversion and word form restoration; S23, word segmentation: split the continuous text string into meaningful word sequences using a commonly used word segmentation algorithm; S24. Retrieve external knowledge based on entity linking technology and perform semantic expansion on key entities in the text.

3. The text multi-label classification method based on knowledge enhanced attention network according to claim 2 is characterized in that: The specific operation steps of step S24 are: S241, Text Entity Recognition, extract key entities from the input text as the basis for associating with the knowledge base, use the pre-trained model for entity recognition, extract entities ; S242, linking entities. For each identified entity, crawl the text in its online knowledge base online and select the one with the highest confidence. If there is more than one entity in the text, retrieve the text in their knowledge base for each entity and combine them in the order of the entities in the input text. These combined texts are regarded as the external knowledge base of the text. Align the extracted entities with the nodes in the external knowledge base and use the entity linking algorithm to find the entities that match the entity. Corresponding knowledge base node , the expression is as follows: ,in, Represents an external knowledge base; S243, knowledge expansion, extracting entity-related relationships and neighbor nodes from external knowledge bases to build knowledge subgraphs ,set up Is an external knowledge base For a certain entity node in , its neighbor node set is: , then the knowledge expansion is expressed as: ,in, represents the extracted knowledge subgraph, for The node's neighbor node set; S244, knowledge representation embedding, using the knowledge graph embedding method to embed the expanded knowledge subgraph Convert to a low-dimensional embedding vector and obtain the entity set through the training process Knowledge Representation ; S245, external knowledge enhanced text representation, combining the external knowledge representation with the input text embedding to generate knowledge enhanced text representation, assuming that the input text embedding is I and the external knowledge embedding is , the combination method can be weighted splicing: ,in, is an aggregation operation for embedding external knowledge, and is a hyperparameter used to balance the contribution of input text and external knowledge, and the final knowledge-enhanced text representation It can be directly used for the following multi-label classification tasks.

4. The text multi-label classification method based on knowledge enhanced attention network according to claim 1 is characterized in that: The feature extraction in step S3 specifically includes the following steps: S31. Using BERT pre-trained language model to enhance text representation To vectorize: , in, Indicates the final knowledge-enhanced text content of the input; S32. Vectorized content As input, bidirectional LSTM is used for unified encoding to obtain a unified contextual representation of the input text content and the acquired external knowledge, and the output dimension is a 2H-dimensional vector. The unified representation of the text content and the acquired external knowledge are expressed as ,in and Respectively represent the length of the input text content and the acquired external knowledge. The specific encoding process is as follows: ,h t Represents the state value at time t, t represents the current processing position in the sequence, and T represents the total number of time steps of the input sequence.

5. The text multi-label classification method based on knowledge enhanced attention network according to claim 1 is characterized in that: The specific operation steps of step S4 are: S41. Obtain the text feature representation related to the label by using the dot product attention mechanism. The dot product attention mechanism is as follows: , In the label-to-text attention mechanism, label embedding is used As a query vector , and text embedding As a key vector Sum value vector ,Finally, the text embedding associated with the tag as follows: ,in, , express is the transpose of ; k is the dimension of label embedding H; S42, Represents a text representation that carries tag information. Represents the label representation that carries text information. A fusion mechanism is used to combine these two parts of information. The fusion feature representation method is as follows: , in , Represents a join operation.

Images