Knowledge-Enhanced Document-Label Attention Method for Multi-Label Text Classification

Abstract

This invention provides a knowledge-enhanced document-tag attention method for multi-tag text classification. First, it innovatively mines and selects external knowledge from multi-tag documents to enrich document content, and jointly encodes and trains documents and knowledge to improve the interactivity of latent semantics between documents and knowledge. Simultaneously, it embeds the constructed tag sets to capture the contextual relationships between the tag sets corresponding to each document. Then, based on a document-knowledge-tag global attention mechanism, a weighted attention mechanism is applied to document-tag pairs and knowledge-tag pairs to fuse global information between documents, knowledge, and tags, assigning weights to obtain dependent and independent tag representations, thereby capturing the interaction features between documents, knowledge, and tag sets respectively. Finally, all tags for each document are predicted based on the global representations of documents, knowledge, and tags. This method solves the problems of insufficient document richness and tag dependency in multi-tag text classification.

Description

Technical Field

[0001] This invention belongs to the field of multi-label text classification, specifically involving a knowledge-enhanced document-label attention method for multi-label text classification. Background Technology

[0002] Multi-label text classification is a challenging task in text classification. It allows multiple labels to exist within a single document, each label representing an aspect of the document's content. Therefore, the overall semantic information of the document is composed of multiple or hierarchical components. Multi-label text classification is widely used in recommender systems, information retrieval, and other fields. For example, the news report "The cultural industry will become a pillar industry of the national economy in 2020" can be classified as either "economy" or "culture"; the movie *Twilight* can be classified as a romance film or a fantasy film. However, multi-label text classification faces several thorny issues: 1) Some documents lack sufficient content to accurately predict labels; these documents often belong to three or more labels; 2) There are interdependencies between labels, making some labels difficult to distinguish, such as "news" and "broadcast."

[0003] Common solutions for multi-label text classification include traditional machine learning methods and deep learning methods. Traditional machine learning algorithms primarily involve problem transformation, converting the multi-label text classification task into multiple single-label text classification tasks. These methods include BR, ML-DT, Rank SVM, LP, ML-KNN, and CC. For example, normalizing label relevance and building a separate classifier for each label transforms the multi-label text classification task into a binary classification problem chain, constructing decision trees in ML-DT, and using learning-like systems like SVM and KNN. Deep neural networks include methods based on CNN, RNN, Seq2Seq, and attention mechanisms. Examples include using fully connected networks, replacing the cross-entropy loss function with a ranking loss, using neurons to label the relevance between labels, and creating information-enhanced representations with additional semantic units based on hybrid attention mechanisms.

[0004] Traditional machine learning algorithms struggle to address high-level label relevance. Deep neural networks also fail to capture higher-order dependencies between labels or identify similar sub-labels. Seq2Seq models heavily rely on label order. Attention-based methods use insufficient attention weights to filter out all labels within a document. Furthermore, all previous research has failed to utilize external knowledge, which has proven crucial in text classification. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a knowledge-enhanced document-tag attention network algorithm for multi-tag text classification. The technical solution adopted in this invention is as follows:

[0006] This invention provides a knowledge-enhanced document-tag attention method for multi-tag text classification, characterized by the following steps: Step S1, retrieving external knowledge related to the document and enriching the document information based on the retrieved knowledge; Step S2, embedding the document and knowledge respectively to obtain the document embedding and the knowledge embedding, where embedding is the context representation; Step S3, encoding the document embedding and knowledge embedding to a uniform length to obtain a uniform-length document representation and knowledge representation; Step S4, constructing a tag set for the document and converting the tag set into an embedding vector to capture the contextual relationship between the tag sets corresponding to each document; Step S5, using a document-knowledge-tag global attention mechanism to fuse the document knowledge representation, knowledge representation, and global information between tags to obtain a document-knowledge-tag global representation; Step S6, predicting the tag set corresponding to the document based on the global representation.

[0007] The knowledge-enhanced document-tag attention method for multi-tag text classification provided by this invention may also have the following technical features, wherein the knowledge retrieval and acquisition process is as follows: based on entities in the input document via encyclopedia links, crawl the unstructured text corresponding to the entities, and concatenate the text into knowledge according to the order of the entities in the document.

[0008] The knowledge-enhanced document-tag attention method for multi-tag text classification provided in this invention may also have the following technical feature, wherein the embedding is: embedding documents and knowledge separately using a pre-trained language model, and the embedding process is represented as follows:

[0009] Em D =PLM(D), Em K =PLM(K)

[0010] In the formula, D represents the input document dataset, and K represents the retrieved knowledge dataset.

[0011] The knowledge-enhanced document-tag attention method for multi-tag text classification provided in this invention may also have the following technical feature, wherein the encoding is as follows: a bidirectional LSTM is used to encode the document embedding and the knowledge embedding, and a vector of dimension 2H is output, represented as follows:

[0012]

[0013]

[0014] In the formula, and For document and knowledge representations of uniform length, l1 and l2 represent the lengths of the input document and knowledge, respectively; t is the time step, ht represents the hidden state vector at time step t, and T is the total number of time steps in a complete training cycle.

[0015] The knowledge-enhanced document-tag attention method for multi-tag text classification provided by this invention may also have the following technical feature: In step S4, the GloVe algorithm is used to convert the tag set into an embedding vector, as shown below:

[0016] Em L =GloVe(L), Em L ∈R M×d

[0017] In the formula, L={l i |l i ={l 1 , l 2 , ..., l m} represents the tag set of the document, and M represents the total number of tags.

[0018] The knowledge-enhanced document-tag attention method for multi-tag text classification provided by this invention may also have the following technical features, wherein step S5 includes the following sub-steps: Step S5-1, applying a self-attention mechanism to the document and knowledge to obtain a document attention representation A. D And knowledge attention representation A K ;

[0019] Step S5-2: Based on the two attention representations obtained in step S5-1, obtain the corresponding independent document weights λ. D and independent knowledge weight λ K , representing the contribution of a document to a document-tag pair and the contribution of knowledge to a knowledge-tag pair, respectively, are represented as follows:

[0020] A D =softmax(W′1tanh(W1(En)) D ) T )),

[0021] A K =softmax(W′2tanh(W2(En)) k ) T )),

[0022] λ D =σ((A) D En D )W″1),λK=σ((A K En K )W″2);

[0023] Step S5-3: Use a document-tag attention mechanism on the document-tag pairs to obtain document-based tag attention representations. Furthermore, a knowledge-label attention mechanism is used to obtain knowledge-based label attention representations for knowledge-label pairs. The process is as follows:

[0024]

[0025]

[0026] Step S5-4: Assign weight factors β1 and β2 to document-tag pairs and knowledge-tag pairs respectively to obtain a multi-attention-based tag representation L. A and its independent label weight λ L The process is as follows:

[0027]

[0028] Where β1+β2=1;

[0029] Step S5-5: Calculate the final global representation S of document-knowledge-tag by multiplying by the dependency tag weight λ. A The process is as follows:

[0030]

[0031] Where W1, W′1, W″1, W2, W′2, W″2, W3, W′3, W4, W′4, W5 are trainable parameters, and σ is the sigmoid activation function.

[0032] Invention Function and Effect

[0033] The knowledge-enhanced document-tag attention method for multi-tag text classification according to the present invention effectively addresses the two problems of insufficient document richness and tag dependency in multi-tag text classification. This method innovatively mines and selects external knowledge corresponding to multi-tag documents, enriching the document content, and jointly encodes and trains documents and knowledge, improving the latent semantic interactivity between documents and knowledge. Simultaneously, it captures the contextual relationships between the tag sets corresponding to each document by embedding the constructed tag sets. Then, a global attention mechanism of document-knowledge-tag is employed, that is, a weighted attention mechanism is used for document-tag pairs and knowledge-tag pairs to fuse global information between documents, knowledge, and tags, assigning weights to obtain dependent and independent tag representations, thereby capturing the interaction features between documents, knowledge, and tag sets respectively. Finally, all tags for each document are predicted based on the global representations of documents, knowledge, and tags. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the architecture of the knowledge-enhanced document-tag attention network algorithm in an embodiment of the present invention. Detailed Implementation

[0035] To make the technical means, creative features, objectives and effects of this invention easy to understand, the following describes the knowledge-enhanced document-tag attention method for multi-tag text classification in detail with reference to embodiments and accompanying drawings.

[0036] <Example>

[0037] Figure 1 This is a schematic diagram of the architecture of the knowledge-enhanced document-tag attention network algorithm in an embodiment of the present invention.

[0038] like Figure 1 As shown, the knowledge-enhanced document-tag attention network algorithm in this embodiment mainly consists of a knowledge retrieval module, a document-knowledge embedding module, a document-knowledge encoding module, a tag embedding module, a document-knowledge-tag attention module, and a tag prediction module.

[0039] The algorithm comprises several modules: a knowledge retrieval module for retrieving external knowledge related to a document to enrich its information; a document-knowledge embedding module for capturing deep semantic information of both the document and the knowledge, extracting semantic features to obtain document embeddings and knowledge embeddings; a document-knowledge encoding module for encoding the document and knowledge embeddings to obtain uniform-length document and knowledge representations; a tag embedding module for mapping each tag to a high-dimensional space to capture the interaction information of sub-tags; a document-knowledge-tag attention module for capturing the global features of the document representation, knowledge representation, and tag embeddings; and a tag prediction module for predicting all tags corresponding to a document based on these global features. Furthermore, the algorithm is trained end-to-end, using cross-entropy loss as the loss function during training.

[0040] This embodiment applies the knowledge-enhanced document-tag attention network algorithm to classify multi-tag text. The following is based on... Figure 1 The classification process is described in detail:

[0041] In step S1, the knowledge retrieval module takes the document as input, retrieves the unstructured text corresponding to the entities in the document from Wikipedia, and concatenates the text into the corresponding knowledge as output.

[0042] In this embodiment, the TAGME tool is first used to retrieve entities from the input document based on Wikipedia links. Entities with a confidence score greater than 0.5, as provided by TAGME, are retained to ensure the accuracy of entity links. Then, for each retained entity, its corresponding Wikipedia text is crawled with the highest confidence score. If a document contains multiple entities, their corresponding Wikipedia texts are retrieved, and the extracted text is concatenated into knowledge according to the order of the entities in the document.

[0043] Step S2: The document-knowledge embedding module uses a pre-trained language model to embed documents and knowledge respectively, thereby obtaining the document embedding of documents and the knowledge embedding of knowledge, i.e., the context representation. The process is as follows:

[0044] Em D =PLM(D), Em K =PLM(K)

[0045] Where D and K represent the document dataset and the knowledge dataset retrieved by the knowledge retrieval module, respectively.

[0046] Step S3: The document-knowledge encoding module encodes the document embedding and knowledge embedding into a uniform length based on the encoder, thereby obtaining a document representation and knowledge representation of uniform length.

[0047] This embodiment uses a bidirectional LSTM to encode document embeddings and knowledge embeddings, and outputs a vector of dimension 2H. The representation of documents and knowledge of uniform length can be denoted as... and Where l1 and l2 represent the lengths of the input document and knowledge, respectively. The specific process is as follows:

[0048]

[0049]

[0050] In the formula, t is the time step, ht represents the hidden state vector at time step t, and T is the total time steps of the complete training cycle.

[0051] Step S4: Construct the tag set of the document. The tag embedding module converts the tag set into an embedding vector.

[0052] The tag embedding module aims to fully establish the contextual relationships between tags, because all tags in a document contain different but related semantic information. In this embodiment, the GloVe algorithm is used to convert the document's tag set into an embedding vector Em. L ∈R M×d , where M represents the total number of tags. The process is as follows:

[0053] Em L =GloVe(L)

[0054] In the formula, L={l i |l i ={l l , l 2 , ..., l m} represents a set of document tags.

[0055] Step S5: The document-knowledge-tag attention module uses a global attention mechanism of document-knowledge-tag to fuse document knowledge representation, knowledge representation and global information between tags to obtain a global representation of document-knowledge-tag.

[0056] The document-knowledge-tag attention module aims to capture the interaction features between documents and their corresponding knowledge sets, respectively, and the tag set. In multi-tag text classification tasks, a document may belong to multiple tags, and a tag may belong to multiple documents. Furthermore, the corresponding knowledge can serve as supplementary information to enrich the document. Therefore, this embodiment employs a global attention mechanism of document-knowledge-tag, specifically a weighted attention of document-tag attention and knowledge-tag attention, to fuse global information between documents, knowledge, and tags. The specific process is as follows:

[0057] First, a self-attention mechanism is applied to documents and knowledge to obtain a document attention representation A. D And knowledge attention representation A K Based on the two attention representations mentioned above, the corresponding independent document weights λ are obtained. D and independent knowledge weight λ K , representing the contribution of a document to a document-tag pair and the contribution of knowledge to a knowledge-tag pair, respectively, are represented as follows:

[0058] A D =softmax(W′1tanh(W1(En)) D ) T )),

[0059] A K =softmax(W′2tanh(W2(En)) k ) T )),

[0060] λ D =σ((A) D En D )W″1), λ K =σ((A) K En K )W″2)

[0061] In the formula, the tanh function is the activation function of the hidden layer in the attention network, which performs nonlinear mapping on the input convolutional signal and outputs a zero-mean distribution in [-1, 1]; the softmax function is the normalization function, which displays the classification result in the form of probability.

[0062] Secondly, a document-tag attention mechanism is used to obtain document-based tag attention representations for document-tag pairs. Furthermore, a knowledge-label attention mechanism is used to obtain knowledge-based label attention representations for knowledge-label pairs. The process is as follows:

[0063]

[0064]

[0065] Then, weight factors β1 and β2 are assigned to document-tag pairs and knowledge-tag pairs, respectively, to obtain a multi-attention-based tag representation L. A and its independent label weight λ L The process is as follows:

[0066]

[0067] Where β1+β2=1.

[0068] Finally, the final global representation S of document-knowledge-tags is calculated by multiplying by the dependency tag weight λ. A The process is as follows:

[0069]

[0070] Where W1, W′1, W″1, W2, W′2, W″2, W3, W′3, W4, W′4, W5 are trainable parameters, and σ is the sigmoid activation function.

[0071] Step S6: The tag prediction module predicts the tag set corresponding to the document based on the global representation.

[0072]

[0073] Among them, W p , W′ p These are trainable parameters.

[0074] Functions and effects of the embodiments

[0075] The knowledge-enhanced document-tag attention method for multi-tag text classification provided in this embodiment first innovatively mines and selects external knowledge corresponding to multi-tag documents, enriching the document content. It then jointly encodes and trains the documents and knowledge, improving the interactivity of latent semantics between them. Simultaneously, it captures the contextual relationships between the tag sets corresponding to each document by embedding the constructed tag sets. Next, a global attention mechanism of document-knowledge-tag is adopted, that is, a weighted attention mechanism is used for document-tag pairs and knowledge-tag pairs to fuse global information between documents, knowledge, and tags, assigning weights to obtain dependent and independent tag representations, thereby capturing the interaction features between documents, knowledge, and tag sets respectively. Finally, all tags for each document are predicted based on the global representations of documents, knowledge, and tags.

[0076] In this embodiment, because external knowledge of the document is retrieved based on entity linking technology and concatenated to form new knowledge, richer information is provided for the document. Furthermore, because attention mechanisms are applied to both document tag pairs and knowledge tag pairs, a dependency tag representation containing global information between the document, knowledge, and tags is obtained. Therefore, the knowledge-enhanced document-tag attention method for multi-tag text classification in this embodiment effectively solves the two problems of insufficient document richness and tag dependency in multi-tag text classification.

[0077] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the scope of the description of the above embodiments.

Claims

1. A knowledge-enhanced document-label attention method for multi-label text classification, characterized in that, The method comprises the following steps: Step S1, retrieving external knowledge related to a document, and enriching information of the document based on the retrieved knowledge; Step S2, embedding the document and the knowledge respectively to obtain a document embedding of the document and a knowledge embedding of the knowledge, wherein the embedding is a context representation; The embedding is obtained by embedding the document and the knowledge using a pre-trained language model, and the embedding process is represented as follows: In the formula, D represents an input document data set, and K represents a knowledge data set obtained by retrieval; Step S3, encoding the document embedding and the knowledge embedding into a unified length to obtain a document representation and a knowledge representation with a unified length; The encoding is performed by encoding the document embedding and the knowledge embedding using a bidirectional LSTM and outputting a vector with a dimension of 2H, and the encoding is represented as follows: wherein and are unified length document representations and knowledge representations, wherein and denote the length of input documents and knowledge; t is a time step, h t denotes the hidden state vector at time step t, and T is the total time steps of a complete training period. Step S4, constructing a label set of the document and converting the label set into an embedding vector to capture a context relationship between the label set corresponding to each document; In the step S4, the label set is converted into an embedding vector using a GloVe algorithm, and the conversion is represented as follows: ， wherein a set of tags representing the document, M representing the total number of tags; Step S5, fusing global information between the document knowledge representation, the knowledge representation, and the label using a document-knowledge-label global attention mechanism to obtain a global representation of the document-knowledge-label; The step S5 comprises the following sub-steps: Step S5-1, applying a self-attention mechanism to the document and the knowledge to obtain a document attention representation and a knowledge attention representation ; Step S5-2, obtaining corresponding independent document weight based on the two attention representations obtained in step S5-1 and independent knowledge weight respectively representing the contribution of the document in the document-label pair and the contribution of the knowledge in the knowledge-label pair, the process is represented as follows: ， ， ； Step S5-3, using a document-label attention mechanism on the document-label pair to obtain a document-based label attention representation , and using a knowledge-label attention mechanism on the knowledge-label pair to obtain a knowledge-based label attention representation The process is represented as follows: ； Step S5-4, assigning weight factors to the document-tag pairs and the knowledge-tag pairs, respectively and to obtain multi-attention based label representations and their individual label weights The process is as follows: wherein ; Step S5-5, the final document-knowledge-label global representation S is calculated by multiplying the dependency label weight λ A The process is as follows: wherein, are trainable parameters, is a sigmoid activation function; Step S6, predicting the label set corresponding to the document based on the global representation.

2. The knowledge-enhanced document-label attention method for multi-label text classification according to claim 1, characterized in that: wherein The retrieval process of the knowledge is as follows: inputting an entity in the document based on a knowledge link, crawling unstructured text corresponding to the entity, and splicing the text into knowledge according to an order of the entity in the document.

Images