A text classification method based on a bidirectional long short-term memory model and a knowledge graph

Abstract

The application provides a text classification method based on combination of bidirectional long short-term memory model and knowledge graph retrieval, which retrieves relevant prior supporting facts from the knowledge graph according to the task by using an attention mechanism, and incorporates the prior supporting facts into a feature space together with features learned from training data to classify the text. It firstly generates a word embedding model of a sentence by using a GloVe tool, and then respectively puts the word embedding model into a knowledge graph retrieval module and a bidirectional long short-term memory network BiLSTM, and then splices the output of the retrieval model and the output of the BiLSTM model to obtain a final classification. Compared with a traditional method, the accuracy is obviously improved by using the method of the knowledge graph. Finally, the model is evaluated on a 20Newsgroups text classification data set, and the experimental results prove the effectiveness of the model.

Description

Technical Field

[0001] This invention relates to the field of service natural language processing, specifically to a text classification method based on a combination of a bidirectional long short-term memory model and knowledge graph retrieval. Background Technology

[0002] Text classification is a classic problem in machine learning and natural language processing. Given descriptive text, the goal is to classify the text by analyzing its words and sentences. Based on whether deep learning is used, text classification techniques can be divided into two main categories: text classification based on traditional machine learning, such as TF-IDF text classification; and text classification based on deep learning, such as Long Short-Term Memory (LSTM) networks and the FastText method. However, existing machine learning models are overly reliant on the source of the data; the quality of the training dataset determines the final result of the model algorithm. Furthermore, existing methods only classify based on information within the text and cannot supplement this with common sense or real-world data.

[0003] Therefore, in order to solve the above problems, it is very important to provide a text classification method based on the combination of bidirectional long short-term memory model and knowledge graph retrieval. Summary of the Invention

[0004] To address the problem that existing text classification methods rely too heavily on datasets and fail to utilize common sense, this invention provides a text classification method based on a combination of a bidirectional long short-term memory model and knowledge graph retrieval.

[0005] The present invention adopts the following technical solution:

[0006] 1. A text classification method based on a combination of a bidirectional long short-term memory model and knowledge graph retrieval, comprising the following steps:

[0007] Step 1: Collect the dataset documents and define the labels as twenty categories. Divide the data into training and test sets according to the distribution of the data in the dataset;

[0008] Step 2: Text preprocessing for the training and test groups: Use the GloVe word embedding tool to train the content of the training and test groups into word embedding models, and put the word embedding vectors into a bidirectional long short-term memory neural network (BiLSTM) to obtain feature vectors;

[0009] Step 3: Input the word embedding vector from Step 2 into the knowledge graph retrieval module to obtain its output feature vector;

[0010] Step 4: Concatenate the BiLSTM output feature vector from Step 2 with the feature vector output by the knowledge graph retrieval module from Step 3 using a multilayer perceptron (MLP), and use an activation function to obtain the final classification.

[0011] Furthermore, in step 1, the dataset collected 18,000 news documents, which were evenly divided into 20 news groups with different topics; and then divided into training and testing sets in an 8:1 ratio.

[0012] Preferably, step 3 specifically includes the following sub-steps:

[0013] Step 3.1: Use the GloVe word embedding tool to process the word embedding of each sentence, and use the word embedding as input to BiLSTM and the knowledge graph retrieval module;

[0014] Step 3.2: The obtained word embeddings are fed into the BiLSTM module for encoding to obtain vector o, and then the context vector C is formed from vector o, as shown in equations (1)(2)(3):

[0015]

[0016] in For the current input, For the propagation of the previous state, f is a nonlinear function;

[0017]

[0018] Where T is the sequence length;

[0019]

[0020] Where W is the weight parameter and ReLU is the activation function;

[0021] Step 3.3: In step 3.2, two different context vectors are generated using separate BiLSTMs through the same process, which are then used for entity retrieval. Relationship retrieval ;

[0022] Step 3.4: Generate spatial attention for entities and relations in the knowledge graph; for entity vectors Attention and Relation Vectors Attention is given by formulas (4) and (5):

[0023]

[0024]

[0025] Where |E| is the number of entities in the knowledge graph, and |R| is the number of relations in the knowledge graph;

[0026] Step 3.5: The final retrieved entity and relation vectors are calculated by weighting the sum and the attention value of each retrieved entity / relation vector, as shown in equations (6) and (7):

[0027]

[0028]

[0029] Step 3.6: The entity vector e and relation vector r obtained in Step 3.5 are used to form a knowledge graph embedding through the knowledge graph embedding technology DKRL, which is used as the input t in Step 4;

[0030] Specifically, step 4 is as shown in equation (8):

[0031]

[0032] Where softmax is the activation function. t is an MLP multilayer perceptron, b is the output of step 2, and t is the output of step 3.

[0033] The beneficial effects of this invention are:

[0034] (1) It can be applied to text classification tasks and has good versatility;

[0035] (2) Simple text classification methods based on datasets are inaccurate. This method improves the accuracy of classification results by using a knowledge graph retrieval module combined with common knowledge in the public domain.

[0036] (3) Even when using 70% of the dataset, the accuracy of this method is close to that of the LSTM model using the complete dataset, thus reducing the dependence on the dataset; Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the overall model.

[0038] Figure 2 This is a schematic diagram of the knowledge graph retrieval module and context vector generation. Detailed Implementation

[0039] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and specific examples:

[0040] Combination Figures 1 to 2 A text classification method based on a combination of a bidirectional long short-term memory model and knowledge graph retrieval includes the following steps:

[0041] 1. A text classification method based on a combination of a bidirectional long short-term memory model and knowledge graph retrieval, characterized by the following steps:

[0042] Step 1: Collect the dataset documents and define the labels as twenty categories. Divide the data into training and test sets according to the distribution of the data in the dataset;

[0043] Step 2: Text preprocessing for the training and test groups: Use the GloVe word embedding tool to train the content of the training and test groups into word embedding models, and put the word embedding vectors into a bidirectional long short-term memory neural network (BiLSTM) to obtain feature vectors;

[0044] Step 3: Input the word embedding vector from Step 2 into the knowledge graph retrieval module to obtain its output feature vector;

[0045] Step 4: Concatenate the BiLSTM output feature vector from Step 2 with the feature vector output by the knowledge graph retrieval module from Step 3 using a multilayer perceptron (MLP), and use an activation function to obtain the final classification.

[0046] 2. In step 1, the dataset contains 18,000 news documents, which are evenly divided into 20 news groups with different topics; and divided into training set and test set in an 8:1 ratio.

[0047] 3. Step 3 specifically includes the following sub-steps:

[0048] Step 3.1: Use the GloVe word embedding tool to process the word embedding of each sentence, and use the word embedding as the input of BiLSTM and the knowledge graph retrieval module;

[0049] Step 3.2: The obtained word embeddings are fed into the BiLSTM module for encoding to obtain vector o, and then the context vector C is formed from vector o, as shown in equations (1)(2)(3):

[0050]

[0051] in For the current input, For the propagation of the previous state, f is a nonlinear function;

[0052]

[0053] Where T is the sequence length;

[0054]

[0055] Where W is the weight parameter and ReLU is the activation function;

[0056] Step 3.3: In step 3.2, two different context vectors are generated using separate BiLSTMs through the same process, which are then used for entity retrieval. Relationship retrieval ;

[0057] Step 3.4: Generate spatial attention for entities and relations in the knowledge graph; for entity vectors Attention and Relation Vectors Attention is given by formulas (4) and (5):

[0058]

[0059]

[0060] Where |E| is the number of entities in the knowledge graph, and |R| is the number of relations in the knowledge graph;

[0061] Step 3.5: The final retrieved entity and relation vectors are calculated by weighting the sum and the attention value of each retrieved entity / relation vector, as shown in equations (6) and (7):

[0062]

[0063]

[0064] Step 3.6: The entity vector e and relation vector r obtained in Step 3.5 are used to form a knowledge graph embedding through the knowledge graph embedding technology DKRL, which is used as the input t in Step 5;

[0065] 4. Step 4 is as shown in equation (8):

[0066]

[0067] Where softmax is the activation function. t is an MLP multilayer perceptron, b is the output of step 2, and t is the output of step 3.

[0068] Example 1

[0069] The following are specific embodiments of the application of this invention:

[0070] The method was validated using the 20Newsgroups dataset. This dataset contains 18,000 news documents, evenly divided into 20 newsgroups on different topics, and split into training and test sets in an 8:1 ratio.

[0071] Step 1 involves collecting the dataset documents and categorizing them into 20 classes. Some newsgroups have very similar themes, while others are completely unrelated. Detailed categorization is shown in Table 1.

[0072] Table 1: Classification in the 20Newsgroup dataset

[0073]

[0074] Step 2: Convert the text in the dataset from Step 1 into word vectors using the GloVe text processing tool;

[0075] Step 3: Input the word vectors from Step 2 into a Bidirectional Long Short-Term Memory (BiLSTM) neural network for encoding to obtain feature vectors;

[0076] Step 4: Input the word vectors from Step 2 into the knowledge graph retrieval module. First, the word vectors are encoded using the BiLSTM module to generate their feature vectors; then, the feature vectors of all word vectors are summed and input into the activation function to generate their context vectors; the context vectors are compared with the knowledge graph embeddings for similarity and querying, and the attention is calculated and weighted according to the attention mechanism to obtain the final output feature vector;

[0077] Step 5: The BiLSTM output feature vector from Step 3 and the feature vector output by the knowledge graph retrieval module from Step 4 are concatenated using a multilayer perceptron (MLP), and an activation function is used to obtain the final classification. The model parameters are shown in Table 2.

[0078] Table 2 Model Parameter Settings

[0079]

[0080] After step 5, once the model parameters are basically fixed, the model is evaluated using a test set, with the evaluation metric being the accuracy of classification.

[0081] Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the examples given above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A text classification method based on a combination of a bidirectional long short-term memory model and knowledge graph retrieval, characterized in that, Comprising the following steps: Step 1: Collecting dataset documents, defining labels as twenty categories, and dividing data into training set and test set according to the distribution of data in the dataset; Step 2: Text preprocessing for training group and test group: using GloVe word embedding tool to train the content in the training group and test group into word embedding model, and putting the word embedding vector into bidirectional long short-term memory neural network BiLSTM to obtain feature vector; Step 3: Input the word embedding vector in step 2 into the knowledge graph retrieval module to obtain its output feature vector, which specifically includes the following sub-steps: Step 3.1: Use Glove word embedding tool to process the word embedding of each sentence, and take the word embedding as the input of BiLSTM and the input of knowledge graph retrieval module; Step 3.2: The obtained word embedding is sent to the BiLSTM module for coding to obtain vector o, and then context vector C is formed from vector o, the process is as shown in formula (1) (2) (3): ， wherein is the current input, is the transfer of the previous state, f is a non-linear function; ， Wherein T is the sequence length; ， Wherein W is the weight parameter, and ReLU is the activation function; Step 3.3: Two different context vectors are formed using respective BiLSTMs in step 3.2 by the same process, one for entity retrieval and the other for relation retrieval with relation retrieval ; Step 3.4: Generate space attention for entities and relations in the knowledge graph; for entity vector Attention and relation vectors Attention is given as formula (4) (5): ， ， Wherein |E| is the number of entities in the knowledge graph, and |R| is the number of relations in the knowledge graph; Step 3.5: The final retrieved entity and relation vectors are calculated by weighted sum and attention value of each retrieved entity / relation vector, as shown in formula (6) (7): ， ， Step 3.6: The entity vector e and relation vector r obtained in step 3.5 are formed into knowledge graph embedding through knowledge graph embedding technology DKRL, which is used as input t in step 4; Step 4: The BiLSTM output feature vector in step 2 and the feature vector output by the knowledge graph retrieval module in step 3 are spliced together using a multi-layer perceptron MLP, and an activation function is used to obtain its final classification.

2. The text classification method based on bidirectional long short-term memory model combined with knowledge graph retrieval according to claim 1, characterized in that, In step 1, the dataset collects 18000 news documents, which are evenly divided into 20 different topic news group sets; According to the ratio of 8:1, it is divided into training set and test set. 3.The text classification method based on bidirectional long short-term memory model combined with knowledge graph retrieval of claim 1, characterized in that, The step 4 is shown in formula (8): ， where softmax is an activation function, is an MLP multi-layer perceptron, b is the output of step 2, and t is the output of step 3.

Images