A news headline short text classification method based on dual knowledge graphs

Abstract

A news headline short text classification method based on dual knowledge graphs includes: preprocessing the news headline short text to remove special characters; extracting keywords in the news headline through the jieba word segmentation tool and removing stop words; linking keywords to the external knowledge base to obtain entity sets through the API provided by the CNDBPedia external knowledge base; disambiguating the entity set through cosine similarity to obtain a candidate entity set; constructing a domain knowledge graph based on global keyword co-occurrence information to solve the OOV problem; obtaining entity-related explanatory information by linking to an external knowledge base to enrich contextual semantic information; using BERT to obtain character-level vector representations of the original news headline and the explanatory information of the entity link, and fusing the vector representations of these two parts to make up for the shortcoming of insufficient information in short texts; using TextCNN to extract N‑grams features between multiple consecutive words to capture deep semantic information; finally, classifying through the Softmax function to obtain the final classification result.

Description

Technical Field

[0001] The present invention relates to a news headline short text classification method based on dual knowledge graphs, in particular to the classification of domain news headlines. The present invention utilizes a Chinese text segmentation tool to extract a number of keywords in each news headline, and links the keywords to entities in an external knowledge base through entity linking technology. If the link fails, a suitable node is queried from the domain knowledge graph to replace the original keyword. Otherwise, the entity is added to the candidate entity set. By relinking the candidate entity to the external knowledge base, the explanatory information related to the entity can be obtained. The present invention utilizes BERT to obtain the vector representation of the news headline short text and the explanatory information, and finally classifies them through softmax. The present invention relates to the fields of probability models, speech models, deep learning, and more specifically to the field of natural language processing based on deep learning. Background Art

[0002] With the popularity of news network platforms, the digital development of the news industry is very rapid, massive data is constantly generated, and the types of news are becoming more and more diverse. Compared with paragraphs and documents, news headlines have fewer words and lack contextual semantic information, which is sparse and ambiguous. Correctly classifying these news headlines can better organize and utilize this information. Therefore, how to accurately classify massive news headline data into the correct category is of great significance.

[0003] Faced with the huge and growing amount of news data, it is inefficient and costly to classify news headlines manually. In recent years, with the rapid development of machine learning and deep learning, more and more problems can be solved with the help of computers, which has become a popular technology in the era of big data. Therefore, using deep learning to solve the tedious task of news headline classification has become a trend. In recent years, many methods for classifying short texts of news headlines have been proposed, which can be roughly divided into two types:

[0004] 1) Classification methods based on machine learning: This type of method mainly preprocesses and extracts features from news headlines, then vectorizes the processed text, and finally models the training data through common machine learning algorithms, such as space vector models, decision tree models, and support vector machine models.

[0005] 2) Classification methods based on deep learning: This type of method mainly vectorizes each character in the news title, and then uses convolutional neural network (CNN) or recurrent neural network (RNN) to obtain deep local information or sequential information of the text. In recent years, more and more methods use external knowledge bases to obtain relevant concepts of short texts to enhance the semantic information of news title short texts.

[0006] Classification methods based on machine learning are relatively simple, but the quality of text feature extraction has a great impact on classification accuracy, and these methods rely heavily on manually designed features, which is very costly. In addition, these feature representations are often very sparse and have weak feature expression capabilities, which cannot meet the needs of news headline short text classification.

[0007] Classification methods based on deep learning can avoid complicated manual feature engineering. The impact on classification accuracy is more about the amount of data and the number of training iterations. Through word vector technology, text data can be converted into low-dimensional dense vectors, which does not rely on manually designed features and can learn deep semantic information.

[0008] However, there are still several problems in using deep learning technology to implement news headline classification:

[0009] (1) News headlines are mostly short texts with a small number of words and lack of contextual semantic information. They are sparse and ambiguous, which makes it difficult for current mainstream natural language processing methods to achieve good classification.

[0010] (2) Currently, many methods use external knowledge bases to enhance the semantic information of short texts. When obtaining entities through keywords, entities from multiple fields may be obtained. How to disambiguate entities to obtain a correct and reasonable entity is a challenge.

[0011] (3) Since the word segmentation boundaries in Chinese are not as clear as those in English, the accuracy of word segmentation will be affected by the word segmentation tool. Therefore, an inevitable problem when performing entity linking is the out-of-vocabulary (OOV) problem. How to avoid the OOV problem is also a challenge for the classification of short texts of news headlines.

[0012] In the field of natural language processing, in order to capture more semantic and syntactic information from short texts, the current mainstream method is to use external knowledge bases to enrich the semantic information of short texts. First, use common word segmentation tools to segment short texts. Then, through entity linking technology, keywords are linked to external knowledge bases to obtain related entities, and concepts related to entities are obtained from external knowledge bases. Finally, the original text information and concept information are spliced ​​to enhance the semantic information of short texts.

[0013] The above method has made some effective progress in the field of short text classification, but it also ignores the fact that irregular keywords may appear in news headlines during word segmentation, which may lead to OOV problems. The OOV problem refers to the inability of keywords to successfully link to external knowledge bases during entity linking, which will lead to a decrease in classification performance. Therefore, the present invention constructs a graph oriented to a specific field based on global keyword information. When the OOV problem occurs, appropriate keywords can be retrieved from the domain knowledge graph to replace the OOV words.

[0014] Therefore, how to solve the problem of insufficient information and OOV in short text of news headlines is a difficult problem that needs to be solved urgently in today's big data era. Summary of the invention

[0015] In order to solve the problem that the existing technology cannot handle the insufficient short text information and the OOV problem in entity linking, the present invention proposes a news headline short text classification method based on dual knowledge graphs.

[0016] With the help of external knowledge base, the present invention can extract additional explanatory information to enrich the semantics of short texts. When entity linking fails, the domain knowledge base can be used to reselect appropriate keywords to correct the results of entity linking and solve the OOV problem. Finally, the character-level features of the short text and the external knowledge features are combined to extract semantic information and obtain the final classification results. Compared with the existing methods, this method can achieve more advanced performance.

[0017] To solve the above problems, the technical solution provided by the present invention is:

[0018] A news headline short text classification method based on dual knowledge graphs includes the following steps:

[0019] Step 1: Preprocess the short text of the news title, mainly removing some special characters and stop words.

[0020] Step 2: Extract keywords from news titles using the Jieba word segmentation tool.

[0021] Step 3: Through the API provided by the CNDBPedia external knowledge base, link the keywords to the external knowledge base to obtain the entity set.

[0022] Step 4: Disambiguate the acquired entity set through cosine similarity to obtain a candidate entity set.

[0023] Step 5: Based on the global keyword co-occurrence information, build a domain knowledge graph to solve the OOV problem.

[0024] Step 6: For each entity in the candidate entity set, obtain the explanation information related to the entity by linking to the external knowledge base to enrich the contextual semantic information.

[0025] Step 7: Use BERT to obtain the character-level vector representation of the original news title and the explanation information of the entity link, and fuse the vector representations of these two parts to make up for the shortcomings of insufficient information in short texts.

[0026] Step 8: Use TextCNN to extract N-grams features between multiple consecutive words to capture deep semantic information.

[0027] Step 9: Finally, the Softmax function is used for classification to obtain the final classification result.

[0028] The present invention enhances the semantic information of short texts by extracting keywords from news headline short texts, retrieving related entities from external knowledge bases, and obtaining entity interpretations using entity linking technology. In addition, the present invention also constructs a domain knowledge graph based on local data sets to solve the OOV problem that occurs when linking entities and enhance the domain information of news short texts. Finally, the TextCNN model is used to capture the deep semantic features of news headline short texts and complete the corresponding classification. Compared with previous methods, the present invention has certain improvements in accuracy and efficiency.

[0029] The advantages of the present invention are:

[0030] 1. The present invention utilizes dual knowledge graphs to obtain more additional information, making up for the shortcoming of insufficient short text information and improving the accuracy of news headline short text classification.

[0031] 2. The present invention constructs a domain knowledge graph to solve the OOV problem, providing a new idea for solving the OOV problem in natural language processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0033] Figure 2 This is the core architecture diagram of the TextCNN of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0035] This example uses the THUCNews public data set of the Sina News RSS subscription channel as an example:

[0036] A news headline short text classification method based on dual knowledge graphs includes the following steps:

[0037] Step 1: Preprocess the short text of the news headline, mainly removing some special characters, such as Chinese and English punctuation marks, English characters, numbers and special symbols. In addition, stop words are removed according to the stop word list of Harbin Institute of Technology. The preprocessing results of some news headlines are shown in Table 1.

[0038] Table 1. Examples of news title preprocessing results

[0039] Original News Pre-processed news China Mobile launches the first commercial TD single chip China Mobile launches the first commercial single-chip Google buys thousands of IBM patents to fight Apple lawsuit Google buys thousands of patents to fight Apple lawsuit Motorola releases new Klassic phone with all-metal body Motorola releases new phone with all-metal body JD.com: Eligible iPhone users can receive price difference compensation JD.com eligible users can receive price difference compensation Sanyo's new ultra-light and ultra-thin linear PCM voice recorder is launched Sanyo's new ultra-light and ultra-thin linear voice recorder is launched

[0040] Step 2: Use the Jieba word segmentation tool to extract keywords from the preprocessed news headline short text. For example, for the short text S1: "Google buys another thousand ICM patents to respond to Apple's lawsuit", we can get the keyword set {"patent", "apple"}.

[0041] Step 3: The present invention uses the CNDBPedia external knowledge base developed by the Knowledge Factory Laboratory of Fudan University to obtain entities related to keywords according to the ment2ent entity linking API provided by it.

[0042] Step 4: For the entity set obtained in the previous step, use BERT to obtain the vector representation of the entity set and the news title. Then use cosine similarity to calculate the entity E i With news title S i The similarity of the two entities is calculated, and the entity with the highest similarity score is selected to be added to the candidate entity set. For the keyword "apple", the entity set E = {"Apple", "Apple (film)", "Apple (fruit)"} can be obtained. After calculating the cosine similarity, the following result is obtained: E = {"Apple": 90.26, "Apple (film)": 87.74, "Apple (fruit)": 87.88}. Therefore, the "Apple" with the highest score is added to the candidate entity set.

[0043]

[0044] Step 5: Since short texts may have irregular expressions, not every keyword can be successfully linked to entities in the external knowledge base, and OOV problems may occur here.

[0045] For Chinese entity linking, there are two main reasons for the OOV problem: (1) the entity is not covered by the external knowledge base and (2) the short text segmentation result is incorrect.

[0046] In order to solve the OOV problem, the present invention constructs a domain knowledge graph with keywords as nodes. Specifically, the present invention uses a sliding window of fixed size to collect keyword co-occurrence information. Point-wise mutual information (PMI) is used to calculate the weight between two keyword nodes. The higher the probability of two keywords appearing at the same time in the text, the stronger the correlation between the two keywords. When the value of PMI is less than 0, the relationship between the two keywords is considered to be weakly correlated. Only when the PMI value is greater than 0 will an edge be created between the two keywords. The calculation process of PMI is as follows:

[0047]

[0048]

[0049]

[0050] Here #W(i) represents the number of sliding windows in the corpus that contain keyword i, #W(i,j) represents the number of sliding windows that contain both keyword i and keyword j, and #W represents the total number of sliding windows.

[0051] With the help of the domain knowledge graph, when the entity linking has an OOV problem, the neighbor information of this keyword can be queried from the domain knowledge graph. The top three neighbors are sorted according to the weight value calculated by PMI. First, the original keyword is replaced with the lowest-ranked neighbor, and the entity is re-linked from the external knowledge base. If the OOV problem still occurs, the next ranked neighbor is taken out and re-linked in turn until success or the traversal ends.

[0052] In the news headline short text S2 "Wanfang Data agrees with the verdict of the paper infringement case", the keyword "infringement case" will have an OOV problem when performing ment2ent operations. At this time, take "infringement case" as a node, query the neighbor information in the domain knowledge graph, and get the node set {"litigation request", "copyright law", "litigation valley"}. Use "litigation request" to replace the keyword "infringement case", and relink to the external knowledge base. No OOV problem occurs, so "litigation request" is added to the candidate entity set.

[0053] Step 6: Semantic enhancement can make up for the lack of information in short texts. For the candidate entity sets obtained in the previous step, link them to the external knowledge base in turn to obtain explanation information related to the entity and enrich the semantic information of the short text.

[0054] For the candidate entity "litigation request", its explanation information K1 = {"The concept of litigation request has a broad and narrow meaning in foreign civil litigation. In a broad sense, a litigation request is a request made to the court for the court to make a judgment"};

[0055] For the candidate entity "Apple Inc.", its explanation information K2 = {"Apple Inc. is a high-tech company in the United States"} can be obtained.

[0056] Step 7: For Chinese short texts, the distribution of words is uneven, so we use the fine-tuned pre-trained model BERT to obtain character-level semantic information. There are two reasons for using character-level embedding instead of word embedding: (1) The length of news headlines is short, and word embedding has the problem of data sparsity. (2) It is convenient for TextCNN to extract N-grams information between multiple consecutive words.

[0057] Assume that the length of the news title short text S is n, the length of the explanation information K is l, and the vector dimension is d. If the length of the news title or the explanation information is not long enough, use <pad>To fill the sentence, otherwise cut off the redundant part. In this way, we can obtain the short text semantic matrix W s and the explanation information semantic matrix W k .

[0058]

[0059]

[0060] Represents the d-dimensional vector representation of the i-th word in the news title short text S, represents the vector concatenation operation. Therefore, the feature representation matrix after semantic enhancement is

[0061] Step 8: Although CNN is not suitable for learning long-distance semantic information, it can better learn the local information of the short text of the news headline. Therefore, the present invention adopts TextCNN to capture deep semantic information, which mainly includes convolutional layer, pooling layer and fully connected layer.

[0062] The convolutional layer uses The convolution kernel of different sizes acts on the semantic matrix of n+l words in length to obtain deep semantic features.

[0063] c i =f(w·x i:i+h-1 +b) (7)

[0064] here represents the bias term, f is the nonlinear activation function, and finally a new feature matrix c can be obtained.

[0065] c=[c1,c2,…,c n-h+1 ] (8)

[0066] The pooling layer is used to capture the most important eigenvalues, and dropout is used to randomly initialize the eigenvalues ​​to 0. This is a regularization method to avoid overfitting of the model. The feature matrices obtained by convolution kernels of different sizes are then concatenated together and input into the fully connected layer for classification.

[0067] Step 9: Use the softmax activation function to output the probability value of each category to obtain the final classification result.

[0068] The present invention mainly solves the problem of insufficient information and OOV of short text in natural language processing. A dual knowledge base model based on external knowledge base and domain knowledge base is proposed. The model uses the CNDBPedia external knowledge base to obtain semantic enhancement information of short text. When entity linking fails, it is replaced by finding suitable keywords in the domain knowledge graph. Then TextCNN is used to capture the features between multiple consecutive words. Finally, a fully connected layer is used for classification.

[0069] The present invention has been described through the above examples, but it is obvious that the examples here are only for the purpose of explanation, and the present invention is not limited to the scope of the examples. Researchers in the technical field of the present invention can make non-substantial changes in form and content according to the above steps without departing from the scope of the substantial protection of the present invention. Therefore, the present invention is not limited to the above specific forms and details.< / pad>

Claims

1. A news headline short text classification method based on dual knowledge graphs, comprising the following steps: Step 1: pre-process the short text of the news title to remove special characters; and remove stop words according to the stop word list of Harbin Institute of Technology; the special characters include Chinese and English punctuation marks, English characters, numbers and special symbols; Step 2: Extract keywords from news titles using the Jieba word segmentation tool; Step 3: Use the API provided by the CN-DBPedia external knowledge base to link keywords to the external knowledge base to obtain the entity set; Step 4: Disambiguate the obtained entity set through cosine similarity to obtain a candidate entity set; for the entity set obtained in step 3, use BERT to obtain the vector representation of the entity set and the news title; then use cosine similarity to calculate the entity E i With news title S i The similarity between the entities is selected, and the entity with the highest similarity score is added to the candidate entity set; Step 5: Based on the global keyword co-occurrence information, build a domain knowledge graph to solve the OOV problem; 5.1) Build a domain knowledge graph with keywords as nodes; Specifically, use a sliding window of fixed size to collect keyword co-occurrence information; Use point-by-point mutual information PMI to calculate the weight between two keyword nodes; The higher the probability of two keywords appearing at the same time in the text, the stronger the correlation between the two keywords; When the value of PMI is less than 0, the relationship between the two keywords is considered to be weakly correlated; Only when the PMI value is greater than 0, an edge will be created between the two keywords; The calculation process of PMI is as follows: Here #W(i) represents the number of sliding windows in the corpus that contain keyword i, #W(i,j) represents the number of sliding windows that contain both keyword i and keyword j, and #(W) represents the total number of sliding windows; 5.2) When the entity linking has an OOV problem, query the neighbor nodes of this keyword from the domain knowledge graph; sort the neighbor nodes according to the weight value calculated by PMI, and take out the top three neighbor nodes; first use the lowest-ranked neighbor node to replace the original keyword, and re-link the entity from the external knowledge base. If the OOV problem still occurs, take out the next ranked neighbor node in turn and re-link until success or the traversal ends; Step 6: For each entity in the candidate entity set, obtain the explanation information related to the entity by linking to the CN-DBPedia external knowledge base to enrich the contextual semantic information; Step 7: Use BERT to obtain the character-level vector representation of the original news title and the explanation information of the entity link, and fuse the vector representations of these two parts to make up for the shortcoming of insufficient information in short texts; Use the fine-tuned pre-trained model BERT to obtain character-level semantic information; use character-level embedding instead of word embedding; Assume that the length of the news title short text S is n, the length of the explanation information K is l, and the vector dimension is d; if the length of the news title or the explanation information is not long enough, use <pad>To fill the sentence, otherwise cut off the redundant part; get the short text semantic matrix W s and the explanation information semantic matrix W k ;< / pad> Represents the d-dimensional vector representation of the i-th word in the news title short text S, represents the vector concatenation operation; therefore, the semantically enhanced feature representation matrix Step 8: Use TextCNN to extract N-grams features between multiple consecutive words to capture deep semantic information; the TextCNN includes a convolutional layer, a pooling layer, and a fully connected layer; The convolutional layer uses The convolution kernel of different sizes acts on the semantic matrix of n+l words to obtain deep semantic features; c i =f(w·x i:i+h-1 +b)#(7) here represents the bias term, f is the nonlinear activation function, and finally a new feature matrix c is obtained; c=[c1,c2,…,c n-h+1 ]#(8) The pooling layer is used to capture the most important eigenvalues, and dropout is used to randomly initialize the eigenvalues ​​to 0; dropout is a regularization method used to avoid overfitting of the model; the feature matrices obtained by convolution kernels of different sizes are concatenated together and input into the fully connected layer for classification; Step 9: Output the probability value of each category through the softmax activation function to obtain the final classification result.

Images