An automatic summarization method and system based on the fusion of TextRank and multi-dimensional semantic features

Abstract

The invention discloses an automatic summarization method and system based on the fusion of TextRank and multi-dimensional semantic features. The method extracts news keywords, obtains news topics, and extracts sentence feature information and vector representation of sentences from pre-processed news. Four factors, namely, sentence-topic similarity, sentence-title similarity, keyword coverage, and whether a feature word is contained, are used as factors affecting sentence weights. These factors and the original TextRank score are applied to the automatic summarization in an optimized combination. The TextRank score and the sentence feature score are added and averaged to obtain a sentence comprehensive score. The MMR algorithm is used to calculate the similarity between sentences to remove sentences with high redundancy. Finally, the first n sentences are taken as summaries according to a set number according to the similarity sorting. The efficiency of removing redundant sentences by the original formula is improved, the summary content of the news can be accurately obtained, and the accuracy of the news content overview is greatly improved, thereby helping people reduce reading time.

Description

Technical Field

[0001] The invention belongs to the technical field of natural language processing, and specifically relates to an automatic summarization method and system based on the fusion of TextRank and multi-dimensional semantic features. Background Art

[0002] In today's Internet age, a large amount of news appears in people's daily lives, which requires people to spend more energy and enough time to search and identify the content we are interested in in the huge information database, making it particularly inconvenient for us to obtain the information resources we want to know. Text summarization technology can represent the originally complex and numerous texts in a concise news digest without changing the meaning of the article and without losing its important information, thereby helping people reduce reading time.

[0003] At present, the implementation methods of automatic summarization are mainly divided into extractive methods and generative methods. Among them, the extractive method directly extracts sentences that can represent the key information and main content of the article from the original text, and uses them as summaries without changing their grammatical structure. The generative method forms a summary based on the understanding of the input original text. The model attempts to understand the content of the text and can generate words that are not in the original text. It is closer to the essence of the summary and has the potential to generate high-quality summaries. However, this method not only requires high-quality training data, but also has great difficulty. The generated summaries often have problems such as word order errors and length dependence. The quality of the summary is far from meeting the requirements of practical applications. Therefore, it is of greater research significance to study how to improve the extractive summary.

[0004] At present, there are four main methods for extractive summarization: topic modeling, graph model, feature scoring, and sequence labeling. Among them, the graph-based method regards a sentence as a vertex of a graph, and the similarity between sentences constitutes the edge connecting the vertices. The sentences are sorted by iteratively calculating the sentence weights to obtain a summary. This method does not require training data and can directly use the information of the article itself to obtain a summary, so it has become the current mainstream method. The classic graph model algorithm TextRank was first proposed by Mihalcea in 2004. This method scores sentences by iteratively calculating the similarity between sentences to select key sentences in the text to form a summary. Subsequent studies have been improved based on this method. Zhang Lu et al. believe that the higher the coverage of keywords in a sentence, the more important the sentence is. The ROUGE score on the DUC2002 dataset is 13%-30% higher than TextRank. Li Feng et al. integrated keyword expansion to improve the effect of the summary, but ignored other factors that affect the effect of the summary, such as other topics and semantics. Yu Shanshan and others introduced information such as titles, paragraphs, special sentences, sentence positions and lengths into the construction of the TextRank network graph, and proposed an improved sentence similarity calculation method, thereby improving the effect of the TextRank algorithm. S. Sehgal added the similarity between the article and the title to the weight calculation of the sentence, which improved the accuracy of the text summary. Cao Yang and others compared the automatic summarization effects of different similarity calculation methods, selected the best similarity calculation method, and combined the sentence position, clue words and classic TextRank to calculate the sentence weight. Yu Chuanming and others proposed an extractive text summary model based on maximum boundary relevance. The model combines maximum boundary relevance (MMR) with deep learning, and comprehensively considers features such as sentence and full text similarity, keywords and position information to extract summaries. Liu Zhiming and others proposed a topic-based sentiment summary method, which obtains the article topic through the LDA model and then integrates traditional multi-features to extract the purpose summary.

[0005] Although the above-mentioned related studies have promoted the development of extractive summarization, they all have the problem of incomplete consideration of factors when scoring sentences based on TextRank. They only consider one factor, or consider multiple factors at the same time but ignore the topic factor of the article. The LDA model is used to extract the topic of the article, but the influence of news feature words on the sentence weight is ignored. At the same time, when targeting a specific field, the summaries generated by the above-mentioned related studies may not contain the content that the field is really concerned about. Summary of the invention

[0006] The purpose of the present invention is to provide an automatic summarization method and system based on the fusion of TextRank and multi-dimensional semantic features to overcome the shortcomings of the prior art.

[0007] An automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features includes the following steps:

[0008] S1, remove useless symbols and spaces from the original news;

[0009] S2, extract news keywords, obtain news topics, sentence feature information and sentence vector representation for the preprocessed news;

[0010] S3, converting the news keywords, news topics, sentence feature information and sentence vector representation obtained in step S2 into their respective feature scores, and then adding the respective feature scores in proportion to obtain the sentence feature score; at the same time, using the TextRank method to perform similarity calculation on the sentence vector representation to obtain the sentence TextRank score, and adding the TextRank score and the sentence feature score to obtain the average to obtain the sentence comprehensive score;

[0011] S4, using the MMR algorithm to calculate the similarity between sentences to remove sentences with high redundancy, and finally taking the first n sentences as the summary according to the set number of similarities.

[0012] Preferably, news keywords are obtained by the TF-IDF method and the domain words of the field to which the news belongs; news topics are obtained by training the LDA model through an unsupervised method; the feature information of the sentence is to determine whether the sentence contains feature words; the vector representation of the sentence is to train the word vector through the Word2vec deep learning method, and then obtain the sentence vector.

[0013] Preferably, the candidate keywords are obtained by weighting the candidate keywords obtained by the TF-IDF algorithm according to the weighted formula of the word weight:

[0014] weight(i,M)=h×count(i,M)

[0015] Among them, represents the weight of word i in text M; represents the frequency of word i in text M, which is calculated by the TF-IDF algorithm; h is the part-of-speech influence factor, which can be assigned different weights according to the different importance of nouns and verbs.

[0016] Preferably, a domain word dictionary of interest in a specific field is established. If the document contains a defined domain word, the domain word and the candidate keywords obtained by the TF-IDF algorithm are proposed as keywords for the document.

[0017] Preferably, a topic generation model based on LDA (Latent Dirichlet Allocation) is used to obtain the topic distribution of documents and the topic distribution of sentences, wherein the topic distribution of documents can be directly obtained from the parameters of the LDA generation model, and the topic distribution formula of sentences is as follows:

[0018]

[0019] Among them, P(W i |T) indicates the word W under this topic i The distribution probability of is obtained by the LDA model parameter φ; len(S) represents the length of sentence S.

[0020] Preferably, the feature information of the sentence includes the keywords: it is reported that, it is said, in short, it shows, therefore, and obviously.

[0021] Preferably, the sentence vector is obtained by adding and squaring, and the sentence vector is used to represent the overall semantic information of the sentence:

[0022]

[0023] in, Represents the sentence vector of sentence S; represents the spatial vector of each word that constitutes the sentence; n is the length of the sentence; and is composed of a k-dimensional spatial vector calculated by word2vec.

[0024] Preferably, the sentence keyword coverage score is:

[0025]

[0026] Among them, weight(j,M) represents the weight of the candidate keywords obtained; m represents the number of candidate keywords contained in sentence S; n represents the number of domain words contained in sentence S; W represents the weight of the domain words, in order to balance the weight of the candidate keywords;

[0027] Sentence and topic similarity score:

[0028] Theme(S)=1-JS(P||Q)

[0029] Through 1-JS(P||Q), we can get the sentence weight calculation formula that the greater the correlation between the document topic and the sentence topic, the greater the sentence weight, and the value range is still controlled between 0 and 1;

[0030] Sentence feature information score:

[0031]

[0032] Sentence and title similarity score:

[0033]

[0034] Among them, and represent the sentence vector of the sentence; then represents the similarity of the sentence. Among them, and Represents sentence S i ,S j sentence vector; Similarity(S i ,S j ) means that the sentence S i ,S j The similarity.

[0035] Preferably, the MMR algorithm is used to remove redundant sentences to increase the readability of the abstract:

[0036] MMR(S i )=λ×W(S i )-(1-λ)×max[Similarity(S i ,D)]

[0037] Among them, D is the set of selected summary sentences.

[0038] An automatic summarization system based on the fusion of TextRank and multi-dimensional semantic features, including a preprocessing module, a feature extraction module, a sentence scoring module and a summary extraction module:

[0039] The preprocessing module removes useless symbols and spaces from the original news;

[0040] The feature extraction module extracts news keywords, obtains news topics, and extracts sentence feature information and vector representation of sentences from the preprocessed news;

[0041] The sentence scoring module converts the acquired news keywords, news topics, sentence feature information and sentence vector representation into their own feature scores, and then adds up the respective feature scores in proportion to obtain the sentence feature score; at the same time, the TextRank method is used to perform similarity calculation on the sentence vector representation to obtain the sentence's TextRank score, and the TextRank score and sentence feature score are added and averaged to obtain the sentence's comprehensive score;

[0042] The summary extraction module calculates the similarity between sentences to remove sentences with high redundancy, and finally sorts them according to the similarity and takes the first n sentences as the summary output according to the set number.

[0043] Compared with the prior art, the present invention has the following beneficial technical effects:

[0044] The invention discloses an automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features. The method extracts news keywords, obtains news topics, and extracts sentence feature information and vector representation of sentences from pre-processed news. Four factors, namely, sentence-topic similarity, sentence-title similarity, keyword coverage, and whether a feature word is contained, are used as factors affecting sentence weights. The factors and the original TextRank score are applied to the automatic summarization in an optimized combination. The TextRank score and the sentence feature score are added and averaged to obtain a sentence comprehensive score. The MMR algorithm is used to calculate the similarity between sentences to remove sentences with high redundancy. Finally, the first n sentences are taken as summaries according to a set number according to the similarity sorting. The efficiency of removing redundant sentences by the original formula is improved, the summary content of the news can be accurately obtained, and the accuracy of the news content overview is greatly improved, thereby helping people reduce reading time.

[0045] Preferably, the JS divergence formula is referenced in the summary generation task, and the correlation between the article topic distribution and the sentence topic distribution is calculated by the JS divergence formula to obtain the similarity between the two. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flow chart of the automatic summarization method in an embodiment of the present invention.

[0047] Figure 2 This is a flowchart for removing redundant components in an embodiment of the present invention.

[0048] Figure 3 This is a comparison chart of the summary results generated by the method of the present application in the embodiment of the present invention in three aspects: ROUGE-1, ROUGE-2, and ROUGE-L.

[0049] Figure 4 A ROUGE score comparison chart of a summary generated for each set of coefficient calculation models in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] like Figure 1 As shown, the present invention provides an automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features. The MD-TextRank model established by the present invention is based on the traditional TextRank algorithm. It not only introduces word2vec to represent news text information, but also combines different features to update the weights of sentences. The method includes the following steps:

[0053] S1, data preprocessing: remove useless symbols and spaces from the original news, and save the original news and the manually annotated summary separately;

[0054] S2, feature extraction: extract news keywords, obtain news topics, sentence feature information and sentence vector representation from the preprocessed news;

[0055] Among them, news keywords are obtained by TF-IDF method and domain words of the field to which the news belongs; news topics are obtained by LDA model through unsupervised training; the feature information of the sentence is to determine whether the sentence contains feature words; the vector representation of the sentence is to train the word vector through the Word2vec deep learning method, and then obtain the sentence vector;

[0056] The more keywords a sentence contains, the more likely it is to be a summary sentence, and keyword extraction is performed:

[0057] The basic method adopted by this application is the TF-IDF algorithm: if a word or phrase appears repeatedly in an article, but rarely appears in other articles, then this word or phrase expresses the characteristics of the article, and is considered to be a keyword in the article. TF refers to the term frequency, that is, the number of times a word appears in the article; IDF refers to the inverse document frequency, which is the importance adjustment coefficient of the keyword, and can measure whether a word is a common word; the specific formula is as follows:

[0058]

[0059]

[0060] TF-IDF w =TF w *IDF w

[0061] Where N w It refers to the number of times word w appears in a certain article, which is the total number of words in the article. Y is the total number of documents in the corpus, which is the number of documents containing word w.

[0062] Then, based on the candidate keywords obtained by the TF-IDF algorithm, only nouns and verbs that can better represent the key information of the article are retained. Therefore, the following weighted formula for word weights is proposed:

[0063] weight(i,M)=h×count(i,M)

[0064] Among them, represents the weight of word i in text M; represents the frequency of word i in text M, which is calculated by the TF-IDF algorithm; h is the part-of-speech influence factor, which can be assigned different weights according to the different importance of nouns and verbs.

[0065] In some specific fields, keywords extracted solely based on word frequency may not be what the field is really concerned about. Therefore, a domain word dictionary that is of concern to the specific field is established. If the article contains defined domain words, the domain words and the candidate keywords obtained by the TF-IDF algorithm are proposed as the keywords of this article.

[0066] Get news topics:

[0067] The topic of an article often contains the most important content of the article. Therefore, the more similar the topic of a sentence is to the topic of a document, the more likely it is to be the summary of the article. This application uses a topic generation model based on LDA (Latent Dirichlet Allocation) to obtain the topic distribution of documents and the topic distribution of sentences. The topic distribution of documents can be directly obtained from the parameters of the LDA generation model. The topic distribution formula of sentences is as follows:

[0068]

[0069] Among them, P(W i |T) indicates the word W under this topic i The distribution probability of is obtained by the LDA model parameter φ; len(S) represents the length of sentence S.

[0070] Feature information of the sentence:

[0071] For a piece of news, if a sentence contains the characteristic words defined in Table 1, then the sentence is likely to be a summary sentence and the main sentence of the article.

[0072] Table 1 News feature words

[0073]

[0074] The more similar a sentence is to a news title, the more the sentence can express the main idea of ​​the text and the more likely it is to be a key sentence. However, the traditional formula only considers the words that appear together between two sentences when calculating the similarity between sentences, while ignoring the semantic similarity between the two sentences. Therefore, this paper proposes to use the word vector calculated by the word2vec algorithm to represent the semantic information of words, and then use the sentence vector to represent the overall semantic information of the sentence. The specific formula is as follows:

[0075]

[0076] in, Represents the sentence vector of sentence S; represents the spatial vector of each word that constitutes the sentence; n is the length of the sentence; and is composed of a k-dimensional spatial vector calculated by word2vec.

[0077] S3, sentence scoring: convert the news keywords, news topics, sentence feature information and sentence vector representation obtained in step S2 into their respective feature scores, and then add the respective feature scores in proportion to obtain the sentence feature score; at the same time, use the TextRank method to perform similarity calculation on the sentence vector representation to obtain the sentence TextRank score, and add the TextRank score and sentence feature score to obtain the average to obtain the sentence comprehensive score;

[0078] Sentence scoring:

[0079] When calculating sentence scores, the traditional TextRank algorithm only performs a simple statistics on the co-occurring words between two sentences, ignoring the impact of other factors on sentence weights. Therefore, this paper proposes to comprehensively consider the impact of four factors on sentence weights, including keyword coverage, sentence feature information, sentence-topic similarity, and sentence-title similarity, on sentence weights based on TextRank.

[0080] Sentence keyword coverage score:

[0081] The impact of keyword coverage on sentence weight is defined as Key(S). The higher the keyword coverage, the more likely the sentence is to be the summary of the article. The specific formula is as follows:

[0082]

[0083] Among them, weight(j,M) represents the obtained candidate keyword weight; m represents the number of candidate keywords contained in sentence S; n represents the number of domain words contained in sentence S; W represents the weight of domain words. In order to balance the weight of candidate keywords, the weight of domain words in this paper is set to 0.5; the sum of keyword weights divided by len(S) sentence length represents the keyword weight normalized to one word.

[0084] Sentence and topic similarity score:

[0085] The topic distribution of the document and the topic distribution of the sentence are obtained respectively. The more similar the two distributions are, the more topic information the sentence contains, and the more likely it is to be the summary sentence of the article. This paper first applies the JS (Jensen-Shannondivergence) divergence formula to the summary generation task to calculate the correlation between the two probability distributions as the similarity between the sentence and the topic. The JS divergence formula is based on the KL (Kullback-Leibler divergence) divergence formula, which solves the problem of asymmetric distance when comparing the correlation between distributions using the KL divergence formula; the specific formula is as follows:

[0086]

[0087]

[0088]

[0089] Among them, P represents the document topic probability distribution, Q represents the sentence topic probability distribution, and P(i) and Q(i) represent the i-th component of the P and Q distributions, respectively. JS(P||Q) is the correlation between the two distributions, and the value range is between 0 and 1. The higher the correlation, the smaller the value. The influence of the similarity between the sentence and the topic on the sentence weight is defined as Theme(S). The more similar the sentence is to the article topic, the higher the sentence weight is. The formula is as follows:

[0090] Theme(S)=1-JS(P||Q)

[0091] Through 1-JS(P||Q), we can get the sentence weight calculation formula, which states that the greater the correlation between the document topic and the sentence topic, the greater the sentence weight, and the value range is still controlled between 0 and 1.

[0092] Sentence feature information score:

[0093] The influence of feature words on sentence weight is defined as Feature(S). If a sentence contains a defined feature word, the sentence is weighted. The formula is as follows:

[0094]

[0095] Sentence and title similarity score:

[0096] Based on the sentence vector representation formula, each sentence and title is represented as a sentence vector that can represent the semantic information of the sentence. On the basis of the obtained sentence vector, this paper proposes the following new sentence similarity calculation formula:

[0097]

[0098] Among them, and represent the sentence vector of the sentence; then represents the similarity of the sentence. Among them, and Represents sentence S i ,S j sentence vector; Similarity(S i ,S j ) means that the sentence S i ,S j The similarity.

[0099] The influence of the similarity between sentence S and the title on the sentence weight is defined as Headline(S). If the similarity between the sentence and the title is higher, it means that the sentence is more likely to be the main sentence of the article, and the weight of the sentence is weighted. The formula is as follows:

[0100] Headline(S)=Similarity(S,S 标题 ).

[0101] Based on the traditional TextRank scoring method, this paper comprehensively considers the influence of four factors on sentence weight: keyword coverage, sentence-topic similarity, sentence-title similarity, and sentence feature information, to obtain the final score of the sentence. The formula is as follows:

[0102] Score(S i )=λ 1 Key(S i )+λ 2 Theme(S i )+λ 3 Feature(S i )+λ 4 Headline(S i )

[0103]

[0104] Among them, λ 1 +λ 2 +λ 3 +λ4 =1,λ 1 is the coverage weight coefficient, λ 2 is the sentence-topic similarity weight coefficient, λ 3 is the sentence feature information weight coefficient, λ 4 is the similarity weight coefficient between the sentence and the title; W(S i ) is a sentence S i The final score.

[0105] S4, summary extraction: use the MMR algorithm to calculate the similarity between sentences to remove sentences with high redundancy, and finally sort and output the top n sentences as summaries.

[0106] like Figure 2 As shown in the figure, after scoring each sentence, the first n most important sentences are extracted as summary sentences. However, if the first n sentences with the largest weights are directly extracted, they may contain redundant components, that is, different summary sentences express the same meaning, which violates the principle of simplicity and novelty of the summary. Therefore, this paper proposes the following formula based on the MMR (Maximal Marginal Relevance) algorithm to remove redundant sentences and increase the readability of the summary:

[0107] MMR(S i )=λ×W(S i )-(1-λ)×max[Similarity(S i ,D)]

[0108] Among them, D is the set of selected summary sentences. The first half of the formula is the score of the key information contained in the sentence, and the second half is the similarity between the sentence and the selected summary sentences, that is, whether the sentence is a redundant sentence. When considering the redundancy of candidate sentences, this paper takes the maximum similarity between the candidate sentence and the sentence in the summary set to judge the redundancy, that is, when the candidate sentence is similar to a sentence in the summary set, the similarity score of the sentence is large, which can effectively judge that the sentence has more redundant components. In addition, in order to reduce the search space and improve the efficiency of the algorithm, this paper sets a threshold for the similarity. When the maximum similarity between the candidate sentence and the sentence in the selected summary set reaches the set threshold, it means that the candidate sentence contains more redundant components, then the sentence is deleted from the candidate sentence set, that is, the sentence will no longer be considered in the future. By reducing the number of candidate sentences, the search space of the algorithm can be effectively reduced and the execution efficiency of the algorithm can be improved.

[0109] To ensure the accuracy and scientificity of the experiment, this paper selects 1,000 news articles in a specific field that also contain titles as a validation set, and the titles are used as reference summaries. And these 1,000 news articles are randomly divided into ten equal parts. For each news article, the jieba tool is first used to segment sentences and words, mark the part of speech, and then use the self-established stop word list to remove meaningless words in the text. Then the data processing method in the NLTK package is used to calculate the TF-IDF value of each word in the text, and the word vector obtained by the trained word2vec model and the self-established field words are used to weight the words to obtain keywords. Finally, the sentence weight is calculated by the scoring algorithm based on the fusion of topic keyword information proposed in this application, and the first two sentences are output as summaries after removing redundancy.

[0110] This experiment uses three aspects of ROUGE, ROUGE-1, ROUGE-2, and ROUGE-L, in the standard summary evaluation tool ROUGE to evaluate the summary results. The tool achieves the purpose of automatically evaluating the quality of automatic summarization by comparing the number of overlaps of the same evaluation units (unigrams, bigrams, and longest substrings) in the standard summary and the summary generated by the model.

[0111] This application uses the traditional TextRank model and the improved TextRank model proposed by other scholars as benchmark experiments, and compares the summary results generated by the model MD-TextRank in this application in three aspects: ROUGE-1, ROUGE-2, and ROUGE-L. The results are as follows: Figure 3 The specific experimental parameters of MD-TextRank in this paper are shown in Table 2, and the specific description of the benchmark model is shown in Table 3:

[0112] Table 2 Experimental parameter settings

[0113]

[0114] Table 3. Description of the benchmark model

[0115]

[0116] from Figure 4It can be seen from the data that the summary generated by MD-TextRank in this application performs best on ROUGE-1, ROUGE-2, and ROUGE-L compared with traditional TextRank and other improved models. Among them, the highest scores of the model in this paper on ROUGE-1, ROUGE-2, and ROUGE-L are 0.571, 0.321, and 0.486, respectively, while the highest scores of TextRank are only 0.482, 0.253, and 0.427, respectively, which are improved by 8.9, 6.8, and 5.9 percentages. And relative to the SW-TextRank model with better performance, the statistical results of the method of the present invention are improved by 1.4, 1.6, and 1.1 percentages, respectively. It can be seen that compared with the traditional TextRank model and the improved models proposed by other scholars, the MD-TextRank model proposed in this paper, based on the traditional TextRank model, introduces word2vec to represent news text information, and updates the weights of sentences from four dimensions: sentence-to-topic similarity, sentence-to-title similarity, keyword coverage, and whether it contains feature words, which can significantly improve the generation quality of news summaries.

[0117] In addition, the experimental results show that the summary generation model PGNet performs the worst on ROUGE-1, ROUGE-2 and ROUGE-L compared to the other four extractive models, with scores of only 0.358, 0.196 and 0.295 respectively, which are 12.4, 5.7 and 13.2 percentage points lower than the extractive model TextRank respectively. This shows that although the generative model is the mainstream research method, its performance is still far behind the extractive model from the generated results, and the grammar and readability of the summary are not satisfactory. Therefore, it is still very important to improve the extractive model.

[0118] When the MD-TextRank model in this paper scores sentences, it combines the influence of four features on the sentence weight, namely, keyword coverage, sentence-topic similarity, sentence-title similarity, and sentence feature information, to obtain the final score of the sentence. Table 4 shows the ROUGE scores of the summaries generated when no other factors are considered, only one factor is considered, and all four factors are considered at the same time. It can be seen that the algorithm of the sentence scoring by integrating four features in this paper's model improves the scores by 8.9, 6.8, and 5.9 percentage points respectively compared to the scores obtained without integrating any features. If only one feature is considered, the highest ROUGE score is obtained by integrating the sentence and title similarity features to score the sentence. However, the scores obtained by this model are still 2.7, 1.7, and 2.4 percentage points higher than this score, respectively. This shows that the automatic summary model MD-TextRank in this paper, which integrates the score obtained by TextRank with four semantic features that affect the sentence weight, can significantly improve the quality of the summary.

[0119] Table 4 The impact of different factors on summary results

[0120]

[0121]

[0122] This paper adopts the CBOW method in Word2vec for word vector representation, and the corpus resources trained in the experiment come from Sogou's full-network news corpus, which is about 1.6G in size. According to the size of the corpus, the dimension of the word vector starts from 50 dimensions and ends at 400 dimensions. As shown in Table 5, the ROUGE score of the AX-TextRank model is shown when the dimension ranges from 50 dimensions to 400 dimensions. As can be seen from the table, when the word vector dimension is 250, the ROUGE-1, ROUGE-2, and ROUGE-L scores are all the highest, but when the dimension is 200 dimensions, the ROUGE-L score is the same as the dimension 250, and the ROUGE-1 and ROUGE-2 scores are not much different. Considering that a larger vector dimension will affect the calculation speed, this paper chooses 200 dimensions as the word vector dimension, and finally we get a word vector of about 56MB for Chinese words.

[0123] Table 5 The impact of different dimensions on ROUGE scores

[0124]

[0125] This method needs to comprehensively consider the impact of different factors on sentence weights. Therefore, we set five sets of coefficient combinations for four different influencing factors, as shown in Table 6. And calculate the ROUGE score of the summary generated by the model for each set of coefficients, as shown in Figure 4 shown.

[0126] Table 6 Combinations of impact factor coefficients

[0127]

[0128] from Figure 4 It can be seen that the group is 5, λ 1 , 2 , 3 , 4 The ROUGE scores are highest when they are 0.2, 0.2, 0.4, and 0.2, respectively, indicating that the similarity between a sentence and a title has the greatest influence on the sentence weight. That is, the more similar a sentence is to a title, the more key information the sentence contains, and the more likely it is to be the summary of the article. Based on the results of this experiment, this paper will 1 , 2 , 3 , 4 The value of is set according to group 5.

Claims

1. An automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features, characterized in that: The following steps are involved: S1, remove useless symbols and spaces from the original news; S2, extract news keywords, obtain news topics, sentence feature information and sentence vector representation for the preprocessed news; S3, converting the news keywords, news topics, sentence feature information and sentence vector representation obtained in step S2 into their respective feature scores, and then adding the respective feature scores in proportion to obtain the sentence feature score; at the same time, using the TextRank method to calculate the similarity between the sentence vector representations to obtain the sentence TextRank score, and adding the TextRank score and the sentence feature score to obtain the average to obtain the sentence comprehensive score; According to the weighted formula of word weights, the candidate keywords are weighted based on the candidate keywords obtained by the TF-IDF algorithm to obtain the candidate keywords: weight(i,M)=h×count(i,M) Among them, represents the weight of word i in text M; represents the frequency of word i in text M, which is calculated by the TF-IDF algorithm; h is the part-of-speech influence factor, which can be assigned different weights according to the different importance of nouns and verbs; For a specific field, a dictionary of domain terms that are of interest to the field is established. If the article contains a defined domain term, the domain term and the candidate keywords obtained by the TF-IDF algorithm are proposed as the keywords of the article. The topic distribution of documents and sentences is obtained by using the topic generation model based on LDA Latent Dirichlet Allocation. The topic distribution of documents can be directly obtained from the parameters of the LDA generation model. The topic distribution formula of sentences is as follows: Among them, P(W i |T) indicates the word W under this topic i The distribution probability of is obtained by the LDA model parameter φ; len(S) represents the length of sentence S; The sentence vector is obtained by adding and squaring, and the sentence vector is used to represent the overall semantic information of the sentence: in, The sentence vector of sentence S; the space vector of each word that constitutes the sentence; n is the length of the sentence; it is composed of a k-dimensional space vector calculated by word2vec; S4, using the MMR algorithm to calculate the similarity between sentences to remove sentences with high redundancy, and finally taking the first n sentences as the summary according to the set number of similarities.

2. According to claim 1, an automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features is characterized in that: News keywords are obtained through the TF-IDF method and the domain words of the field to which the news belongs; news topics are obtained by training the LDA model through unsupervised methods; the feature information of the sentence is to determine whether the sentence contains feature words; the vector representation of the sentence is to train the word vector through the Word2vec deep learning method, and then obtain the sentence vector.

3. The automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features according to claim 1 is characterized in that: The characteristic information of the sentence includes the key words: it is reported that, it is said, in short, it shows, therefore and obviously.

4. The automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features according to claim 1 is characterized in that: Sentence keyword coverage score: Among them, weight(j,M) represents the weight of the candidate keywords obtained; m represents the number of candidate keywords contained in sentence S; n represents the number of domain words contained in sentence S; W represents the weight of the domain words, in order to balance the weight of the candidate keywords; Sentence and topic similarity score: Theme(S)=1-JS(P||Q) Through 1-JS(P||Q), we can get the sentence weight calculation formula that the greater the correlation between the document topic and the sentence topic, the greater the sentence weight, and the value range is still controlled between 0 and 1; Sentence feature information score: Sentence and title similarity score: Among them, and represent the sentence vector of the sentence; then represents the similarity of the sentence; among them, and Represents sentence S i ,S j sentence vector; Similarity(S i ,S j ) means sentence S i ,S j The similarity.

5. The automatic summarization method based on the fusion of TextRank and multi-dimensional semantic features according to claim 1 is characterized in that: Based on the MMR algorithm, it is proposed to remove redundant sentences to increase the readability of the abstract: MMR(S i )=λ×W(S i )-(1-λ)×max[Similarity(S i ,D)] Among them, D is the set of selected summary sentences.

6. An automatic summarization system based on the TextRank of the method of claim 1 and the fusion of multi-dimensional semantic features, characterized in that: Including preprocessing module, feature extraction module, sentence scoring module and summary extraction module: The preprocessing module removes useless symbols and spaces from the original news; The feature extraction module extracts news keywords, obtains news topics, and extracts sentence feature information and vector representation of sentences from the preprocessed news; The sentence scoring module converts the acquired news keywords, news topics, sentence feature information and sentence vector representation into their own feature scores, and then adds up the respective feature scores in proportion to obtain the sentence feature score; at the same time, the TextRank method is used to perform similarity calculation on the sentence vector representation to obtain the sentence's TextRank score, and the TextRank score and sentence feature score are added and averaged to obtain the sentence's comprehensive score; The summary extraction module calculates the similarity between sentences to remove sentences with high redundancy, and finally sorts them according to the similarity and takes the first n sentences as the summary output according to the set number.

Images