A news text summary extraction method, system and medium

Abstract

The invention discloses a news text abstract extraction method, comprising the following steps: obtaining news and dividing the news into a plurality of sentences, extracting sentence vectors of the sentences, and constructing a similarity matrix according to sentence vector similarity; extracting news title sentence vectors, and calculating similarity with the sentences, and adjusting text weight values ​​for the first time according to the calculated similarity; extracting keywords from news texts and news titles, and adjusting text weight values ​​for the second time based on keyword data contained in the sentences; constructing a clue word list, and adjusting text weight values ​​for the third time according to clue words contained in the sentences; calculating sentence scores according to the similarity matrix and each sentence weight value; and sorting according to the sentence scores to complete text abstract extraction. The invention takes factors such as sentence position, title similarity, and keywords into consideration, so that the extracted abstracts can better fit the theme of the news, and has broad application prospects.

Description

Technical Field

[0001] The present invention relates to the research field of natural language processing, and in particular to a news text summary extraction method, system and medium. Background Art

[0002] With the vigorous development of the Internet industry and the rapid improvement of information technology, people's lifestyles have changed a lot. In this era of big data where information is widely disseminated, it is accompanied by a wealth of information. Due to the huge amount of information on the Internet, the various contents, and the continuous updating and dissemination, the phenomenon of information overload has become increasingly serious. The forms of expression of network information have also become diversified, such as text, images, and voice. Among them, text is an important way to carry information. Compared with images and voice, the expression of text is not intuitive and clear, but requires manual reading and comprehension. Especially when facing long articles, it is time-consuming and labor-intensive, and it is difficult to screen effective information, which seriously affects people's reading interest and reading experience. Therefore, how to deal with the phenomenon of information overload and effectively alleviate and solve the problem of users' difficulty in obtaining key information has become one of the most concerned hot topics in the world. Summary of the invention

[0003] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a news text summary extraction method, system and medium, which use artificial intelligence technology to extract key news information and refine and summarize it, so that people can quickly and accurately obtain key information from rich and massive network data within an effective time, thereby saving reading time and improving the reading experience.

[0004] The first object of the present invention is to provide a news text summary extraction method.

[0005] The second object of the present invention is to provide a news text summary extraction system.

[0006] A third object of the present invention is to provide a medium.

[0007] The first object of the present invention is achieved by the following technical solutions:

[0008] A news text summary extraction method, characterized in that it comprises the following steps:

[0009] Acquire news and divide the news into a number of sentences, extract sentence vectors of the sentences, and construct a similarity matrix according to the similarity of the sentence vectors;

[0010] Extracting the news headline sentence vector, and calculating the similarity with the sentence, and adjusting the text weight value for the first time according to the calculated similarity;

[0011] Extract keywords from news texts and news titles, and adjust text weights for the second time based on the keyword data contained in the sentences;

[0012] Construct a clue word list, and adjust the text weight value for the third time according to the clue words contained in the sentence;

[0013] Calculate the sentence score based on the similarity matrix and the weight value of each sentence;

[0014] The sentences are sorted according to their scores to complete text summary extraction.

[0015] Furthermore, the news is divided into several sentences, the sentence vectors of the sentences are extracted, and the similarity matrix is ​​constructed according to the similarity of the sentence vectors. Specifically, the news is divided into several sentences based on a preset separator, and an initial weight value is set for each sentence; the sentence vector of each sentence in the news is extracted using Albert, the similarity score between the sentence vectors is calculated using dot multiplication, and the similarity matrix is ​​constructed according to the similarity score between the sentence vectors:

[0016] similarityMatrix[i][j],

[0017] similarityMatrix[i][j] represents the i-th row and j-th column of the similarity matrix, and the value of similarityMatrix[i][j] is the similarity score between the i-th sentence and the j-th sentence in the news text.

[0018] Furthermore, the sentence vector of the news title is extracted, and the similarity is calculated with the text, and the text weight value is adjusted for the first time according to the calculated similarity, specifically: the sentence vector of the news title is extracted using Albert, and the similarity is calculated with each text, and the weight value of the text is adjusted for the first time according to the calculated similarity; for the first adjustment, let M represent the similarity between a sentence and the title, then the adjusted weight value of the sentence is M*A, A is a hyperparameter, that is, the more similar the text is to the news title, the higher the text weight value is adjusted.

[0019] Furthermore, the extraction of keywords from news texts and news titles, and the second adjustment of text weight values ​​based on keyword data contained in sentences, are specifically as follows: news texts are segmented and stop words are removed, keywords from news texts and news titles are extracted by a keyword extraction method, and the second adjustment of sentence weight values ​​is performed based on keyword data contained in each sentence; in the second adjustment, if a total of X keywords are extracted and a sentence contains Y of the X keywords, then the sentence weight is adjusted to Y / X*B, where B is a hyperparameter, i.e., the greater the proportion of keywords contained in the text, the higher the text weight adjustment.

[0020] Furthermore, the construction of the clue word list is specifically as follows: clue words refer to general indicator words, including: in summary, in a word, in short, and generally speaking.

[0021] Furthermore, the text weight value is adjusted for the third time according to the clue words contained in the sentence, specifically: according to the clue word list comparison, whether the sentence contains the clue words, if the sentence contains the clue words, the sentence weight value is adjusted for the third time; if the sentence does not contain the clue words, the text weight value is not adjusted; the third adjustment increases the text weight value of the text with clue words, if the sentence contains the clue words, the sentence weight is increased by C.

[0022] Furthermore, the sentence score is calculated based on the similarity matrix and the weight value of each sentence, specifically:

[0023] The similarity between a sentence and the sentence before it is multiplied by the text weight of any sentence before the sentence to obtain the adjusted sentence similarity; all the adjusted sentence similarities before the sentence are added together and multiplied by the weight of the sentence and then multiplied by a hyperparameter to obtain the score of the forward edge of the sentence;

[0024] The similarity between a sentence and the sentence after the sentence is multiplied by the text weight of any sentence after the sentence to obtain the adjusted sentence similarity; all the adjusted sentence similarities after the sentence are added and multiplied by the weight of the sentence and then multiplied by a hyperparameter to obtain the score of the backward edge of the sentence;

[0025] Add the scores of the forward and backward edges of the sentence to get the final score of the sentence.

[0026] Furthermore, the text summary extraction is completed by sorting according to the sentence scores, specifically: the matrix is ​​sorted according to the final scores of the sentences, and a certain proportion of sentences are selected, that is, the higher the score, the higher the priority is, until the set length requirement is met, and then the sentences are extracted according to the order of the sentences in the original text to obtain a summary set.

[0027] The second object of the present invention is achieved by the following technical solutions:

[0028] A news text summary extraction system, comprising:

[0029] A text acquisition and processing module, used to acquire news and divide the news into a number of sentences, extract sentence vectors of the sentences, and construct a similarity matrix according to the similarity of the sentence vectors;

[0030] The first adjustment module is used to extract the news headline sentence vector, calculate the similarity with the sentence, and adjust the text weight value for the first time according to the calculated similarity;

[0031] The second adjustment module is used to extract keywords from news texts and news titles, and to adjust the text weight value for the second time based on the keyword data contained in the sentence;

[0032] The third adjustment module is used to construct a clue word list and adjust the text weight value for the third time according to the clue words contained in the sentence;

[0033] The score calculation module calculates the sentence score based on the similarity matrix and the weight value of each text;

[0034] The summary extraction module sorts the sentences according to their scores and completes the text summary extraction.

[0035] The third object of the present invention is achieved by the following technical solutions:

[0036] A medium is a computer-readable storage medium, wherein the computer-readable storage medium stores at least one program, and the at least one program can be executed by at least one processor to implement a news text summary extraction method.

[0037] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0038] The present invention adopts Albert to extract features, making full use of the semantic relationship of sentences. In addition, a directed graph is adopted to consider the influence of the sentence position, so that sentences in different positions have different extraction priorities. In addition, factors such as titles and keywords are also considered, and the weight of each sentence is corrected, and the weight will affect the sentence score. When calculating the sentence score, the similarity score and weight of the forward node, the similarity score and weight of the backward node are considered, and the sentence score is obtained by combining the weight of the sentence itself. Compared with the current summary extraction method, this method has a higher accuracy rate and is particularly suitable for news summary extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a news text summary extraction method described in the present invention;

[0040] Figure 2 The present invention is a structural block diagram of a news text summary extraction system. DETAILED DESCRIPTION

[0041] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0042] Example:

[0043] A news text summary extraction method, such as Figure 1 As shown, the following steps are included:

[0044] Acquire news and divide the news into a number of sentences, extract sentence vectors of the sentences, and construct a similarity matrix according to the similarity of the sentence vectors;

[0045] Extracting the news headline sentence vector, and calculating the similarity with the sentence, and adjusting the text weight value for the first time according to the calculated similarity;

[0046] Extract keywords from news texts and news titles, and adjust text weights for the second time based on the keyword data contained in the sentences;

[0047] Construct a clue word list, and adjust the text weight value for the third time according to the clue words contained in the sentence;

[0048] Calculate the sentence score based on the similarity matrix and the weight value of each sentence;

[0049] The sentences are sorted according to their scores to complete text summary extraction.

[0050] As shown below:

[0051] 1) Divide the article into sentences, use Albert to extract the sentence vector of each sentence in the news, and build a similarity matrix based on the similarity scores between the sentence vectors. Set a weight value for each sentence, which is initially 1. Divide the article into sentences based on the preset separator. Use Albert to extract the sentence vector of each sentence in the news, and use dot multiplication to calculate the similarity scores between the sentence vectors. Build a similarity matrix based on the similarity scores between the sentence vectors. Then set an initial weight value for each sentence, which is initialized to 1.

[0052] Specifically: define a series of separators between sentences, usually some punctuation marks, such as periods and question marks. Use these separators to divide the article into sentences. If the sentence is too long, truncate it. Add the "[CLS]" symbol before the sentence and add "[SEP]" at the end of the sentence. Encode each word in the sentence into an id value through Albert's encoder. Input the encoded value into Albert to extract the sentence vector of each sentence in the article. Use dot multiplication to calculate the similarity score of the sentence, and construct the similarity score matrix similarityMatrix between sentences based on the similarity score. similarityMatrix[i][j] represents the similarity score between the i-th sentence and the j-th sentence. And set a weight value for each sentence, initially 1, stored as the array weights, weights[i] represents the weight value of the i-th sentence.

[0053] 2) Use Albert to extract the sentence vector of the news title, calculate the similarity with each sentence, and adjust the sentence weight value with reference to the similarity. Use Albert to extract the sentence vector of the news title, calculate the similarity with each sentence, and adjust the sentence weight value with reference to the similarity. The more similar it is to the title, the higher the sentence weight adjustment value.

[0054] Specifically, the same steps are used for the title of the article. Use Albert to extract the sentence vector of the news title, and calculate the cosine similarity with the sentence vector of each sentence calculated in step 1), denoted as similarity_cosine. Set a parameter title_w as a coefficient. Use the similarity and weight to adjust the sentence weight. The weight value of the i-th sentence is adjusted as weights[i]=weights[i]+similarity_cosine*title_w.

[0055] 3) Segment the article and remove stop words. Use the TF-IDF algorithm to extract keywords from news texts and news titles. Adjust the sentence weights based on the number of keywords in each sentence. The more keywords a sentence contains, the higher the sentence weight adjustment value.

[0056] Specifically: After using Jieba word segmentation to segment the article and title and remove stop words, use the TF-IDF algorithm to extract 10 keywords from the news text and news title as a set. Traverse each sentence of the article and calculate how many words appear in the keyword set. Suppose there are A words in the sentence after word segmentation, and B words appear in the keyword set, then the score is C = B / A. Set a parameter words_w as a coefficient, then the weight value of the i-th sentence is adjusted as weights[i] = weights[i] + C*words_w.

[0057] 4) Construct a clue word list, and if the sentence contains clue words, adjust the sentence weight value. If there are clue words, increase the sentence weight value, and if there are no clue words, do not increase the weight value.

[0058] Specifically, clue words refer to general indicative words such as "in summary", "in a word", "in short", "in general", etc. Construct a clue word table. If a sentence contains a clue word, adjust the sentence weight value. The adjustment method of the i-th sentence weight value is weights[i]=weights[i]+w. Among them, w is the weight value adjusted when the clue word is included.

[0059] 5) Calculate the sentence scores by combining the similarity matrix constructed in step 1) with the weight value of each sentence. The higher the sentence score, the higher the priority of the sentence during extraction. Based on the similarity matrix constructed in step 1), a directed graph is established. Combine the weight value and similarity score of the forward sentences to comprehensively calculate the edge weight score for this sentence. Similarly, combine the weight value and similarity score of the backward sentences to comprehensively calculate the edge weight score for this sentence. Sum all the forward edges of this sentence to obtain the forward total score, and sum all the backward edges to obtain the backward total score. Considering both the forward and backward scores and the weight of this sentence, obtain the final sentence score. The higher the sentence score, the higher the priority of the sentence during extraction.

[0060] Specifically: In step 1), a similarity score matrix similarityMatrix has been constructed. Where similarityMatrix[i][j] represents the similarity score between the i-th sentence and the j-th sentence. When i < j, this edge is a backward edge, and when i > j, this edge is a forward edge. When calculating the score of sentence i, combine the weight value and similarity score of the forward sentences to comprehensively calculate the edge weight score for this sentence, and calculate the sum of the edge weights of all the forward edges of this sentence, denoted as forward_scores[i]. The calculation method is as follows:

[0061]

[0062] Combine the weight value and similarity score of the backward sentences to comprehensively calculate the edge weight score for this sentence, and calculate the sum of all the backward edges of this sentence, denoted as backward_scores[i]. The calculation method is Take two parameters lambda1 and lambda2, which represent the weight ratios of the forward edges and backward edges respectively. Considering both the forward and backward scores and the weight of this sentence, obtain the final sentence score. The score of sentence i is:

[0063] scores[i] = (forward_scores[i] * lambda1 + backward_scores[i] * lambda2) * weights[i].

[0064] After calculating the sentence scores, sort the sentences according to the size of the final scores and select a certain number of sentences. The higher the score, the more优先 selected until the set length requirement is met. Finally, extract the sentences according to the order of the sentences in the original text to obtain the summary set.

[0065] A news text summary extraction system, as Figure 2 shown, includes:

[0066] A text acquisition and processing module, used to acquire news and divide the news into a number of sentences, extract sentence vectors of the sentences, and construct a similarity matrix according to the similarity of the sentence vectors;

[0067] The first adjustment module is used to extract the news headline sentence vector, calculate the similarity with the sentence, and adjust the text weight value for the first time according to the calculated similarity;

[0068] The second adjustment module is used to extract keywords from news texts and news titles, and to adjust the text weight value for the second time based on the keyword data contained in the sentence;

[0069] The third adjustment module is used to construct a clue word list and adjust the text weight value for the third time according to the clue words contained in the sentence;

[0070] The score calculation module calculates the sentence score based on the similarity matrix and the weight value of each text;

[0071] The summary extraction module sorts the sentences according to their scores and completes the text summary extraction.

[0072] A medium is a computer-readable storage medium, wherein the computer-readable storage medium stores at least one program, and the at least one program can be executed by at least one processor to implement a news text summary extraction method.

[0073] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A news text summary extraction method, characterized in that: The following steps are involved: Acquire news and divide the news into a number of sentences, extract sentence vectors of the sentences, and construct a similarity matrix according to the similarity of the sentence vectors; Extract the sentence vector of the news title, and calculate the similarity with the sentence, and adjust the text weight value for the first time according to the calculated similarity, specifically: use Albert to extract the sentence vector of the news title, and calculate the similarity with each text, and adjust the text weight value for the first time according to the calculated similarity; in the first adjustment, let M represent the similarity between a sentence and the title, then the adjusted weight value of the sentence is M*A, A is a hyperparameter, that is, the more similar the text is to the news title, the higher the text weight value is adjusted; Extract keywords from news texts and news titles, and adjust the text weight value for the second time based on the keyword data contained in the sentence, specifically: segment the news text and remove stop words, extract keywords from the news text and news title by a keyword extraction method, and make a second adjustment to the sentence weight value based on the keyword data contained in each sentence; in the second adjustment, if a total of X keywords are extracted and a sentence contains Y of the X keywords, then the sentence weight is adjusted to Y / X*B, where B is a hyperparameter, i.e., the greater the proportion of keywords contained in the text, the higher the text weight value adjustment; Construct a clue word list, and adjust the text weight value for the third time according to the clue words contained in the sentence; The construction of the clue word list is specifically as follows: clue words refer to general indicative words, including: in summary, in a word, in short, in general; The third adjustment of the text weight value according to the clue word contained in the sentence is specifically as follows: comparing according to the clue word list, comparing whether the sentence contains the clue word, if the sentence contains the clue word, the third adjustment of the sentence weight value is performed; if the sentence does not contain the clue word, the text weight value is not adjusted; the third adjustment increases the text weight value of the text containing the clue word; According to the similarity matrix and the weight value of each sentence, the sentence score is calculated as follows: The similarity between a sentence and the sentence before it is multiplied by the text weight of any sentence before the sentence to obtain the adjusted sentence similarity; all the adjusted sentence similarities before the sentence are added together and multiplied by the weight of the sentence and then multiplied by a hyperparameter to obtain the score of the forward edge of the sentence; The similarity between a sentence and the sentence after the sentence is multiplied by the text weight of any sentence after the sentence to obtain the adjusted sentence similarity; all the adjusted sentence similarities after the sentence are added and multiplied by the weight of the sentence and then multiplied by a hyperparameter to obtain the score of the backward edge of the sentence; Add the scores of the forward and backward edges of the sentence to get the final score of the sentence; The sentences are sorted according to their scores to complete text summary extraction.

2. A news text summary extraction method according to claim 1, characterized in that: The news is divided into several sentences, the sentence vectors of the sentences are extracted, and the similarity matrix is ​​constructed according to the similarity of the sentence vectors. Specifically, the news is divided into several sentences based on a preset separator, and an initial weight value is set for each sentence; the sentence vector of each sentence in the news is extracted using Albert, the similarity score between the sentence vectors is calculated using dot multiplication, and the similarity matrix is ​​constructed according to the similarity score between the sentence vectors: similarityMatrix[i][j], similarityMatrix[i][j] represents the i-th row and j-th column of the similarity matrix, and the value of similarityMatrix[i][j] is the similarity score between the i-th sentence and the j-th sentence in the news text.

3. A news text summary extraction method according to claim 1, characterized in that: The text summary extraction is completed by sorting according to the sentence scores, specifically: sorting the matrix according to the final scores of the sentences, selecting a certain proportion of sentences, that is, the higher the score, the higher the priority is selected, until the set length requirement is met, and then extracting the sentences according to the order of the sentences in the original text to obtain a summary set.

4. A news text summary extraction system, characterized in that: The method for extracting news text summaries according to any one of claims 1 to 3 above comprises: A text acquisition and processing module, used to acquire news and divide the news into several sentences, extract sentence vectors of the sentences, and construct a similarity matrix according to the similarity of the sentence vectors; The first adjustment module is used to extract the news headline sentence vector, calculate the similarity with the sentence, and adjust the text weight value for the first time according to the calculated similarity; The second adjustment module is used to extract keywords from news texts and news titles, and to adjust the text weight value for the second time based on the keyword data contained in the sentence; The third adjustment module is used to construct a clue word list and adjust the text weight value for the third time according to the clue words contained in the sentence; The score calculation module calculates the sentence score based on the similarity matrix and the weight value of each text; The summary extraction module sorts the sentences according to their scores and completes the text summary extraction.

5. A medium, characterized in that The medium is a computer-readable storage medium, and the computer-readable storage medium stores at least one program. The at least one program can be executed by at least one processor to implement the news text summary extraction method described in any one of claims 1 to 3.

Images