News summary generation method, device and computer equipment

Abstract

The present application relates to the field of language processing technology, and specifically discloses a method, device and computer equipment for generating news summaries. The method for generating news summaries includes: preprocessing news data to obtain a single-sentence text set; reordering each single-sentence text in the single-sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set; extracting a number of keywords from the single-sentence text set, and sorting the keywords according to a second preset importance sorting algorithm to obtain a keyword set; extracting the first summary sentence from the single-sentence text set; and logically splicing the first summary candidate sentence set, the keyword set and the first summary sentence to obtain the news summary. The news summary thus formed has good generalization, coherence and logic.

Description

Technical Field

[0001] The present invention relates to the field of language processing technology, and in particular to a method, device and computer equipment for generating news abstracts. Background Art

[0002] Text Summarization is a common basic task in Natural Language Processing (NLP). It refers to the extraction and refinement of text or text collection data through various technologies or methods to summarize its main ideas.

[0003] Text summarization algorithms are mainly divided into extractive and generative methods. The extractive method directly selects representative phrases, sentences or paragraphs from the original text to form a summary of the original text. Conventional extractive algorithms are often designed only from the perspective of the sentence's generalization of the text, and the resulting text summary often lacks logic and coherence. Summary of the invention

[0004] Based on this, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium and computer program product for generating a news summary in order to solve the above problems.

[0005] A method for generating a news summary, comprising:

[0006] Preprocess the news data to obtain a single sentence text set;

[0007] Reordering the single sentence texts in the single sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set;

[0008] Extracting a plurality of keywords from the single sentence text set, and sorting the plurality of keywords according to a second preset importance sorting algorithm to obtain a keyword set;

[0009] Extracting the first sentence of the summary from the single sentence text set;

[0010] The news summary is obtained by performing logical concatenation based on the first summary candidate sentence set, the keyword set and the first summary sentence.

[0011] In one embodiment, the step of preprocessing the news data to obtain a single sentence text set includes:

[0012] Filtering the news data according to preset rules;

[0013] The filtered news data is divided into single sentences to obtain several single sentence texts, forming a single sentence text set.

[0014] In one embodiment, the step of filtering the news data according to the preset rules is implemented by any one or more of the following methods:

[0015] Filtering the title data at all levels in the news data;

[0016] Filtering useless information in the news data;

[0017] Filtering the logical words at the beginning of each single sentence in the news data;

[0018] Filter the preset sentence patterns in the news data.

[0019] In one embodiment, the step of reordering the single sentence texts in the single sentence text set according to the first preset importance sorting algorithm to obtain the first summary candidate sentence set includes:

[0020] Using a summary extraction algorithm, each of the single-sentence texts in the single-sentence text set is used as a node to establish a graph model;

[0021] According to the relationship between the single sentence texts, importance index data of the single sentence texts in the news data are determined, and the single sentence texts are sorted according to the importance index data to form the first summary candidate sentence set.

[0022] In one embodiment, the step of extracting the first sentence of the summary from the single sentence text set includes:

[0023] Forming a first sentence classification model;

[0024] The single sentence texts in the single sentence text set are sequentially input into the first sentence classification model, and the first single sentence text determined as the first sentence of the summary by the first sentence classification model is used as the extracted first sentence of the summary.

[0025] In one embodiment, the step of performing logical concatenation based on the first summary candidate sentence set, the keyword set and the first summary sentence to obtain the news summary includes:

[0026] Screening the candidate summary sentences in the first candidate summary sentence set by using the keywords in the keyword set to obtain a second candidate summary sentence set;

[0027] The summary candidate sentences in the second summary candidate sentence set are concatenated with the summary first sentence to obtain the news summary.

[0028] In one embodiment, the step of filtering the candidate summary sentences in the first candidate summary sentence set by using the keywords in the keyword set to obtain the second candidate summary sentence set includes:

[0029] Extracting a preset number of target keywords from the keyword set in order of importance index data from high to low;

[0030] Single sentence texts containing any one of the target keywords are extracted from the first summary candidate sentence set in descending order of importance index data to form the second summary candidate sentence set.

[0031] A device for generating a news summary, comprising:

[0032] A preprocessing module is used to preprocess news data to obtain a single sentence text set;

[0033] A first generating module, configured to reorder the single sentence texts in the single sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set;

[0034] A second generating module is used to extract a plurality of keywords from the single sentence text set, and sort the plurality of keywords according to a second preset importance sorting algorithm to obtain a keyword set;

[0035] An extraction module, used to extract the first sentence of the summary from the single sentence text set;

[0036] The third generation module is used to perform logical splicing based on the first summary candidate sentence set, the keyword set and the first sentence of the summary to obtain the news summary.

[0037] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned news summary generation method when executing the computer program.

[0038] A computer-readable storage medium stores a computer program, which implements the steps of the above-mentioned news summary generation method when executed by a processor.

[0039] A computer program product includes a computer program, which implements the steps of the above-mentioned news summary generation method when executed by a processor.

[0040] The above-mentioned news summary generation method preprocesses the news data before the traditional extraction of summary candidate sentences, which can avoid the incoherent sentences and logical errors in the subsequent summary to a certain extent. At the same time, on the basis of extracting the summary candidate sentences, it also extracts the keywords in the news data and extracts the appropriate single sentence text as the first sentence of the summary. Finally, the news summary is spliced ​​together by combining the keywords, the summary candidate sentences and the first sentence of the summary. Since the keywords are highly generalized, combining the keywords to form a summary can make up for the generalization of the summary obtained by the traditional extraction method. The first sentence of the summary is extracted in advance, and the summary candidate sentences are directly added after the first sentence of the summary, which simplifies the difficulty of splicing the summary and can further improve the fluency and logic between sentences. The news summary formed in this way has good generalization, coherence and logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a method for generating a news summary provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of step S100 in a method for generating a news summary provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of step S200 in the method for generating a news summary provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of step S400 in the method for generating a news summary provided in an embodiment of the present application;

[0045] Figure 5 A flowchart of step S500 in the method for generating a news summary provided in an embodiment of the present application;

[0046] Figure 6 A flowchart of step S510 in the method for generating a news summary provided in an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of a device for generating a news summary provided in an embodiment of the present application;

[0048] Figure 8 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to understand the disclosure of the present invention more thoroughly and comprehensively.

[0050] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0051] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0053] Text Summarization is a common basic task in Natural Language Processing (NLP). It refers to the extraction and refinement of text or text collection data through various technologies or methods to summarize its main ideas.

[0054] With the rapid development of the Internet, people are paying more and more attention to information, and news websites generate a large amount of news every day. However, with this comes the increasingly serious problem of text information overload. When recommending and broadcasting daily news to users, in order to reduce the reading cost of users, it is necessary to extract and refine the news content and obtain the corresponding summary of the text.

[0055] Text summarization algorithms are divided into extractive and generative methods. The extractive method directly selects representative phrases, sentences or paragraphs from the original text to form a summary of the original text. For example, the traditional experience-based lead-3 algorithm directly takes the first three sentences of the original text as the summary of the text; the feature-based word-significance algorithm first selects the keywords in the text, and then sorts them according to the weight of the keywords, and selects the first sentence containing the keywords from the text in turn to form a summary; the neural network-based algorithm uses the neural network model's ability to understand the text semantics to determine whether a sentence is a summary sentence.

[0056] The above extraction algorithms, whether based on experience, keywords or neural networks, are all designed from the perspective of the sentence's generalization of the text, ignoring the coherence and logic between sentences, which can easily lead to problems such as inappropriate logic and incoherence.

[0057] Based on the above considerations, the present application provides a method for generating a news summary, a generating apparatus, a computer device, a computer-readable storage medium, and a computer program product.

[0058] In one embodiment, a method for generating a news summary is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server.

[0059] Reference Figure 1 In this embodiment, the method includes the following steps:

[0060] Step S100: pre-process the news data to obtain a single sentence text set.

[0061] After obtaining the news data, that is, the edited press release, the news data is first preprocessed. There are many ways to preprocess, such as filtering out content that is not conducive to the formation of a summary, or correcting obvious errors in the news data. The specific preprocessing method can be determined according to actual needs and is not limited here.

[0062] The preprocessing process also includes the segmentation of the news data, that is, the news data can be segmented into several single sentences according to the end symbol of each single sentence, thereby forming a single sentence text set.

[0063] Step S200: reorder the single sentence texts in the single sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set.

[0064] After the single sentence text set is determined, each single sentence text can be sorted according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set, and each single sentence text in the first summary candidate sentence set is arranged in order according to importance index data.

[0065] Among them, the first preset importance sorting algorithm generally refers to calculating the importance index data of each single sentence text in the single sentence text set according to the preset rules, and sorting each single sentence text according to the importance index data of each single sentence text. The importance index data is used to characterize the importance of the single sentence text in the single sentence text set. Generally, the importance index data can be set according to actual needs, and can be the degree of correlation with other single sentence texts in the single sentence text set, or can be the frequency of occurrence of the words contained in each single sentence text in the set as the importance index data. Of course, it can also be other data types, which are not specifically limited here.

[0066] Step S300: extract a number of keywords from the single sentence text set, and sort the keywords according to a second preset importance sorting algorithm to obtain a keyword set.

[0067] In this embodiment, on the basis of traditional extraction of summary candidate sentences, keywords are also extracted from a single sentence text set, and the extracted keywords are sorted according to a second preset importance sorting algorithm to form a keyword set, and each keyword in the keyword set is arranged in order of importance.

[0068] Among them, the second preset importance ranking algorithm generally refers to calculating the importance index data of each keyword in the single sentence text set according to the preset rules, and ranking each keyword according to the importance index data of each keyword. The importance index data is used to characterize the importance of the keyword in the single sentence text set. Generally, the importance index data can be set according to actual needs, and can include the frequency of occurrence of the keyword in the single sentence text set, and of course it can also be other data types, which are not specifically limited here.

[0069] Step S400: extract the first sentence of the summary from the single sentence text set.

[0070] Often the first sentence in an abstract is related to the fluency and logical connection of the entire abstract. Therefore, this embodiment further extracts a suitable sentence from the formed single-sentence text set as the first sentence of the abstract, which can improve the overall fluency and logic of the abstract.

[0071] Step S500: logically concatenate the first summary candidate sentence set, the keyword set and the first summary sentence to obtain a news summary.

[0072] After the first summary candidate sentence set, the keyword set and the first summary sentence are determined, they can be logically spliced ​​together. Specifically, the keywords in the keyword set fully reflect the generalization of the news data. The first summary candidate sentence set can be further screened and narrowed down according to the keywords to obtain more general summary candidate sentences, which are then spliced ​​together with the first summary sentence while ensuring the logic of the previous and next sentences to obtain the news summary.

[0073] The above-mentioned news summary generation method preprocesses the news data before the traditional extraction of summary candidate sentences, which can avoid the incoherent sentences and logical errors in the subsequent summary to a certain extent. At the same time, on the basis of extracting the summary candidate sentences, it also extracts the keywords in the news data and extracts the appropriate single sentence text as the first sentence of the summary. Finally, the news summary is spliced ​​together by combining the keywords, the summary candidate sentences and the first sentence of the summary. Since the keywords are highly generalized, combining the keywords to form a summary can make up for the generalization of the summary obtained by the traditional extraction method. The first sentence of the summary is extracted in advance, and the summary candidate sentences are directly added after the first sentence of the summary, which simplifies the difficulty of splicing the summary and can further improve the fluency and logic between sentences. The news summary formed in this way has good generalization, coherence and logic.

[0074] In one embodiment, referring to Figure 2 Step S100, i.e., preprocessing the news data to obtain a single sentence text set, comprises:

[0075] Step S110: Filter the news data according to preset rules.

[0076] That is, filtering rules can be set in advance to filter out content that is not conducive to the formation of summaries. For example, the titles of various levels in the news data can be extracted and filtered, the content in the news data that is irrelevant to the news event itself can be filtered out, and some sentences that are obviously not suitable as summaries, such as interrogative sentences, exclamatory sentences, subheadings, etc. can also be filtered out. Transition words, connecting words and other logical connecting words contained in the sentence can also be filtered out to avoid logical errors in the subsequent logical splicing. The above are just a few pre-processing methods listed, but are not limited to this and can be determined according to needs.

[0077] Step S120: dividing the filtered news data into single sentences to obtain a number of single sentence texts to form a single sentence text set.

[0078] When the news data is filtered, it can be segmented into single sentences. Specifically, the end symbol of each sentence can be identified, such as a period, question mark, exclamation mark, ellipsis, etc., and the sentences can be segmented according to the end symbol to form a single sentence text set.

[0079] Among them, after obtaining several single-sentence texts, the word segmentation tool can also be used to perform word segmentation processing on each single-sentence text.

[0080] In one embodiment, step S110, i.e., the step of filtering the news data according to the preset rules, is implemented by any one or more of the following methods:

[0081] (1) Filtering the title data of all levels in the news data. In order to reflect a better text structure, there are often various forms of titles in the news data. The title data is often not suitable as the content of the summary, so it can be filtered out. Since this type of title data has the characteristics of short text length, no ending, and starting with serial number data, the title data can be extracted and filtered out based on these characteristics.

[0082] (2) Filtering useless information in news data. News data usually contains a lot of information about editors, media, tips, etc. This information is irrelevant to the news event itself and can be treated as useless information and filtered out.

[0083] (3) Filter the logical words at the beginning of each sentence in the news data. Logical words can include transition words that appear at the beginning of a sentence, such as "but", "however", etc., and can include connecting words, such as "therefore", "moreover", "also", "secondarily", etc. These logical words are likely to cause logical errors between the previous and next sentences in the subsequent spliced ​​summary. Therefore, in the preprocessing process, the logical words at the beginning of each sentence are filtered to avoid logical errors in the subsequent splicing summary.

[0084] (4) Filtering preset sentences in news data. Due to the length limit of the summary and the fact that the summary is generally dominated by declarative sentences, subheadings, exclamatory sentences, interrogative sentences and other sentence patterns in news data are not suitable as summaries. These sentence patterns can be used as preset sentence patterns that need to be filtered and filtered according to the end mark. For subheadings, they can be identified based on the judgment method of whether there is an end mark. When it is identified that there is no end mark, it is determined to be title data and filtered.

[0085] In one embodiment, referring to Figure 3 Step S200, i.e., reordering the single sentence texts in the single sentence text set according to the first preset importance sorting algorithm to obtain the first summary candidate sentence set, comprises:

[0086] Step S210: Using a summary extraction algorithm, each single sentence text in the single sentence text set is used as a node to establish a graph model.

[0087] Step S220: Determine the importance index data of each single sentence text in the news data according to the relationship between each single sentence text, and sort each single sentence text according to the importance index data to form a first summary candidate sentence set.

[0088] In this embodiment, the TextRank algorithm can be used to take each single sentence text in the single sentence text set as a node to establish a graph model, and the relationship between the single sentence texts can be used to determine the importance index data of each single sentence text in the news data, and the single sentence texts in the single sentence text set can be sorted according to the importance index data of each single sentence text.

[0089] Specifically, the single sentence text collection 1 ,S 2 ,…,S n > is represented as a directed weighted graph G = (V, E), consisting of a point set V and an edge set E. The point set is a set of sentences, and each two sentences are connected by an edge. i ,V j The weight W between i,j , for a given point V i ,In(V i ) is the set pointing to the point, Out(V i ) is point V i The set of points pointed to. Point V i The score is calculated as follows:

[0090]

[0091] Where d is the damping coefficient, which ranges from 0 to 1. The scores of each point are initialized to the same value, and the weight between two points is the sentence similarity between the two points. After initialization, the scores of each point are recursively calculated until convergence. At this time, the score of each point will tend to be stable. The stable score is the importance index data. The size of the score determines the importance of the sentence in the entire news data. After sorting the sentences by score, we get 1 ′,S 2 ′,...,S n ′>, which is the first summary candidate sentence set.

[0092] In one of the embodiments, in step S300, i.e., extracting a plurality of keywords from a single sentence text set and sorting the keywords according to a second preset importance sorting algorithm to obtain a keyword set, a TF-IDF (term frequency–inverse document frequency) algorithm can be used to extract keywords.

[0093] ​​Specifically, the core idea of ​​the TF-IDF algorithm is to assign corresponding weights to different words based on their frequency of occurrence from a statistical perspective. This method can be used to evaluate the importance of a word in a specific corpus. The specific steps are as follows:

[0094] For a single sentence text collection, TF ij It indicates the frequency of the i-th word T in the j-th document after the word segmentation of the data set. The specific calculation formula is:

[0095]

[0096] where n ij It represents the number of times the i-th word T appears in the j-th document, and the denominator is the sum of the number of times all words appear in the j-th document.

[0097] Inverse document frequency is used to measure the general importance of a word. For the specific word T above, its IDF i It can be calculated by taking the logarithm of the quotient obtained by dividing the total number of documents by the number of files containing the document. The specific calculation formula is:

[0098]

[0099] Where |D| represents the total number of documents, and the denominator represents the number of documents containing the word T among all documents.

[0100] Then the term frequency inverse document frequency index can be expressed as:

[0101] TFIDF i,j =TF i,j ×IDF i

[0102] Finally, all words are sorted by their TF-IDF values ​​to get the importance ranking of all words in the single sentence text set. <W 1 ,W 2 ,...,W m >, that is, a set of keywords.

[0103] In one embodiment, referring to Figure 4 Step S400, i.e., the step of extracting the first sentence of the summary from the single sentence text set, comprises:

[0104] Step S410: forming a first sentence classification model.

[0105] Step S420: input each single sentence text in the single sentence text set into the first sentence classification model in sequence, and take the first single sentence text determined as the first sentence of the summary by the first sentence classification model as the extracted first sentence of the summary.

[0106] Generally speaking, single-sentence texts suitable as the first sentence of a summary often have consistency in vocabulary, sentence structure, and structure. In terms of vocabulary, there are generally no unclear pronouns. In terms of semantics, there is generally no continuation or transition from the previous sentence. In terms of sentence structure, question sentences and other forms are not suitable as the first sentence of a summary. In addition, due to the characteristics of news data writing, it is necessary to place key information in a prominent part of the article. Therefore, the first few sentences of most news data contain suitable first sentences for summaries. Therefore, when annotating positive samples, only the first few sentences of the news data need to be annotated, which effectively reduces the data annotation cost. In view of this, a first sentence classification model can be first trained to filter out the first sentence in the news data that is suitable as the first sentence of the summary.

[0107] In this embodiment, the fastText model can be used, which has only three layers, namely, the input layer, the hidden layer, and the output layer. The input of the model is the character vector and n-gram vector of the sentence, and the output is the category to which the sentence belongs. The first half of the model, i.e., from the input layer to the hidden layer, is mainly used to generate the vector representation of the sentence, and the second half of the model, i.e., from the hidden layer to the output layer, is mainly used for classification using hierarchical softmax. In the classification task, the model can better have both training accuracy and training time.

[0108] For any sentence in the single-sentence text set, assuming that the number of input feature vectors is N, the input vector can be trained by word2vec or other methods, and the feature vector is mapped to the middle layer through linear transformation as the vector representation of the sentence, and finally mapped to the label through the softmax layer. Similarly, the sentences in the single-sentence text set are input into the model one by one, and the first sentence determined by the model as the first sentence of the summary is taken. i >.

[0109] In one embodiment, referring to Figure 5 Step S500, i.e., performing logical splicing based on the first summary candidate sentence set, the keyword set and the first sentence of the summary to obtain the news summary, comprises:

[0110] Step S510: Screening the candidate summary sentences in the first candidate summary sentence set by using the keywords in the keyword set to obtain a second candidate summary sentence set.

[0111] ​The key information of a general text is mostly in the sentences containing text keywords. Therefore, after obtaining the second summary candidate sentence set, this embodiment can further screen the summary candidate sentences in the second summary candidate sentence set by keywords, and then obtain the summary candidate sentences with high relevance to the news event in the second summary candidate sentence set to form the second summary candidate sentence set, thereby improving the content of the news information in the subsequent summary. Specifically, sentences containing keywords can be screened out from the first summary candidate sentence set to serve as summary candidate sentences to be subsequently spliced ​​with the first sentence of the summary.

[0112] Step S520: concatenate the summary candidate sentences in the second summary candidate sentence set with the first summary sentence to obtain a news summary.

[0113] When the second set of candidate summary sentences is determined, the candidate summary sentences in the second set of candidate summary sentences can be spliced ​​with the first sentence of the summary to obtain a news summary. In practical applications, considering that the length of the summary has a certain influence on the summary effect, if the length of the summary is too long, the order and logical relationship between the sentences are difficult to grasp, and if the length of the summary is too short, it is difficult to cover the main theme of the news. Therefore, when splicing, it is necessary to select an appropriate number of candidate summary sentences and the first sentence of the summary for splicing. Specifically, two sentences with higher importance can be selected from the second set of candidate summary sentences as splicing objects, or three or four sentences can be selected, etc., depending on actual needs.

[0114] In one embodiment, referring to Figure 6 Step S510, i.e., filtering the candidate summary sentences in the first candidate summary sentence set by the keywords in the keyword set to obtain the second candidate summary sentence set, comprises:

[0115] Step S511 : extracting a preset number of target keywords from the keyword set in order of importance index data from high to low.

[0116] Assume that the keyword set is <W 1 ,W 2 ,...,W m >, where W 1 To W m The keywords are sorted from high to low according to the importance index data of each keyword. The top 5 keywords can be taken as target keywords. Of course, the number of extracted keywords is not unique, and can be 4, 6, 7, or 10, etc., depending on actual needs. For example, when the amount of information in news data is large, the number of target keywords can be increased to improve the coverage of key information in the summary; when the amount of information in news data is small, the number of target keywords can be reduced to simplify the summary generation process.

[0117] It should be noted that, since the importance index data of each keyword may be the same, for example, the maximum value of the importance index data is a, and there are 4 keywords with importance index data of a, if the preset number is 5, then these 4 keywords with importance index data of a can be extracted, and another keyword that is behind a and close to a can be extracted; if there are 5 keywords with importance index data of a, if the preset number is 5, then only these 5 keywords with importance index data of a can be extracted; if there are 6 keywords with importance index data of a, if the preset number is 5, then 5 keywords with importance index data of a can be randomly extracted. That is, after the preset number is determined, only the preset number of keywords need to be extracted.

[0118] Step S512: extracting single sentence texts containing any target keyword from the first summary candidate sentence set in descending order of importance index data to form a second summary candidate sentence set.

[0119] After the target keyword is obtained, the candidate abstract sentences in the first candidate abstract sentence set can be screened according to the target keyword to obtain the candidate abstract sentences containing the key news information. Specifically, the candidate abstract sentences in the first candidate abstract sentence set can be screened in descending order of importance index data to extract the candidate abstract sentences containing the target keyword. For example, the keyword set contains 5 keywords, and the first candidate abstract sentence set is 1 ′,S 2 ′,...,S n ′>,S 1 ′ to S n ′ Sort the importance index data of each single sentence text from high to low, and you can 1 ′,S 2 ′,...,S n ′>, select the first five candidate abstract sentences that contain any one of the five target keywords, and combine them to form the second candidate abstract sentence set 1 ″,S 2 ″,...,S 5 ″>, then add the first sentence of the abstract i > and the second summary candidate sentence set 1 ″,S 2 ″,...,S 5 ″>First concatenate and remove duplicates. In practical applications, considering that the length of the abstract should not be too long, the second abstract candidate sentence set 1 ″,S 2 ″,...,S 5 ​​​​​​″> A small number of candidate summary sentences are extracted from the summary and concatenated with the first sentence of the summary to control the length of the summary, for example, two or three sentences are extracted.

[0120] It should be noted that the news summary generation method provided in this embodiment, in addition to being able to make the formed summary have good generalization, coherence and logic, also requires binary classification data annotation only when classifying the first sentence, reducing the original chapter-level annotation requirement to sentence-level binary classification annotation requirement, taking into account both summary effect and data annotation cost.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a news summary generation device for implementing the above-mentioned news summary generation method. The implementation scheme for solving the problem provided by the news summary generation device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations of one or more news summary generation device embodiments provided below can refer to the limitations of the news summary generation method above, and will not be repeated here.

[0123] Reference Figure 7 The news summary generation device provided in this embodiment includes a preprocessing module 100, a first generation module 200, a second generation module 300, an extraction module 400 and a third generation module 500. Among them:

[0124] The preprocessing module 100 is used to preprocess the news data to obtain a single sentence text set;

[0125] The first generating module 200 is used to reorder the single sentence texts in the single sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set;

[0126] The second generation module 300 is used to extract a number of keywords from the single sentence text set, and sort the keywords according to a second preset importance sorting algorithm to obtain a keyword set;

[0127] The extraction module 400 is used to extract the first sentence of the summary from the single sentence text set;

[0128] The third generation module 500 is used to perform logical splicing based on the first summary candidate sentence set, the keyword set and the first sentence of the summary to obtain a news summary.

[0129] The above-mentioned news summary generation device pre-processes the news data before traditionally extracting summary candidate sentences, which can avoid incoherent sentences and logical errors in subsequent summaries to a certain extent. At the same time, on the basis of extracting summary candidate sentences, it also extracts keywords in the news data and extracts suitable single-sentence text as the first sentence of the summary. Finally, the news summary is spliced ​​together by combining the keywords, the summary candidate sentences and the first sentence of the summary. Since keywords are highly general, combining keywords to form a summary can make up for the generality of the summary obtained by the traditional extraction method. The first sentence of the summary is extracted in advance, and the summary candidate sentences are directly added after the first sentence of the summary, which simplifies the difficulty of splicing the summary and can further improve the fluency and logic between sentences. The news summary formed in this way has good generalization, coherence and logic.

[0130] In one embodiment, the preprocessing module is used to: extract and filter the news data according to preset rules; divide the extracted and filtered news data into single sentences to obtain a plurality of single sentence texts to form a single sentence text set.

[0131] In one embodiment, the preprocessing module extracts and filters the news data by any one or more of the following methods:

[0132] Extract title data at all levels from news data;

[0133] Filter out useless information from news data;

[0134] Filter the logical words at the beginning of each sentence in the news data;

[0135] Filters the preset sentence patterns in news data.

[0136] In one of the embodiments, the first generation module is used to: use a summary extraction algorithm to use each single sentence text in a single sentence text set as a node to establish a graph model; determine the importance index data of each single sentence text in the news data based on the relationship between the single sentence texts, and sort the single sentence texts according to the importance index data to form a first summary candidate sentence set.

[0137] In one embodiment, the extraction module is used to: form a first sentence classification model; input each single sentence text in the single sentence text set into the first sentence classification model in sequence, and use the first single sentence text determined by the first sentence classification model as the first sentence of the summary as the extracted first sentence of the summary.

[0138] In one of the embodiments, the third generation module is used to: extract a preset number of target keywords from the keyword set in order of importance index data from high to low; extract single sentence texts containing any target keyword from the first summary candidate sentence set in order of importance index data from high to low to form a second summary candidate sentence set; and concatenate the single sentence texts in the second summary candidate sentence set with the first sentence of the summary to obtain a news summary.

[0139] Each module in the above-mentioned news summary generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0140] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store various types of data involved in the method for generating a news summary. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating a news summary is implemented.

[0141] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0145] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0146] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for generating a news summary, characterized in that: include: Preprocess the news data to obtain a single sentence text set; Reordering the single sentence texts in the single sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set; Extracting a plurality of keywords from the single sentence text set, and sorting the plurality of keywords according to a second preset importance sorting algorithm to obtain a keyword set; Extracting the first sentence of the summary from the single sentence text set; The news summary is obtained by performing logical concatenation based on the first summary candidate sentence set, the keyword set and the first summary sentence.

2. The method for generating a news summary according to claim 1, characterized in that: The step of preprocessing the news data to obtain a single sentence text set comprises: Filtering the news data according to preset rules; The filtered news data is divided into single sentences to obtain several single sentence texts, forming a single sentence text set.

3. The method for generating a news summary according to claim 2, characterized in that: The step of filtering the news data according to the preset rules is implemented by any one or more of the following methods: Filtering the title data at all levels in the news data; Filtering useless information in the news data; Filtering the logical words at the beginning of each single sentence in the news data; Filter the preset sentence patterns in the news data.

4. The method for generating a news summary according to claim 1, characterized in that: The step of reordering the single sentence texts in the single sentence text set according to the first preset importance sorting algorithm to obtain a first summary candidate sentence set comprises: Using a summary extraction algorithm, each of the single-sentence texts in the single-sentence text set is used as a node to establish a graph model; According to the relationship between the single sentence texts, importance index data of the single sentence texts in the news data are determined, and the single sentence texts are sorted according to the importance index data to form the first summary candidate sentence set.

5. The method for generating a news summary according to claim 1, characterized in that: The step of extracting the first sentence of the summary from the single sentence text set comprises: Forming a first sentence classification model; The single sentence texts in the single sentence text set are sequentially input into the first sentence classification model, and the first single sentence text determined as the first sentence of the summary by the first sentence classification model is used as the extracted first sentence of the summary.

6. The method for generating a news summary according to claim 1, characterized in that: The step of performing logical splicing based on the first summary candidate sentence set, the keyword set and the first sentence of the summary to obtain the news summary comprises: Screening the candidate summary sentences in the first candidate summary sentence set by using the keywords in the keyword set to obtain a second candidate summary sentence set; The summary candidate sentences in the second summary candidate sentence set are concatenated with the summary first sentence to obtain the news summary.

7. The method for generating a news summary according to claim 6, characterized in that: The step of filtering the candidate summary sentences in the first candidate summary sentence set by using the keywords in the keyword set to obtain the second candidate summary sentence set comprises: Extracting a preset number of target keywords from the keyword set in order of importance index data from high to low; In descending order of importance index data, single sentence texts containing any one of the target keywords are sequentially extracted from the first summary candidate sentence set to form the second summary candidate sentence set.

8. A device for generating a news summary, characterized in that: include: A preprocessing module is used to preprocess news data to obtain a single sentence text set; A first generating module, configured to reorder the single sentence texts in the single sentence text set according to a first preset importance sorting algorithm to obtain a first summary candidate sentence set; A second generating module is used to extract a plurality of keywords from the single sentence text set, and sort the plurality of keywords according to a second preset importance sorting algorithm to obtain a keyword set; An extraction module, used to extract the first sentence of the summary from the single sentence text set; The third generation module is used to perform logical splicing based on the first summary candidate sentence set, the keyword set and the first sentence of the summary to obtain the news summary.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for generating a news summary according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a news summary according to any one of claims 1 to 7 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating a news summary according to any one of claims 1 to 7 are implemented.

Images