A news event thread carding method and system

Abstract

The application discloses a news event context carding method and system, and relates to the field of internet big data.The method comprises the following steps: extracting a plurality of keywords of today's event according to the TF-IDF method; traversing historical news, if the news contains at least one keyword, the news is added to a news list; traversing each sentence of each news in the news list, selecting a plurality of candidate abstract sentences related to today's event and adding the candidate abstract sentences to an abstract list; rearranging the sentences in the abstract list based on the TextRank+MMR method, and selecting the first p abstract sentences as the final abstract of today's event; the system comprises a reading module, a keyword extraction module, an abstract extraction module and an integration module.The application can enable a user to have a general idea of the event context and obtain the most critical and concise information of the event, and can significantly reduce the cost and investment of manpower search and integration.

Description

TECHNICAL FIELD

[0001] The application relates to the field of Internet big data, in particular to a news event context carding method and system. BACKGROUND

[0002] With the advent of the Internet era, the amount of information increases dramatically every day, if you want to understand the origin and development of a hot event, not only need to search the relevant information of the event from the mass information, but also need to sort and filter it, distinguish the authenticity of the information and the chronological order, which consumes a lot of time, therefore, a kind of technology capable of automatically mining event context is particularly important. SUMMARY

[0003] The application aims to provide a news event context carding method and system, which extracts a number of historical news related to the event of interest from mass data according to the clustering idea, extracts key sentences from a large number of related news, generates an event summary by using the TextRank and MMR methods, enables the user to have a bird's eye view of the event context and obtain the most critical and concise information of the event, and significantly reduces the cost and investment of human search and integration, and has strong application value.

[0004] In order to achieve the above purpose, the application adopts the following technical scheme:

[0005] A news event context carding method, characterized in that it comprises the following steps:

[0006] S1: according to the date of today's event, select the news of today and a period of time before today as historical news for standby, and read the news json file of the historical news;

[0007] S2: extract a number of keywords of today's event according to the TF-IDF method;

[0008] S3: traverse the historical news, if a news contains at least one keyword, add the news to the news list;

[0009] S4: traverse each sentence of each news in the news list, calculate the minimum common substring length of the news and today's event, select a number of sentences in the news whose minimum common substring length with today's event is greater than 3, and splice the context of each sentence with one sentence before and after the sentence, and the number of the preliminary selected sentences and the sentences before and after the sentences are regarded as a number of candidate summary sentences related to the news and today's event;

[0010] S5: Select the top K most important sentences from the list of candidate summary sentences obtained in the previous step using the TextRank method as the summary sentences of the news; extract the summary sentences of all related news in the news list using the same method and put them into a summary list as the summary list of today's events;

[0011] S6: Rearrange the sentences in the summary list based on the TextRank+MMR method, and select the top p summary sentences as the final summary of today's events.

[0012] Preferably, in S1, between S1 and S2, there is also a step:

[0013] S11: Calculate the similarity between each pair of historical news, and remove the historical news with a similarity higher than a preset threshold.

[0014] Preferably, the similarity calculation method in S11 is a weighted fusion of three granularity similarity calculations, including word level, word level, and sentence vector level.

[0015] Further, the TF-IDF method in S2 is as shown in the formula:

[0016]

[0017] wherein, is the number of times the word w appears in the document d corresponding to today's event, is the total number of words in the document d, and N is the total number of documents, is the number of documents containing the word w;

[0018] Calculate the TF-IDF value of each word, and take the top several words in the sorted value as the keywords.

[0019] Preferably, the number of keywords is five.

[0020] Preferably, S6 further includes integrating today's event information, extracting and organizing key information into a json file for output. The key information includes: the time, content, and summary list of today's event; wherein the summary list includes the id of the source news, the time of the source news, the title of the source news, the summary sentence, and the url.

[0021] Preferably, it further includes step S7: input the json file containing the key information, and use the TextRank method to extract the summary of the key information; for the summary of each news, take the two sentences with the highest textrank scores as the summary sentences and add them to the original news file about today's event.

[0022] A news event context sorting system for implementing the method, comprising:

[0023] Read module: Used to read JSON files containing historical news.

[0024] Keyword extraction module: Used to extract keywords from today's events;

[0025] Summary extraction module: Used to extract summaries of today's events from historical news based on keywords;

[0026] Integration module: Used to integrate key information about today's events, including the time, content, and summary of today's events;

[0027] Output module: Used to output the key information in a JSON file.

[0028] Preferably, the system further includes:

[0029] Redundancy elimination module: This module calculates the similarity of each pair of historical news items and removes those with similarity scores exceeding a preset threshold. The advantage of this invention over existing technologies lies in its establishment of a relatively complete system for tracing the sequence of events. Based on today's events, it traces back and merges related past events to achieve high coverage. Simultaneously, it extracts key information from the merged news content to generate summaries, allowing users to easily obtain rich yet concise information. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0031] The specific embodiments of the present invention will now be described with reference to the accompanying drawings.

[0032] like Figure 1 The diagram shown is a schematic representation of the method flow of the present invention, which is described in detail below:

[0033] 1. News Clustering:

[0034] (1) First, based on today's date, read the news JSON files from today and the previous month, and calculate the similarity of news within a short period of time (such as within two days or one week, hereinafter referred to as historical news or historical events) pairwise. Remove those with high similarity to obtain historical news after eliminating redundant information.

[0035] The similarity calculation method is a weighted fusion of three granularity similarity calculations: character-level, word-level, and sentence vector-level. Specifically, the character and word-level similarity is the co-occurrence frequency of each character and word in pairs of news events. For the sentence vector level, the vector representation of each news event is the average of the word vectors contained in that event, and then the cosine similarity is calculated pairwise for the vectors of different events.

[0036] (2) Clustering:

[0037] First, according to the TF-IDF method, extract five keywords of each event today; as shown in the formula, is the number of times the word w appears in the document d, is the total number of words in the document d, N is the total number of documents, is the number of documents containing the word w, calculate the TF-IDF value of each word, and take the top five words after sorting as the keywords;

[0038]

[0039] Traverse today's news, judge whether it has the keywords of today's event, if k news contains a keyword, add the news to the news list we need.

[0040] Similarly, traverse the news of this month, judge whether it has the keywords of today's event, if k news contains a keyword, and these news are in the historical news, add the news to the news list we need.

[0041] Finally, combine the two news lists obtained by the above steps. At this point, the news list contains today's news (after eliminating redundant information) and the news related to today's event that occurred in the past period of time. Organize and output as a json file, including today's date, event date, historical event list, and clustered news list.

[0042] 2, Event-related summary:

[0043] (1) First, calculate the similarity between the events in a small period of time (such as yesterday, or within a week) and today's event, and remove the events with high similarity. Keep the events after deduplication as today's event.

[0044] (2) Traverse each sentence of each clustered news obtained in the previous module, calculate the minimum common substring length of each sentence and today's event, select a number of sentences in the news that have a minimum common substring length greater than 3 with today's event, and concatenate each sentence with its context. One sentence to improve sentence coherence and avoid splitting causes and effects. The selected sentences (and their preceding and following sentences) are a number of candidate summary sentences related to the news and the event.

[0045] (3) Use TextRank+MMR algorithm to select event summary sentences:

[0046] Firstly, the top K sentences are selected from the list of candidate summary sentences obtained in the previous step using the TextRank method as the summary sentences of the news. The same method is used to extract the summary sentences of all related news (clustered news obtained in the previous module) under the event and put them into a summary list as the summary of the event. The sentences in the summary list are rearranged based on the TextRank+MMR method, and the top p summary sentences are selected as the final summary sentences of the event.

[0047] The principle of the TextRank method is as follows:

[0048] TextRank is a graph-based algorithm that extracts keywords / sentences based on the co-occurrence information between words in a document. The nodes are words, and the edges are the similarity between words, which can be cosine similarity. After continuous iteration, this Markov chain forms a stationary distribution, and the TextRank value of each word / sentence is the stationary probability.

[0049]

[0050] d is the damping coefficient, is the weight of the edge, i.e., the similarity between words. According to the above formula, the weights of each node are iteratively propagated until convergence, i.e., the TextRank value of each node is obtained.

[0051] The principle of the MMR model is as follows:

[0052] The MMR model measures both relevance and diversity. It can easily adjust the weights of relevance and diversity to meet the requirements of "need similar content" or "need different aspects of content".

[0053]

[0054] The two terms of the formula are connected by a minus sign. While maximizing the similarity between the sentence and the remaining sentences, the redundancy of the sentence (i.e., the similarity between the sentence and the selected sentences) is reduced, making the selected sentences more diverse. is the sentence vector of a sentence, Q is the average of the sentence vectors of all key sentences, S is initialized as empty, and the sentence with the highest score is input into S through continuous iteration. The final result set is obtained after iteration.

[0055] Finally, the event information is integrated, and the key information is extracted and arranged into a json file. The content includes: the time, content, and summary list of the event; the summary list includes the id of the source news, the time of the source news, the title of the source news, the summary sentence, and the url.

[0056] 3、content abstract:

[0057] Input the json file containing content content, and use TextRank method to extract the summary of content. For the content of each news, take the two sentences with the highest textrank scores as the summary sentences, and add them to the original file.

[0058] The above is only the preferred specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can make equivalent replacement or change within the technical range disclosed by the present application according to the technical solution and the inventive concept of the present application, which should be covered within the protection scope of the present application.

Claims

1. A news event thread teasing method, characterized in that, Comprising the following steps: S1: Selecting news of today and a period of time before today as historical news according to the date of today's event, and reading the news json file of historical news; S2: Extracting several keywords of today's event according to the TF-IDF method; S3: Traversing the historical news, if a news contains at least one keyword, the news is added to the news list; S4: Traversing each sentence of each news in the news list, calculating the minimum common substring length of the news and today's event, selecting several sentences in the news whose minimum common substring length with today's event is greater than 3, and splicing the several selected sentences with the context of each sentence to obtain several candidate summary sentences related to today's event; S5: Selecting the top K most important sentences from the candidate summary sentence list obtained in the previous step as the summary sentences of the news using the TextRank method, and extracting the summary sentences of all related news in the news list using the same method and putting them into a summary list as the summary list of today's event; S6: Rearranging the sentences in the summary list based on the TextRank+MMR method, and selecting the first p summary sentences as the final summary of today's event.

2. The news event thread teasing method of claim 1, wherein, In S1, between S1 and S2, there is also a step: S11: Calculate the similarity between two historical news, and remove the historical news with a similarity higher than a preset threshold.

3. The news event thread teasing method of claim 2, wherein, The similarity calculation method in S11 is a weighted fusion of three granularity similarity calculations, including word level, word level, and sentence vector level.

4. The news event thread teasing method of claim 1, wherein, The TF-IDF method in S2 is as shown in the formula: Wherein, count(d,w) is the number of times the word w appears in the document d corresponding to today's event, count(d,*) is the total number of words in the document d, N is the total number of documents, N(w) is the number of documents containing the word w; Calculate the TF-IDF value of each word, sort and take the top several words as keywords.

5. The news event thread teasing method according to claim 1 or 4, characterized in that, The number of keywords selected is five.

6. The news event thread teasing method of claim 1, wherein, S6 also includes integrating today's event information, extracting and organizing key information into a json file and outputting.

7. The news event thread teasing method of claim 6, wherein, The key information includes: the time, content and summary list of today's event; Wherein the summary list includes the id of the news source, the time of the news source, the title of the news source, the summary sentence and the url.

8. The news event thread teasing method of claim 7, wherein, It also includes step S7: inputting the json file containing the key information and using the TextRank method to extract the summary of the key information; For the summary of each news, take the two sentences with the highest textrank scores as the summary sentences and add them to the original news file about today's event.

9. A news event thread mining system for implementing the method of claim 1, characterized by, It includes: Reading module: for reading the json file of historical news; Keyword extraction module: for extracting keywords of today's event; Summary extraction module: for extracting summaries about today's event from historical news according to keywords; Integration module: for integrating key information about today's event, including the time, content and summary of today's event; Output module: for outputting the json file of the key information.

10. The news event thread teasing system of claim 9, wherein, The system also includes: Redundancy elimination module: for calculating the similarity between two historical news, and eliminating the historical news with similarity higher than a preset threshold.

Images