Streaming news clue discovery method based on clue cluster

Abstract

The present invention is a streaming news clue discovery method based on clue clusters, which involves news discovery. It obtains news from the UGC platform, extracts the event category and argument of each news based on the set trigger words, and uses the clues in the clue cluster to classify and save the news obtained from the UGC platform. The clues in the clue cluster are predicted in terms of value using a news value prediction model, and are weighted with the UGC platform trust score to obtain a news value prediction score, and generate a real-time clue list, which solves the problems of difficulty in data capture of news events and low timeliness in the prior art. The present invention is suitable for news discovery.

Description

Technical Field

[0001] The present invention relates to news discovery, and in particular to a streaming news clue discovery method based on clue clusters. Background Art

[0002] The production of a news report typically involves a complete process of topic planning, source material collection, drafting, editing, and finalization, as well as review and publication. With the development of the internet, online events have become a primary source of news topics. Advanced lead discovery systems can more quickly and accurately identify useful news topics within the vast amount of online information.

[0003] Existing news lead discovery methods can be broadly categorized into two types: analysis and identification based on dissemination paths and event identification. The former relies on large amounts of real-time user data on reposts, likes, and comments, placing unrealistic demands on both the volume and timeliness of data capture. Currently, only large user-generated content platforms like Sina Weibo possess data sufficient for dissemination path-based analysis. The latter is the preferred approach for most companies without real-time data. However, event identification requires extensive manual definition and data annotation upfront, lacks timely coverage for novel event categories, and results in lengthy iteration cycles. Summary of the Invention

[0004] The technical problem solved by the present invention is to provide a streaming news clue discovery method based on clue clusters to solve the problems of difficulty in data capture of news events and low timeliness in the existing technology.

[0005] The present invention solves the above technical problems by adopting a technical solution: a streaming news clue discovery method based on clue clusters, comprising the following steps:

[0006] S01. Obtain news from UGC platforms;

[0007] S02. Extract news event categories and arguments based on set trigger words;

[0008] S03. If the event category is extracted, determine whether the extracted event category and argument match an existing clue in the clue cluster, wherein the clue includes the event category and argument. If they match, place the news item in the matched clue. If they do not match, place the extracted event category and argument as a new clue in the clue cluster.

[0009] S04. If the event category is not extracted, extract the elements of the news, which include time, place, and people. Use multimodal deep matching technology to determine the matching degree between the news and the existing news in the clue cluster. If the person and place elements are the same, the time element difference is within a preset value, and the matching degree is greater than a threshold, place the news in the event category of the matched existing news. Otherwise, place the elements of the news as a new clue in the clue cluster, and place the news in the new clue.

[0010] S05. Use the news value prediction model to predict the value of the clues in the clue cluster to obtain a news value score, and weight it with the UGC platform trust score to obtain a news value prediction score, sort it, and generate a real-time clue list.

[0011] Furthermore, obtaining news from the UGC platform includes the following steps:

[0012] S101. Sort out the major UGC platforms and rate the credibility of each one;

[0013] S102. Dynamically allocate crawling resources based on credibility, build a real-time news crawling algorithm, and obtain news from each UGC platform;

[0014] S103. Construct a sensitive detection algorithm based on dictionaries, pinyin, variant characters, and deep learning to filter news, and construct a false information detection algorithm based on rules, knowledge graphs, and deep learning methods to filter news again;

[0015] S104. Perform field cleaning, standardization, and deduplication processing on the news.

[0016] Furthermore, the sensitive detection algorithm is used to filter out pornographic, violent and advertisement-related content, and the false information detection algorithm is used to filter out false information.

[0017] Furthermore, in step S02, extracting the event category and arguments of the news based on the set trigger words includes the following steps:

[0018] S201, classifying sentences in the news into event categories according to pre-set event trigger words;

[0019] S202, using the joint model DMCNN to extract open domain events at the sentence level to obtain sentence-level event arguments;

[0020] S203. Calculate a weighted score based on the importance of the argument entity words in the sentence, and select the sentence argument with the highest score as the news event argument.

[0021] Furthermore, in step S03, whether the extracted event category and argument match the existing clues in the clue cluster is determined by Single-Pass.

[0022] Furthermore, in step S04, extracting the elements of the news includes the following steps:

[0023] S401. Embed the news title and text sentences into semantic vectors based on BERT. Take the title content as the center vector and use TextRank to model and score the text sentences to obtain the importance score of each sentence.

[0024] S402: For each sentence, entity extraction is performed using a combination of a dictionary and deep learning based on bidirectional LSTM+CRF to obtain entity scores for different news elements in each sentence;

[0025] S403. Utilize the confidence of the entity extracted from the title content and the importance of the sentence to calculate the weighted score of the entity of the same news element in each sentence, and select the entity with the highest score as the entity of the same news element.

[0026] Furthermore, in step S04, obtaining the matching degree between the news and the existing news in the clue cluster using multimodal deep matching technology includes the following steps:

[0027] S4001, obtaining the character entity of the news. If the character entity of the news is inconsistent with the news character in the clue cluster, the matching degree is 0;

[0028] S4002. If the character entity of the news is consistent with the news character in the clue cluster, multiple embeddings in the news title, text, summary, picture and video are converted into semantic vectors, and a fully connected multimodal deep matching model is constructed to obtain the matching degree between the news and the existing news in the clue cluster.

[0029] Furthermore, the news value prediction model obtains a news value score by embedding the title, text, summary, picture, and video into semantic vectors and combining the news elements and the number of likes and comments using a deep prediction model.

[0030] Beneficial effects of the present invention: The present invention is based on a streaming news clue discovery method based on clue clusters, which obtains news from the UGC platform, extracts the event category and arguments of each news based on the set trigger words, and uses the clues in the clue cluster to classify and save the news obtained from the UGC platform. The clues in the clue cluster are predicted in terms of value using a news value prediction model, and are weighted with the UGC platform trust score to obtain a news value prediction score, and generate a real-time clue list, thereby solving the problems of difficulty in data capture of news events and low timeliness in the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Attachment Figure 1 It is a flow chart of the streaming news clue discovery method based on clue clusters of the present invention. DETAILED DESCRIPTION

[0032] The present invention is based on the streaming news clue discovery method of clue clusters, as shown in the attached Figure 1 As shown, the following steps are included:

[0033] S01. Obtain news from UGC platforms;

[0034] Specifically, obtaining news from the UGC platform includes the following steps:

[0035] S101. Sort out the major UGC platforms and rate the credibility of each one;

[0036] S102. Dynamically allocate crawling resources based on credibility, build a real-time news crawling algorithm, and obtain news from each UGC platform;

[0037] S103. Constructing a sensitive detection algorithm based on a dictionary, pinyin, variant characters, and deep learning to filter news, and constructing a false information detection algorithm based on rules, knowledge graphs, and deep learning methods to filter news again; the sensitive detection algorithm is used to filter out pornography, violence, and advertisements, and the false information detection algorithm is used to filter out false information;

[0038] S104. Perform field cleaning, standardization, and deduplication processing on the news.

[0039] S02. Extract news event categories and arguments based on set trigger words;

[0040] Specifically, extracting the event category and arguments of the news based on the set trigger words includes the following steps:

[0041] S201, classifying sentences in the news into event categories according to pre-set event trigger words;

[0042] S202, using the joint model DMCNN to extract open domain events at the sentence level to obtain sentence-level event arguments;

[0043] S203. Calculate a weighted score based on the importance of the argument entity words in the sentence, and select the sentence argument with the highest score as the news event argument.

[0044] S03. If the event category is extracted, determine whether the extracted event category and argument match an existing clue in the clue cluster, wherein the clue includes the event category and argument. If they match, place the news item in the matched clue. If they do not match, place the extracted event category and argument as a new clue in the clue cluster.

[0045] Specifically, determining whether the extracted event categories and arguments match the existing clues in the clue cluster is achieved through Single-Pass.

[0046] S04. If the event category is not extracted, extract the elements of the news, which include time, place, and people. Use multimodal deep matching technology to determine the matching degree between the news and the existing news in the clue cluster. If the person and place elements are the same, the time element difference is within a preset value, and the matching degree is greater than a threshold, place the news in the event category of the matched existing news. Otherwise, place the elements of the news as a new clue in the clue cluster, and place the news in the new clue.

[0047] Specifically, extracting the elements of the news includes the following steps:

[0048] S401. Embed the news title and text sentences into semantic vectors based on BERT. Take the title content as the center vector and use TextRank to model and score the text sentences to obtain the importance score of each sentence.

[0049] S402: For each sentence, entity extraction is performed using a combination of a dictionary and deep learning based on bidirectional LSTM+CRF to obtain entity scores for different news elements in each sentence;

[0050] S403. Utilize the confidence of the entity extracted from the title content and the importance of the sentence to calculate the weighted score of the entity of the same news element in each sentence, and select the entity with the highest score as the entity of the same news element.

[0051] The multimodal deep matching technology is used to obtain the matching degree between the news and the existing news in the clue cluster, including the following steps:

[0052] S4001, obtaining the character entity of the news. If the character entity of the news is inconsistent with the news character in the clue cluster, the matching degree is 0;

[0053] S4002. If the character entity of the news is consistent with the news character in the clue cluster, multiple embeddings in the news title, text, summary, picture and video are converted into semantic vectors, and a fully connected multimodal deep matching model is constructed to obtain the matching degree between the news and the existing news in the clue cluster.

[0054] S05. Use the news value prediction model to predict the value of the clues in the clue cluster to obtain a news value score, and weight it with the UGC platform trust score to obtain a news value prediction score, sort it, and generate a real-time clue list.

[0055] Specifically, the news value prediction model embeds the title, text, summary, picture, and video into semantic vectors, and uses a deep prediction model to predict news value scores based on news elements and the number of likes and comments. Based on the real-time clue list, the latest and most valuable news can be known.

Claims

1. A streaming news clue discovery method based on clue clusters, characterized by: The following steps are involved: S01. Obtain news from UGC platforms; S02. Extract news event categories and arguments based on set trigger words; S03. If the event category is extracted, determine whether the extracted event category and argument match an existing clue in the clue cluster, wherein the clue includes the event category and argument. If they match, place the news item in the matched clue. If they do not match, place the extracted event category and argument as a new clue in the clue cluster. S04. If the event category is not extracted, extract the elements of the news, which include time, place, and people. Use multimodal deep matching technology to determine the matching degree between the news and the existing news in the clue cluster. If the person and place elements are the same, the time element difference is within a preset value, and the matching degree is greater than a threshold, place the news in the event category of the matched existing news. Otherwise, place the elements of the news as a new clue in the clue cluster, and place the news in the new clue. S05. Use the news value prediction model to predict the value of the clues in the clue cluster to obtain a news value score, and weight it with the UGC platform trust score to obtain a news value prediction score, sort it, and generate a real-time clue list.

2. The streaming news clue discovery method based on clue clusters according to claim 1 is characterized in that: Obtaining news from UGC platforms involves the following steps: S101. Sort out the major UGC platforms and rate the credibility of each one; S102. Dynamically allocate crawling resources based on credibility, build a real-time news crawling algorithm, and obtain news from each UGC platform; S103. Construct a sensitive detection algorithm based on dictionaries, pinyin, variant characters, and deep learning to filter news, and construct a false information detection algorithm based on rules, knowledge graphs, and deep learning methods to filter news again; S104. Perform field cleaning, standardization, and deduplication processing on the news.

3. The streaming news clue discovery method based on clue clusters according to claim 2 is characterized in that: The sensitive detection algorithm is used to filter out pornographic, violent and advertisement-related information, and the false information detection algorithm is used to filter out false information.

4. The streaming news clue discovery method based on clue clusters according to any one of claims 1 to 3, characterized in that: In step S02, extracting the event category and arguments of the news based on the set trigger words includes the following steps: S201, classifying sentences in the news into event categories according to pre-set event trigger words; S202, using the joint model DMCNN to extract open domain events at the sentence level to obtain sentence-level event arguments; S203. Calculate a weighted score based on the importance of the argument entity words in the sentence, and select the sentence argument with the highest score as the news event argument.

5. The streaming news clue discovery method based on clue clusters according to any one of claims 1 to 3, characterized in that: In step S03, whether the extracted event category and argument match the existing clues in the clue cluster is determined by Single-Pass.

6. The streaming news clue discovery method based on clue clusters according to any one of claims 1 to 3, characterized in that: In step S04, obtaining the matching degree between the news and the existing news in the clue cluster using the multimodal deep matching technology includes the following steps: S4001, obtaining the character entity of the news. If the character entity of the news is inconsistent with the news character in the clue cluster, the matching degree is 0; S4002. If the character entity of the news is consistent with the news character in the clue cluster, multiple embeddings in the news title, text, summary, picture and video are converted into semantic vectors, and a fully connected multimodal deep matching model is constructed to obtain the matching degree between the news and the existing news in the clue cluster.

7. The streaming news clue discovery method based on clue clusters according to any one of claims 1 to 3, characterized in that: The news value prediction model obtains a news value score by embedding the title, text, summary, picture, and video into semantic vectors and combining the news elements and the number of likes and comments using a deep prediction model.

Images