Hot event mining method and system

Abstract

The invention provides a hot event mining method and system, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining news website data comprising a plurality of pieces of news, and vectorizing a news title of each piece of news to obtain a title vector; based on the title vector, dividing all news of the news website data into a plurality of clusters through a clustering method; performing importance evaluation on news in each cluster according to the news content, then selecting a most important news title from each cluster, and extracting a news trunk from the most important news title to form a candidate event list; the number of news in the cluster is used as the popularity of the events, and the events in the candidate event list are sorted, so that final hot events are screened out; according to the method, the candidate events are selected based on the word frequency-reverse file frequency of the keyword, and information in a news title is concisely focused by extracting a news trunk, so that the accuracy and robustness of hot event mining are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a hot event mining method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the advent of the information age, the amount of information that needs to be processed has increased exponentially, and people are paying more and more attention to hot events. At the same time, the increase in the amount of information has also made it more difficult to discover hot events.

[0004] In the related art, the mainstream method of hot event mining is based on clustering algorithms. For example, the daily corpus is vectorized on a daily basis, and then a list of candidate events is obtained through clustering methods. The candidate events are then filtered and sorted using methods such as heat to obtain the final hot events. In this process, there are two problems: 1) The construction of the event list is usually selected by the distance within the cluster, without considering the impact of the number and frequency of word appearances in the hot event mining scenario on the results, resulting in inaccurate selection of candidate events; 2) The final result is displayed in the form of the original news headlines, resulting in the extracted hot events being not concise and focused, and the characterization of the hot events is inaccurate. Therefore, the existing hot event mining methods are not accurate and robust enough. Summary of the invention

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides a hot event mining method and system, which selects candidate events based on the word frequency-inverse document frequency of keywords, and concisely focuses on the information in the news title by extracting the news trunk, thereby improving the accuracy and robustness of hot event mining.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a hot event mining method.

[0008] A hot event mining method, comprising:

[0009] Get news website data including multiple news items, each of which consists of a news title and news content;

[0010] Vectorize the news title of each news item to obtain a title vector;

[0011] Based on the title vector, all news in the news website data are divided into multiple clusters through clustering method;

[0012] According to the content of the news, the importance of the news in each cluster is evaluated, and then the most important news headline is selected from each cluster. The main news trunk is extracted from the most important news headlines to form a list of candidate events;

[0013] The number of news in the cluster is used as the heat of the event, and the events in the candidate event list are sorted to screen out the final hot events;

[0014] The importance assessment is to extract keywords from news content and evaluate the importance of news based on the keyword frequency-inverse document frequency.

[0015] Furthermore, the acquisition of the news website data is to collect news portal websites, crawl the news on the news portal websites, and form the news website data.

[0016] Furthermore, the vectorization uses Sentence-bert to vectorize the news headlines.

[0017] Furthermore, the clustering method uses a Singlepass clustering method to cluster the title vectors.

[0018] Furthermore, the extracting of keywords from the news content is to use UIE to extract keywords from the news content, including news figures, organizations, places, and events.

[0019] Furthermore, the keyword term frequency-inverse document frequency is calculated by TF-IDF.

[0020] Furthermore, the extracting of the news trunk is to extract subject-verb-object triples from the news title through syntactic analysis, and merge the triples as the news trunk.

[0021] A second aspect of the present invention provides a hot event mining system.

[0022] A hot event mining system, comprising:

[0023] The acquisition module is configured to: acquire news website data including a plurality of news items, each of which consists of a news title and news content;

[0024] The vectorization module is configured to: vectorize the news title of each news item to obtain a title vector;

[0025] The clustering module is configured to: divide all news of the news website data into multiple clusters based on the title vector by clustering method;

[0026] The selection module is configured to: evaluate the importance of news in each cluster according to the content of the news, and then select the most important news headline from each cluster, and extract the news trunk from the most important news headlines to form a candidate event list;

[0027] The sorting module is configured to: use the number of news in the cluster as the heat of the event, sort the events in the candidate event list, and thus filter out the final hot events;

[0028] The importance assessment is to extract keywords from news content and evaluate the importance of news based on the keyword frequency-inverse document frequency.

[0029] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a hot event mining method as described in the first aspect of the present invention.

[0030] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a hot event mining method as described in the first aspect of the present invention are implemented.

[0031] One or more of the above technical solutions have the following beneficial effects:

[0032] After vectorizing the news headlines, the present invention clusters the news headlines, which can effectively summarize the news content of the same event; extract keywords from the news, calculate the word frequency-inverse file frequency to evaluate the importance of keywords, and use it as the basis for selecting events, which can ensure the correct selection of events and improve the fault tolerance of the algorithm; finally, the main information of the extracted sentences is used to make the news hot events more concise and accurate.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 This is a flow chart of the method of the first embodiment. DETAILED DESCRIPTION

[0036] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0037] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0038] Embodiment 1

[0039] In one embodiment of the present disclosure, a hot event mining method is provided. Figure 1 As shown, the following steps are included:

[0040] Step S1: Acquire news website data including multiple news items, each of which consists of a news title and news content;

[0041] Furthermore, the acquisition of the news website data is to collect news portal websites and news on the news portal websites to form the news website data.

[0042] The collection here can adopt the method of scheduled crawling by crawlers to crawl news information from the collected news portals, including news titles and news content.

[0043] Step S2: vectorize the news title of each news item to obtain a title vector;

[0044] The vector encoded by Bert is anisotropic, that is, the model learns inconsistent features in different dimensions, which leads to bias and affects the results when processing certain tasks. In addition, the spatial distribution of word vectors is affected by word frequency, low-frequency words are sparsely distributed, and high-frequency words are densely distributed, which is not conducive to similarity calculation. Therefore, the construction of Bert makes it unsuitable for unsupervised tasks such as semantic similarity search and clustering. This embodiment uses Sentence-bert to complete the vectorization of news headlines, specifically:

[0045] Sentence-bert uses twin networks and a three-layer network structure to generate semantic vectors. The closer the vector distance is, the more semantically similar the sentences are. Therefore, the cosine similarity, Euclidean distance and other methods can be used to calculate the sentence similarity. At the same time, the computational overhead is greatly reduced, making it more suitable for clustering tasks.

[0046] Step S3: Based on the title vector, all news in the news website data are divided into multiple clusters by clustering method;

[0047] The clustering method uses the Singlepass clustering method to cluster the title vectors, specifically:

[0048] Use the first news title as the cluster center to generate a cluster.

[0049] Use cosine similarity to calculate the similarity between the second news headline vector and the cluster center. If it is less than the threshold, it means that they do not belong to the same type of event, and a new cluster with the second news as the cluster center is established; if it is greater than the threshold, the two news belong to the same event, and the second news is added to the cluster. At the same time, the cluster center is recalculated by calculating the average value.

[0050] Similarly, the subsequent news is calculated in turn for similarity with the existing clusters (events). If they are similar, they are classified into the same event and the cluster center is updated. If they are not similar, a new cluster is established.

[0051] Step S4: According to the content of the news, the importance of the news in each cluster is evaluated, and then the most important news headline is selected from each cluster, and the news trunk is extracted from the most important news headlines to form a candidate event list;

[0052] The importance assessment is to extract keywords from news content and evaluate the importance of news based on the keyword frequency-inverse document frequency.

[0053] Furthermore, the extracting of keywords from the news content is to use UIE to extract keywords from the news content, including news figures, organizations, places, and events.

[0054] The UIE model is a unified information extraction model that extracts keywords such as news figures, organizations, locations, and events. The specific steps are as follows:

[0055] Label a certain amount of news content, marking the news figures, organizations, places, events and other keywords in the content;

[0056] Based on the annotated news content, the UIE model is fine-tuned until convergence is achieved;

[0057] Use the convergent UIE model to extract news figures, organizations, places, and events from news content.

[0058] Furthermore, the keyword frequency-inverse document frequency is calculated by TF-IDF, and the formula is:

[0059] TF-IDF=TF(d,w)*IDF(w)

[0060] Wherein, TF(d,w) is the word frequency of w in document d, IDF(w) is the inverse document frequency, which is obtained by dividing the total number of documents by the number of documents containing word w, and then taking the logarithm of the quotient to the base 10. In this embodiment, the news of each cluster is analyzed, each news is regarded as a document d, and the keywords such as people and institutions extracted in the above text are regarded as word w. The importance of each keyword is calculated, and the keywords in the news titles are accumulated to obtain the importance score of each title, and the highest score is selected as the representative news.

[0061] Furthermore, the extraction of the news trunk is centered on the predicate, and through syntactic analysis, subject-predicate-object triples are extracted from the news title, and the triples are combined as the news trunk, specifically:

[0062] (1) Use the stanza tool to perform syntactic analysis on news headlines, extract the dependency information of each word, and represent it with a triple of (word, dependency index, dependency relationship) to form a dependency syntactic analysis tree. The syntactic analysis tree describes the dependency relationship between each word in each news headline. For example, the news headline is "Xiao Ming wins Olympic gold medal". After word segmentation, it consists of four words: Xiao Ming, win, Olympics, and gold medal. The indexes of the four words are 1, 2, 3, and 4 respectively. "Win" is the root node and does not depend on any word, so its dependency index is 0. "Xiao Ming" and "gold medal" depend on "win", so its dependency index is 2. "Olympics" is used to modify gold medal, so it depends on "gold medal". Therefore, the final syntactic analysis tree represented by an array is: [("Xiao Ming", 2, "subject-predicate relationship"), ("win", 0, "root node"), ("Olympics", 4, "noun modification"), ("gold medal", 2, "verb-object relationship")].

[0063] (2) Based on the syntactic analysis tree, a dependency dictionary is constructed for each word in the news title. The dictionary is used to store the words and relationships that depend on the word. The key is the dependency relationship and the value is the dependent word. The dependency dictionary in the above example is:

[0064] "Xiaoming": {}

[0065] "obtain": {"subject-predicate relationship": [Xiao Ming], "verb-object relationship": [gold medal]}

[0066] "Olympic Games": {}

[0067] "gold medal": {"noun modification": [Olympic Games]}

[0068] (3) Traverse each word. If the corresponding dependency relation dictionary of the word contains both subject-predicate relation and verb-object relation, then form a subject-predicate-object triple with the word and the words corresponding to the subject-predicate and verb-object relations, merge the subject-predicate-object triples, and merge the modifying words. For example, in the above example of the dependency relation dictionary, only "obtain" contains both subject-predicate relation and verb-object relation. So the extracted subject-predicate-object triple is ("Xiaoming", "obtain", "gold medal"). During the process of merging the triples, it is found that there is a modifying relation, so they are merged together to obtain the final news backbone "Xiaoming obtains the Olympic gold medal".

[0069] In addition, to avoid information loss and ensure smooth sentences, multiple triple relation extraction rules are constructed, including: adverb-verb relation, prepositional-object relation, object preposition, etc. Example of prepositional-object relation: Xiaoming was born in Beijing. "In" and "Beijing" belong to the prepositional-object relation. Therefore, these two words are merged, and the final output news backbone is: Xiaoming was born in Beijing, rather than Xiaoming was born Beijing.

[0070] Step S5: Use the number of news within the cluster as the heat of the event, sort the events in the candidate event list, and thus screen out the final hot events;

[0071] Embodiment 2

[0072] In one embodiment of the present disclosure, a hot event mining system is provided, including:

[0073] An acquisition module, configured to: acquire news website data including multiple news, and each news consists of a news title and news content;

[0074] A vectorization module, configured to: vectorize the news title of each news to obtain a title vector;

[0075] A clustering module, configured to: based on the title vector, divide all the news in the news website data into multiple clusters by a clustering method;

[0076] A selection module, configured to: evaluate the importance of the news within each cluster according to the news content, then select the most important news title from each cluster, and extract the news backbone from the most important news title to form a candidate event list;

[0077] A sorting module, configured to: use the number of news within the cluster as the heat of the event, sort the events in the candidate event list, and thus screen out the final hot events;

[0078] Among them, the importance evaluation is to extract keywords from the news content and evaluate the importance of the news based on the word frequency-inverse document frequency of the keywords.

[0079] Embodiment 3

[0080] The purpose of this embodiment is to provide a computer-readable storage medium.

[0081] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in a hot event mining method as described in Embodiment 1 of the present disclosure.

[0082] Embodiment 4

[0083] The purpose of this embodiment is to provide an electronic device.

[0084] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a hot event mining method as described in Embodiment 1 of the present disclosure are implemented.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A hot event mining method, characterized in that: include: Get news website data including multiple news items, each of which consists of a news title and news content; Vectorize the news title of each news item to obtain a title vector; Based on the title vector, all news in the news website data are divided into multiple clusters through clustering method; According to the content of the news, the importance of the news in each cluster is evaluated, and then the most important news headline is selected from each cluster. The main news trunk is extracted from the most important news headlines to form a list of candidate events; The number of news in the cluster is used as the heat of the event, and the events in the candidate event list are sorted to screen out the final hot events; The importance assessment is to extract keywords from news content and evaluate the importance of news based on the keyword frequency-inverse document frequency.

2. A hot event mining method according to claim 1, characterized in that: The acquisition of the news website data is to collect news portal websites, crawl news on the news portal websites, and form the news website data.

3. A hot event mining method as claimed in claim 1, characterized in that: The vectorization uses Sentence-bert to vectorize the news headlines.

4. A hot event mining method according to claim 1, characterized in that: The clustering method uses the Singlepass clustering method to cluster the title vectors.

5. A hot event mining method according to claim 1, characterized in that: The extracting of keywords from news content is to use UIE to extract keywords from news content, including news figures, organizations, places, and events.

6. A hot event mining method according to claim 1, characterized in that: The keyword frequency-inverse document frequency is calculated by TF-IDF.

7. A hot event mining method according to claim 1, characterized in that: The extracting of the news trunk is to extract subject-verb-object triples from the news title through syntactic analysis, and merge the triples as the news trunk.

8. A hot event mining system, characterized in that: include: The acquisition module is configured to: acquire news website data including a plurality of news items, each of which consists of a news title and news content; The vectorization module is configured to: vectorize the news title of each news item to obtain a title vector; The clustering module is configured to: divide all news of the news website data into multiple clusters based on the title vector by clustering method; The selection module is configured to: evaluate the importance of news in each cluster according to the content of the news, and then select the most important news headline from each cluster, and extract the news trunk from the most important news headlines to form a candidate event list; The sorting module is configured to: use the number of news in the cluster as the heat of the event, sort the events in the candidate event list, and thus filter out the final hot events; The importance assessment is to extract keywords from news content and evaluate the importance of news based on the keyword frequency-inverse document frequency.

9. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; a processor for executing the computer-readable instructions; Wherein, when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is executed.

10. A storage medium, characterized in that: The computer-readable instructions are non-transitorily stored, wherein when the computer-readable instructions are executed by a computer, the method of any one of claims 1 to 7 is performed.

Images