A method for mining and analyzing event development context and storage medium

Abstract

The present invention relates to the fields of natural language processing and data mining, and in particular to an event development context mining and analysis method and a storage medium, comprising the following steps: cluster center point calculation; calculating a new cluster center point as an event development context node based on a news clustering result; event context construction; after clustering is completed, selecting a cluster center as a key news representing a current hot event, then tracking the evolution of the event, repeating the preceding steps, calculating the similarity between other news and the current cluster center, if the similarity exceeds a set threshold, classifying the news as the event, if the number of newly added news in the cluster reaches a certain threshold, recalculating the cluster center, and updating the event context node. The present invention constructs a dynamic event context deduction model, which can clearly display the development process of a hot event and its key nodes. The time series-based analysis method makes the event context more coherent and accurate.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and data mining, and in particular to an event development context mining and analysis method and a storage medium. Background Art

[0002] Existing event development context analysis methods rely on keyword extraction and time series analysis to identify important event nodes in the text, which often ignore the complex relationships between events, resulting in one-sided analysis results and failure to fully reflect the overall context of the event; some methods use pre-set templates to extract specific types of events. Although this method performs well in specific fields, it lacks flexibility and has difficulty coping with diverse event types and complex event associations; methods based on information propagation path analysis focus more on the network structure of propagation, and less on the event content itself and its causal relationship, resulting in an insufficient understanding of the development of events. Summary of the invention

[0003] In view of the above technical problems, the present invention proposes a method for mining and analyzing the development context of an event, and the specific process of the method is as follows:

[0004] S1) News text preprocessing;

[0005] S2) Feature extraction, including using the TF-IDF model to calculate the importance of words;

[0006] S3) News similarity calculation: including using news feature vectors to calculate the similarity between news texts; S4) News clustering: including using density-based DBSCAN clustering algorithm to identify clusters of arbitrary shapes, remove noise data, and determine clusters through density parameters;

[0007] S5) Cluster center point calculation: Based on the news clustering result of step S4, a new cluster center point is calculated as the development context node of the event;

[0008] S6) Event context construction: After the news clustering is completed, the event context is constructed, the cluster center is selected as the key news representing the current hot event, and then the evolution of the event is tracked; for the newly crawled news text, steps S3-S5 are repeated to calculate its similarity with the current cluster center. If the similarity exceeds the set threshold, the news is classified as the event. If the number of new news in the cluster reaches a certain threshold, the cluster center is recalculated and the event context is updated;

[0009] S7) presents the context of events as a series of orderly news, reflecting the development process of hot events.

[0010] The beneficial effects of this application are:

[0011] 1) The system can automatically update the clustering center of hot events to reflect the development and changes of events in real time. This dynamic update mechanism effectively solves the limitations of traditional static clustering methods in dealing with dynamic events;

[0012] 2) The present invention can flexibly and accurately identify and judge hot events by combining custom logic and news clustering results. The present invention does not rely on fixed templates and can be dynamically adjusted according to actual needs, which significantly improves the flexibility and adaptability of hot event detection;

[0013] 3) The present invention constructs a dynamic event context deduction model that can clearly display the development process and key nodes of hot events. This time series-based analysis method makes the event context more coherent and accurate, and provides deep insights into the development trend of events. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0015] Figure 1 The following is a schematic diagram of the technical principle of constructing the context of event development;

[0016] Figure 2 Shown is the flow chart of event development context mining and analysis. DETAILED DESCRIPTION

[0017] In order to enable those skilled in the art to better understand the technical solution of the present application, the present invention is further described in detail below in conjunction with the accompanying drawings and the best embodiment.

[0018] A method for mining and analyzing event development context includes the following steps:

[0019] S1) News text preprocessing; it includes: 1) using heuristic rules to clean the weird symbols and HTML web page tags contained in the news; 2) using Jieba word segmentation algorithm to decompose the news text into words or phrases; 3) removing common but meaningless words; 4) normalizing the words.

[0020] S2) Feature extraction, including using the TF-IDF model to calculate the importance of words;

[0021] The TF-IDF calculation formula is as follows:

[0022] TF-IDF(t,d)=TF(t,d)×IDF(t)

[0023] Where: TF(t, d) represents the frequency of word t in document d; IDF(t) represents the inverse document frequency of word t in the entire document collection, which is defined as:

[0024]

[0025] Where N is the total number of documents, |{d∈D:t∈d}| is the number of documents containing the word t; the feature vector of the news is constructed by calculating the TF-IDF value of each news text.

[0026] S3) News similarity calculation: including using news feature vectors to calculate the similarity between news texts;

[0027] This application uses cosine similarity to calculate the similarity between news texts;

[0028] The formula is as follows:

[0029]

[0030] Among them, A and B are the feature vectors of two news texts respectively; the similarity result is between 0 and 1, and the larger the value, the more similar the news is.

[0031] S4) News clustering; including using density-based DBSCAN clustering algorithm to identify clusters of arbitrary shapes, remove noise data, and determine clusters by density parameters;

[0032] The main parameters of the DBSCAN clustering algorithm include: the radius of the neighborhood ε, that is, the radius within which the density is considered reachable; the minimum number of neighbors of the core point MinPts;

[0033] DBSCAN algorithm steps:

[0034] 1) Initialization: Randomly select an unvisited point P from the dataset;

[0035] 2) Calculate the ε neighborhood: calculate all points in the ε neighborhood of point P;

[0036] 3) Core point determination: If the number of points in the ε neighborhood of P is greater than or equal to MinPts, P is regarded as a core point and a new cluster is formed; if P is not a core point, it is marked as a noise point;

[0037] 4) Expand clustering: Expand all points that are directly density-reachable from the core point P and include them in the current cluster; repeat the expansion process until the current cluster cannot be expanded further;

[0038] 5) Process the remaining points: For the unvisited points in the dataset, repeat the above steps until all points are processed.

[0039] S5) Cluster center point calculation: Based on the news clustering result of step S4, a new cluster center point is calculated as the development context node of the event;

[0040] The centroid of all points in the cluster is taken as the cluster center point, and the calculation steps are as follows:

[0041] 1) Sum of feature vectors of news text within a cluster;

[0042] Assume that in a cluster C, there are N news texts, each of which corresponds to a d-dimensional feature vector X i , where i = 1, 2, ..., N, find the sum of all news text feature vectors in the cluster:

[0043]

[0044] 2) Calculate the center of mass

[0045] The centroid is defined as the average of the eigenvectors of all points in the cluster, and its formula is:

[0046]

[0047] Among them, C centroid is the eigenvector of the centroid and is defined as the cluster center point;

[0048] 3) Select the point x closest to the centroid k :

[0049]

[0050] Among them, ||C centroid -x i || represents the Euclidean distance between vectors

[0051] S6) event context construction; after the news clustering is completed, the event context is constructed, the cluster center is selected as the key news representing the current hot event, and then the evolution of the event is tracked; for the newly crawled news text, steps S3-S5 are repeated to calculate its similarity with the current cluster center. If the similarity exceeds a set threshold, the news is classified as the event. If the number of new news in the cluster reaches a certain threshold, the cluster center is recalculated and the event context is updated; the set threshold is used by using the statistical analysis method ROC curve to find the optimal threshold, so that the classification accuracy or recall rate reaches the best balance.

[0052] S7) presents the context of events as a series of orderly news, reflecting the development process of hot events.

[0053] The advantage of this application is that it can realize dynamic tracking of event context and dynamically update the event clustering center mechanism, so that the system can track the evolution of hot events in real time. With the addition of new news, the system automatically adjusts and updates the clustering center to ensure the timeliness and continuity of event context analysis and timely reflect the latest developments of events. At the same time, it enhances the flexibility and adaptability of hot event detection. Through the combination of custom logic and news clustering results, the system can flexibly respond to different types of news events and quickly identify and judge hot events. This flexibility enables the system to maintain efficient and accurate hot event detection capabilities in a changing information environment. In addition, a clear and coherent event development context is constructed. Through the construction and deduction of event context based on time series, the system can clearly display the development process and key nodes of hot events. Users can intuitively understand the cause, development and results of the event through this context, and gain a deep insight into the development trend of the event.

[0054] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0055] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera device / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0056] The present invention has been described by the above-mentioned relevant embodiments and drawings, but the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, modifications and equivalent settings included in the spirit and scope of the claims are included in the scope of the present invention.

Claims

1. A method for mining and analyzing the development context of an event, characterized in that: The steps include: S1) News text preprocessing: S2) Feature extraction, including using the TF-IDF model to calculate the importance of words; S3) News similarity calculation: including using news feature vectors to calculate the similarity between news texts; S4) News clustering: including using density-based DBSCAN clustering algorithm to identify clusters of arbitrary shapes, remove noise data, and determine clusters through density parameters; S5) Cluster center point calculation: Based on the news clustering result of step S4, a new cluster center point is calculated as the development context node of the event; S6) Event context construction: After the news clustering is completed, the event context is constructed, the cluster center is selected as the key news representing the current hot event, and then the evolution of the event is tracked; for the newly crawled news text, steps S3-S5 are repeated to calculate its similarity with the current cluster center. If the similarity exceeds the set threshold, the news is classified as the event. If the number of new news in the cluster reaches a certain threshold, the cluster center is recalculated and the event context is updated; S7) presents the context of events as a series of orderly news, reflecting the development process of hot events.

2. The method for mining and analyzing the development context of an event according to claim 1, characterized in that: Step S1) includes: 1) using heuristic rules to clean the strange symbols and HTML web page tags contained in the news; 2) using Jieba word segmentation algorithm to decompose the news text into words or phrases; 3) removing common but meaningless words; 4) normalizing the words.

3. The method for mining and analyzing the development context of an event according to claim 1, characterized in that: Step S2) includes: using the TF-IDF model to calculate the importance of vocabulary; The TF-IDF calculation formula is as follows: TF-IDF(t,d)=TF(t,d)×IDF(t) Where: TF(t,d) represents the frequency of word t in document d; IDF(t) represents the inverse document frequency of word t in the entire document collection, which is defined as: Where N is the total number of documents, |{d∈D:t∈d}| is the number of documents containing the term, t; By calculating the TF-IDF value of each news text, the feature vector of the news is constructed.

4. The method for mining and analyzing the development context of an event according to claim 1, characterized in that: Step S3) comprises: using cosine similarity to calculate the similarity between news texts; The formula is as follows: Among them, A and B are the feature vectors of two news texts respectively; the similarity result is between 0 and 1, and the larger the value, the more similar the news is.

5. The method for mining and analyzing the development context of an event according to claim 1, characterized in that: The main parameters of the DBSCAN clustering algorithm in step S4) include: the radius of the neighborhood ε, that is, the radius within which the density is considered reachable; the minimum number of neighbors of the core point MinPts; DBSCAN algorithm steps: 1) Initialization: randomly select an unvisited point P from the data set; 2) Calculate the ε neighborhood: calculate all points in the ε neighborhood of point P; 3) Core point determination: If the number of points in the ε neighborhood of P is greater than or equal to MinPts, P is regarded as a core point and a new cluster is formed; if P is not a core point, it is marked as a noise point; 4) Expand clustering: Expand all points that are directly density-reachable from the core point P and include them in the current cluster; repeat the expansion process until the current cluster cannot be expanded further; 5) Process the remaining points: For the unvisited points in the dataset, repeat the above steps until all points are processed.

6. The method for mining and analyzing the development context of an event according to claim 1, characterized in that: Step S5) includes: taking the centroid of all points in the cluster as the cluster center point, and the calculation steps are as follows: 1) Sum of feature vectors of news text within a cluster; Assume that in a cluster C, there are N news texts, each of which corresponds to a d-dimensional feature vector X i , where i = 1, 2, ..., N, find the sum of all news text feature vectors in the cluster: 2) Calculate the center of mass The centroid is defined as the average of the eigenvectors of all points in the cluster, and its formula is: Among them, C centroid is the eigenvector of the centroid and is defined as the cluster center point; 3) Select the point x closest to the centroid k : Among them, ||C centroid -x i || represents the Euclidean distance between vectors.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images