A custom event warning monitoring method

Abstract

The present invention belongs to the technical field of network information mining, and specifically relates to a custom event early warning monitoring method. The method uses custom event configuration and early warning configuration, first collects data from multiple channels, mainstream media, and social accounts; then pre-processes the data, uses Elasticsearch to achieve near-real-time matching of events and data, and pushes the matching data to kafka for early warning monitoring by the early warning monitoring system; the early warning monitoring system matches the amount of early warning articles through the inverted index according to the early warning configuration, meets the triggering conditions, and sends early warning information in time. The method can provide a richer and faster monitoring form for the business, can quickly configure the latest events and hot events, and analyze and monitor the events, grasp the high-incidence period of events and the intervention effect, and improve application flexibility and rapidity.

Description

Technical Field

[0001] The invention belongs to the technical field of network information mining, and in particular relates to a user-defined event early warning monitoring method. Background Art

[0002] With the continuous development of information technology, news clients and various social media have become the first source of information for the public. However, due to the huge amount of information, the complexity of information screening, and the endless emergence of various events, multi-channel mainstream media collection, event identification, and early warning monitoring have become a common means.

[0003] At present, event early warning monitoring mainly conducts special event monitoring on certain regions and topics. For example, event monitoring is established by customizing configuration keywords, analyzing the overall content of events, and displaying the total data volume, keywords and other analysis contents of the events. However, this analysis method has the following disadvantages:

[0004] 1. Data collection areas are unevenly distributed and channels are not perfect;

[0005] 2. Unable to quickly customize new events and event monitoring methods;

[0006] 3. It is impossible to monitor custom event warnings in near real time. Summary of the invention

[0007] In view of the defects and problems of current event warning monitoring methods, such as one-sided data collection, inability to quickly customize new events and event monitoring methods, and inability to perform near real-time warning monitoring of customized events, the present invention provides a customized event warning monitoring method.

[0008] The solution adopted by the present invention to solve the technical problem is: a customized event early warning monitoring method, comprising the following steps:

[0009] Step 1: Collect articles from around the world within a certain period of time and classify the topics of the articles; and according to the event monitoring configuration content, improve the channel collection configuration and optimize the collection content.

[0010] Step 2: Data preprocessing:

[0011] (1) Filter out duplicate data in article information data based on URLs, and filter out non-topic content and spam content based on simple keyword rules;

[0012] (2) Perform text analysis on the article information data to form labels including word segmentation, keyword extraction, subject extraction, and basic sentiment analysis;

[0013] (3) Perform word segmentation management based on Elasticsearch storage;

[0014] Step 3: Configure event monitoring information, initialize event-related information, and use ES's inverted index to retrieve articles related to the configured keywords and excluded keywords according to the fields of the configured monitoring information, and store them in the event-related ES index; use minute-level monitoring to monitor new data matching each event every minute, monitor the latest event data in near real time, and perform statistical analysis and display on event data; based on the event configuration information, detect whether the data collection rules have deviations for event-related collection, and automatically improve the channel collection configuration. If the collection configuration cannot be improved, inform the administrator of the possible data collection deviations in event monitoring, and the administrator will improve the collection according to the situation.

[0015] Step 4: Warning information configuration: configure the event, validity period, keywords, excluded keywords, warning monitoring personnel, warning cycle, warning frequency, warning level, and warning notification method; cache the warning configuration information to Redis for use by the warning monitoring system;

[0016] Step 5: Identify the language type of the warning information, and use the corresponding word segmentation method to segment the title and content according to the language type; use the inverted index method to match the content with the warning configuration information. When a match is hit, it is recorded in the redis cache for the program to monitor the number of warnings within the effective time, and at the same time monitor whether the number of warnings reaches the configuration to push the warning message;

[0017] Step 6: Receive the warning information from the early warning monitoring system, store it in the database, and send the warning information to the management personnel in a timely manner according to the configured early warning notification method; then conduct an overall analysis based on the early warning information and perform a visual display.

[0018] In the above-mentioned custom event early warning monitoring method, the global articles in step 1 include articles and updates from multi-channel mainstream news media, social media, and social influencers.

[0019] The above-mentioned custom event warning monitoring method uses a general news collector to configure media website addresses for automatic collection and extraction of traditional media data, and then performs a sample check on the collection results. For data with poor collection results, specific rules can be configured for intervention; for social media, a customized collector is used to perform automatic collection using a manually maintained account pool.

[0020] In the above-mentioned custom event warning monitoring method, in step 1, a streaming text processing single-pass clustering method is used to classify the topics of the article. The classification method is: read a new text and convert the text into a vector; judge the similarity between the vector and the existing cluster. If it is greater than the threshold, the cluster is added; if it is less than the threshold, judge whether the number of clusters exceeds the set cluster value. If it does not exceed the cluster value, create a cluster directly; otherwise, delete the most useless existing cluster and then create a new cluster.

[0021] For the above-mentioned custom event warning monitoring method, if the text is long, tfidf can be used directly for judgment, or the cosine value of the text vector can be used for judgment, or jaccad similarity can be used directly for judgment.

[0022] In the above-mentioned custom event early warning monitoring method, the monitoring information configured in step three includes monitoring period, channel, sentiment tendency, keywords, and excluded keywords.

[0023] In the above-mentioned custom event early warning monitoring method, the content analyzed and displayed in step three includes event summary, event portrait, time series analysis, trend analysis, and hot netizens.

[0024] In the above-mentioned custom event warning monitoring method, the language types in step 5 include Chinese and English. The standard word segmenter is used for English word segmentation, and the ik word segmenter is used for Chinese word segmentation.

[0025] In the above-mentioned custom event warning monitoring method, the visualization display in step 6 includes the total number of daily warnings, the number of daily warning topics, the number of daily single topic warnings, and the number of hot topic warnings.

[0026] Beneficial effects of the present invention: The present invention uses custom event configuration and warning configuration, first collects data from multiple channels, mainstream media, and social accounts; then preprocesses the data, uses Elasticsearch to achieve near real-time matching of events and data, and pushes the matching data to kafka for early warning monitoring by the early warning monitoring system; the early warning monitoring system matches the amount of early warning articles through the inverted index according to the early warning configuration, meets the triggering conditions, and sends early warning information in a timely manner. This method can provide a richer and faster monitoring form for the business, can quickly configure the latest events and hot events, and analyze and monitor the events, grasp the high-incidence period of events and the intervention effect, and improve application flexibility and rapidity.

[0027] The method of the present invention can grasp the event situation in near real time, monitor event abnormalities in a timely manner, and can realize near real-time data collection (minute level), near real-time event monitoring (minute level), and real-time early warning monitoring, achieving a near real-time effect as a whole; through event analysis and early warning monitoring, multi-faceted control of events can be achieved, which is convenient for business personnel to make decisions and judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION

[0029] In view of the defects and problems of the current event early warning monitoring method, such as one-sided data collection, inability to quickly customize new events and event monitoring methods, and inability to monitor customized event early warnings in near real time, the present invention provides a customized event early warning monitoring method. The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0030] Example 1: This example provides a custom event warning monitoring method, which is aimed at custom event analysis and warning monitoring. First, multi-channel, mainstream media, and social account data are collected, and then the data is pre-processed. With the help of Elasticsearch, events and data are matched in near real time, and the matching data is pushed to Kafka for early warning monitoring by the early warning monitoring system. The early warning monitoring system matches the amount of early warning articles through the inverted index according to the early warning configuration, meets the triggering conditions, and sends early warning information in time. Figure 1 As shown, the method specifically includes the following steps.

[0031] Step 1: Data Collection

[0032] Collect articles from around the world during the time period, including articles and updates from mainstream news media, social media, and social influencers. Then use the streaming text processing single-pass clustering method to classify the topics of the articles.

[0033] The basic idea is: read the article data in a certain order, and compare the new data read each time with the data that has been read and clustered. If the corresponding approximate group is found according to certain rules, the new data will be classified into this class; if not, the new data will be regarded as a new class; repeat the process until all the data has been read.

[0034] The algorithm flow is:

[0035] (1) Read a new text and convert it into a vector;

[0036] (2) Determine the similarity between the vector and the existing clusters. Regarding the calculation of similarity, if the text is long, you can directly use tfidf for judgment; or use the cosine value of the text vector for judgment, or directly use jaccad similarity for judgment;

[0037] If it is greater than the threshold, the cluster joining is completed;

[0038] If it is less than the threshold, determine whether the number of clusters exceeds the set cluster value (control the number of clusters to avoid too many clusters). If it does not exceed the cluster value, create a cluster directly; otherwise, delete the most useless existing cluster (determined by time and cluster size), and then create a new cluster.

[0039] Step 2: Data preprocessing

[0040] (1) Filter out duplicate data in article information data based on URLs, and filter out non-topic content and spam content based on simple keyword rules;

[0041] (2) Perform text analysis on the article information data to form labels including word segmentation, keyword extraction, subject extraction, and basic sentiment analysis;

[0042] (3) Based on Elasticsearch storage and word segmentation management, it is easy to search.

[0043] Step 3: Event configuration, including the following steps:

[0044] (1) Configure event monitoring information, including monitoring period, channel, sentiment tendency, keywords, and excluded keywords;

[0045] (2) Initialize event-related information, match fields such as channel and sentiment according to the configuration, use ES's inverted index to retrieve articles related to the configured keywords and excluded keywords, and store them in the event-related ES index;

[0046] (3) Monitor the latest event data in near real time. Use minute-level monitoring to monitor new data matching each event every minute. Use time period retrieval to obtain event-related data. Match the data in the full database that matches the time from the latest article time of the event index to the current time.

[0047] (4) Statistical analysis and display of event data, including event overview, event portrait, time series analysis, trend analysis, hot netizens, etc.;

[0048] Step 4: Warning information configuration: configure the event, validity period, keywords, excluded keywords, warning monitoring personnel, warning cycle (minute level), warning frequency, warning level, and warning notification method; cache warning configuration information to Redis for use by the warning monitoring system.

[0049] Step 5: Early warning monitoring, including the following:

[0050] (1) Identify the content language, mainly Chinese and English;

[0051] (2) Based on the identified language type, use the word segmentation method to segment the title and content;

[0052] (3) Use the inverted index method to match the content with the warning configuration information; when a match is found, it is recorded in the redis cache so that the program can monitor the number of warnings within the effective time and whether the number of warnings reaches the configuration to push the warning message;

[0053] Step 6: Receiving and analyzing early warning information, including the following steps:

[0054] (1) Receive warning information from the early warning monitoring system, store the warning information in the database, and send the warning information to the management personnel in a timely manner according to the configured early warning notification method;

[0055] (2) Provide warning management function to facilitate administrators to change the warning processing status;

[0056] (3) Warning analysis: An overall analysis is conducted based on the warning information, and a visual display is provided, including statistical analysis of business items such as the total number of daily warnings, the number of daily warning topics, the number of daily single topic warnings, and the number of hot topic warnings.

Claims

1. A custom event warning monitoring method, Features: The following steps are involved: Step 1: Collect articles from around the world within a certain period of time and classify the topics of the articles; and improve the channel collection configuration and optimize the collection content according to the event monitoring configuration content; The streaming text processing single-pass clustering method is used to classify the topics of the articles. The classification method is: (1) Read a new text and convert it into a vector; (2) Determine the similarity between the vector and the existing cluster. If the text is long, you can directly use tfidf for judgment, or use the cosine value of the text vector for judgment, or directly use jaccad similarity for judgment; If it is greater than the threshold, the cluster joining is completed; If it is less than the threshold, it is determined whether the number of clusters exceeds the set cluster value. If it does not exceed the cluster value, the cluster is created directly; Otherwise, delete the most useless existing cluster and then create a new cluster; Step 2: Data preprocessing: (1) Filter out duplicate data in article information data based on URLs, and filter out non-topic content and spam content based on simple keyword rules; (2) Perform text analysis on the article information data to form labels including word segmentation, keyword extraction, subject extraction, and basic sentiment analysis; (3) Perform word segmentation management based on Elasticsearch storage; Step 3: Configure event monitoring information, initialize event-related information, and use ES's inverted index to retrieve articles related to the configured keywords and excluded keywords based on the fields of the configured monitoring information, and store them in the event-related ES index; Use minute-level monitoring to monitor new data matching each event every minute, monitor the latest event data in near real time, and perform statistical analysis and display of event data; based on event configuration information, detect whether data collection rules have deviations in event-related collection, and automatically improve channel collection configuration. If the collection configuration cannot be improved, inform the administrator of the possible data collection deviations in event monitoring, and the administrator will improve the collection according to the situation; Step 4: Warning information configuration: configure the event, validity period, keywords, excluded keywords, warning monitoring personnel, warning cycle, warning frequency, warning level, and warning notification method; Cache warning configuration information to Redis for use by the warning monitoring system; Step 5: Identify the language type of the warning information, and use the corresponding word segmentation method to segment the title and content according to the language type; use the inverted index method to match the content with the warning configuration information. When a match is hit, it is recorded in the redis cache for the program to monitor the number of warnings within the effective time, and at the same time monitor whether the number of warnings reaches the configuration to push the warning message; Step 6: Receive the warning information from the early warning monitoring system, store it in the database, and send the warning information to the management personnel in a timely manner according to the configured early warning notification method; then conduct an overall analysis based on the early warning information and perform a visual display.

2. The customized event early warning monitoring method according to claim 1, Features: The global articles in step one include articles and updates from multi-channel mainstream news media, social media, and social influencers.

3. The customized event early warning monitoring method according to claim 1, Features: In step one, for traditional media data, a general news collector is used to configure the media website address for automatic collection and extraction, and then a sample check is performed on the collection results. For data with poor collection results, specific rules can be configured for intervention. For social media, a customized collector is used to perform automatic collection using a manually maintained account pool.

4. The customized event early warning monitoring method according to claim 1, Features: The monitoring information configured in step 3 includes monitoring period, channel, sentiment tendency, keywords, and excluded keywords.

5. The customized event early warning monitoring method according to claim 1, Features: The content analyzed and displayed in step three includes event summary, event portrait, time series analysis, trend analysis, and hot netizens.

6. The customized event early warning monitoring method according to claim 1, Features: In step 5, the language types include Chinese and English. The standard word segmenter is used for English, and the ik word segmenter is used for Chinese.

7. The customized event early warning monitoring method according to claim 1, Features: The visual display in step six includes the total number of daily warnings, the number of daily warning topics, the number of daily single topic warnings, and the number of hot topic warnings.

Images