Hotspot event detection method and device based on open database

Abstract

The invention discloses a hotspot event detection method and device based on an open database, and the method comprises the steps: carrying out the data screening based on a target database, so as to obtain an interested event; an event numerical value is adapted to any event; calculating the comprehensive strength of any event according to the distributed event numerical value and the numeric segment of the event mentioned in the article, and determining an event strength curve in a specified time period of the target area according to the calculated comprehensive strength; and for the determined event intensity curve, adopting a detection method based on double sliding windows to detect a jump point in the event intensity curve so as to determine a hotspot event based on the jump point. The event intensity quantitative expression method is designed by comprehensively considering multiple factors such as the event report number, the event influence and the event mention number.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a hot event detection method and device based on an open database. Background Art

[0002] Hot events such as demonstrations, riots and riots not only affect people's production and life, and the economic development of society, but also have varying degrees of impact on the peace and stability of the entire society and region. Research on the internal mechanism and detection and early warning of hot events can timely perceive social dynamics.

[0003] The existing hot event quantification and detection methods mainly include methods based on traditional statistical quantification, hot spot detection based on classification methods, and detection based on social media and news big data. The traditional statistical data method quantifies the intensity of hot events by manually collecting and organizing annual data, which is time-consuming, difficult, and has a low degree of automation. Hot spot detection based on classification methods requires historical real hot spot data to train the model, but the collection and production of training sets and test sets are very difficult, and the predicted classification results are highly dependent on the quality of input data and event classification methods. The method of detecting hot spots based on social media and news big data mainly includes the collection and filtering process of data source resource information, the establishment process of data information model, the clustering and classification process of network news after modeling, and finally the verification and optimization of the results.

[0004] GDELT (Global Database of Events, Language, and Tone) is a comprehensive and detailed open database. It contains a network platform for real-time monitoring of global media including print media, broadcasting, web-formatted news, etc., recording more than 800 million events from 1979 to the present. GDELT data has the potential to be used for quantitative detection of hot events. Summary of the invention

[0005] The embodiment of the present application provides a hot event detection method and device based on an open database, which comprehensively considers multiple factors such as the number of event reports, the influence of events, the number of event mentions, and designs a method for quantitatively expressing event intensity.

[0006] The present application provides a hot event detection method based on an open database, including:

[0007] Perform data screening based on the target database to obtain events of interest;

[0008] Less than zero is the first type of event. The smaller the score, the worse the potential negative impact of the event. Greater than or equal to 0 is the second type of event. The larger the score, the better the potential impact. An event value is adapted for any event.

[0009] Calculate the comprehensive intensity of any event based on the assigned event value and the numerical segment of the event mentioned in the article, and determine the event intensity curve within the specified time period of the target area based on the calculated comprehensive intensity;

[0010] For the determined event intensity curve, a detection method based on a double sliding window is adopted to detect the jump points in the event intensity curve, so as to determine the hot events based on the jump points.

[0011] Optionally, filter the data based on the target database to obtain events of interest including:

[0012] Based on the target database, select the data of the required time period and clean the data according to the required fields;

[0013] After cleaning, filter by fields associated with the target region to obtain events of interest.

[0014] Optionally, the comprehensive intensity of any event is calculated based on the assigned event value and the numerical segment of the event mentioned in the article to meet:

[0015] CEI=abs(GoldsteinScale)×NumMentions

[0016] Among them, GoldsteinScale is the assigned event value, and NumMentions is the number of event and article mentions.

[0017] Optionally, determining an event intensity curve within a specified time period of the target area according to the calculated comprehensive intensity includes:

[0018] In units of days, the event quantification standard LCEI for the target area is determined to meet:

[0019]

[0020] Among them, i represents the event ordinal number, M is the total number of events in the target area, and N is the total number of all events involving the target area in the news records of the day;

[0021] Based on the calculated LCEI of each event, the event intensity curve within the specified time period of the target area is determined in chronological order.

[0022] Optionally, for the determined event intensity curve, a detection method based on a double sliding window is used to detect a jump point in the event intensity curve, including:

[0023] Based on the event intensity curve, two relatively static continuous sliding windows are set to process the input data in segments;

[0024] The data in each segment is accumulated, and the energy ratio of the two windows is used as the decision statistic to perform jump point detection, where the energy of window a(n) and window b(n) satisfies:

[0025]

[0026]

[0027] Among them, d is the event number in the window, and L is the window length.

[0028] Optionally, for the determined event intensity curve, adopting a detection method based on a double sliding window to detect a jump point in the event intensity curve further includes:

[0029] By tracking the curve fluctuations, the adaptive detection threshold is estimated to meet the following requirements:

[0030]

[0031] Where m(n) is the energy ratio of the two windows;

[0032] Using the adaptive detection threshold value, in the process of continuous sliding window sliding detection, if the energy ratios calculated for a specified number of points are all greater than the adaptive detection threshold value, then it is determined to be a jump point, otherwise it is a noise point;

[0033] Remove noise points, update the adaptive detection threshold, and repeat the sliding window sliding detection jump point.

[0034] An embodiment of the present application also proposes a hot event discovery device based on an open database, which is characterized in that it includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, the steps of the hot event detection method based on the open database are implemented as described above.

[0035] The embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned hot event detection method based on an open database are implemented.

[0036] The embodiment of the present application comprehensively considers multiple factors such as the number of event reports, the impact of the event, the number of event mentions, and designs a method for quantitatively expressing event intensity.

[0037] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 The following is a basic flow chart of the hot event detection method according to an embodiment of the present application;

[0040] Figure 2 This is a schematic diagram of the GDELT data processing flow of the hot event detection method according to an embodiment of the present application;

[0041] Figure 3 The following is a schematic diagram of the jump point detection process of the hot event detection method according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] The present application embodiment provides a hot event detection method based on an open database, such as Figure 1 As shown, the following steps are included:

[0044] In step S101, data screening is performed based on a target database to obtain events of interest.

[0045] In some specific examples, GDELT provides massive data for analysis, but it is still necessary to perform certain analysis and processing on the data to achieve the quantitative characterization of hot events. In some examples, quantitative indicators are constructed based on three field information selected based on GDELT: the event classification field QuadClass, the event score field GoldsteinScale, and the article mention number field NumMentions. The QuadClass field specifies the main classification of event types. Each event in GDELT can ultimately be attributed to a hot event or a cooperative event, which is divided into the following four categories: 1 for verbal cooperation, 2 for material cooperation, 3 for verbal hotspots, and 4 for material hotspots. The QuadClass field can be used as an indicator for hot event screening. In some examples, data screening based on the target database to obtain events of interest (such as hot events) includes: based on the target database, selecting data for the required time period and cleaning the data according to the required fields; after cleaning, filtering through fields related to the target area to obtain events of interest.

[0046] For example, you can select one year of GDELT event database data, remove event data with duplicate attribute record values ​​such as ActionGeo_CountryCode, QuadClass, GoldsteinScale, NumMentions, and empty record values ​​of required fields, and complete data cleaning.

[0047] The data is filtered according to the target region through the ActionGeo_CountryCode and QuadClass fields (eg ActionGeo_CountryCode = RUS, QuadClass is 3 or 4).

[0048] In step S102, a score less than zero is a first-class event, and the smaller the score, the worse the potential negative impact of the event. A score greater than or equal to 0 is a second-class event, and the larger the score, the better the potential impact. An event value is adapted for any event. In some examples, a hot event can refer to a first-class event with an event value less than a specified threshold, or a second-class event with an event value greater than a given threshold.

[0049] GDELT gives each event a GoldsteinScale score (event value), which is used to measure the potential impact of the event on the target area or region, and can be used as an indicator to evaluate the intensity of the event. The NumMentions field attribute value is the number of times the event is mentioned in all articles in the database, which can be used as a standard for evaluating the importance of the event.

[0050] In step S103, the comprehensive intensity of any event is calculated based on the assigned event value and the numerical segment of the event mentioned in the article, and the event intensity curve within the specified time period of the target area is determined based on the calculated comprehensive intensity.

[0051] In step S104, a detection method based on a double sliding window is used for the determined event intensity curve to detect the jump points in the event intensity curve, so as to determine the hot events based on the jump points.

[0052] The embodiment of the present application comprehensively considers multiple factors such as the number of event reports, the impact of the event, the number of event mentions, and designs a method for quantitatively expressing event intensity.

[0053] In some embodiments, the comprehensive intensity of any event is calculated based on the assigned event value and the numerical segment of the event mentioned in the article to meet:

[0054] CEI=abs(GoldsteinScale)×NumMentions (1)

[0055] Among them, GoldsteinScale is the assigned event value, and NumMentions is the number of event and article mentions.

[0056] According to the above application example, the hot events are first screened out using the QuadClass field (whose value is equal to 3 or 4). Then the GoldsteinScale score is processed in absolute value, and its value is multiplied by the article mention number field NumMentions as a quantitative standard for the comprehensive intensity of the event.

[0057] In some embodiments, determining an event intensity curve within a specified time period of the target area according to the calculated comprehensive intensity includes:

[0058] In units of days, the event quantification standard LCEI for the target area is determined to meet:

[0059]

[0060] Among them, i represents the event ordinal number, M is the total number of events in the target area, and N is the total number of events involving the target area in the news records of the day. GoldsteinScale i is the GoldsteinScale score of an event in a single record, NumMentions i The ratio processing of the evaluation criteria of hot events in the target area and the total number of events in the embodiment of the present application is equivalent to normalization processing.

[0061] Based on the calculated LCEI of each event, the event intensity curve within the specified time period of the target area is determined in chronological order.

[0062] For the event intensity curve determined within the specified time period of the target area, the curve jump point can be considered as the occurrence of certain hot events. In some embodiments, for the determined event intensity curve, a detection method based on a double sliding window is used to detect the jump point in the event intensity curve, including:

[0063] Based on the event intensity curve, two relatively static continuous sliding windows are set to process the input data in segments;

[0064] The data in each segment is accumulated, and the energy ratio of the two windows is used as the decision statistic to perform jump point detection. The value of the decision variable of the double sliding window method is only related to the curve fluctuation envelope, and has nothing to do with the size of the curve value brought by different event intensities, thereby solving the problem of inconsistent decision variables in jump point detection. The energy ratio m(n) of the ratio of adjacent windows a(n) to b(n) is calculated as follows:

[0065]

[0066]

[0067]

[0068] Wherein, d is the event sequence number in the window, and L is the window length, which can be expressed in weeks, for example.

[0069] The setting of the threshold is crucial in signal detection, and it will directly affect the detection performance. The horizontal threshold is suitable for the case where the curve is relatively flat. Its setting is relatively simple and intuitive, and is usually set manually by the user. However, it is difficult to set a suitable detection threshold value in the face of a complex curve envelope. In some embodiments, for the determined event intensity curve, a detection method based on a double sliding window is used to detect the jump point in the event intensity curve, which also includes:

[0070] By tracking the curve fluctuations, the adaptive detection threshold is estimated to meet the following requirements:

[0071]

[0072] Where m(n) is the energy ratio of the two windows.

[0073] Therefore, the embodiment of the present application designs an adaptive threshold selection method, which is estimated by tracking the fluctuation of the curve, and the threshold can be automatically adjusted according to the envelope of the intensity curve.

[0074] In some embodiments, using the adaptive detection threshold, during the continuous sliding window sliding detection, if the energy ratios calculated for a specified number of points are all greater than the adaptive detection threshold, then it is determined to be a jump point, otherwise it is a noise point;

[0075] Remove noise points, update the adaptive detection threshold, and repeat the sliding window sliding detection jump point.

[0076] The present application also proposes an implementation case of a hot event detection method based on an open database, such as Figure 2 As shown, the following steps are included:

[0077] Step A1.1: Select one year of GDELT event database data, remove event data with duplicate attribute record values ​​such as ActionGeo_CountryCode, QuadClass, GoldsteinScale, NumMentions, and empty record values ​​of required fields, and complete data cleaning;

[0078] Step A1.2: Filter the data according to the target country or region through the ActionGeo_CountryCode and QuadClass fields (e.g. ActionGeo_CountryCode = RUS, QuadClass is 3 or 4);

[0079] Step A1.3: Calculate the comprehensive intensity of a single event according to formula (1).

[0080] Step A1.4: Take the day as the unit and obtain the intensity values ​​of all hot events on that day according to formula (2).

[0081] Step A1.5: Obtain the intensity curve of hot events in the target country or region in that year in units of years.

[0082] Step A2: Event detection based on event intensity curve jump point identification: This step includes initialization processing, double sliding window intensity ratio calculation and adaptive threshold selection. The specific processing flow is as follows: Figure 3 shown.

[0083] Step A2.1: Parameter initialization, starting from January 14th of each year;

[0084] Step A2.1.1: Take n = 14, L = 7, and use formulas (3), (4), and (5) to obtain a(14), b(14), and m(14);

[0085] Step A2.1.2: Take n = 15, L = 7, and use formulas (3), (4), and (5) to obtain a(15), b(15), and m(15);

[0086] Step A2.1.3: Get threshold initialization value

[0087] Step A2.2: The window slides forward one by one, and determines the relationship between m(n) and the threshold one by one:

[0088] If m(n)≥th, execute step A2.3;

[0089] If m(n)<th, execute step A2.4;

[0090] Step 2.3: Repeat step 2.2 to detect the next point;

[0091] If m(n+1)≥th;

[0092] …

[0093] If seven consecutive points, m(n+1), m(n+2)…m(n+6), are greater than the threshold, the nth point is considered to be a rising point;

[0094] If m(n+1)<th, the nth point is considered to be a burr (not involved in updating the threshold th), and step 2.2 is repeated to detect the next point.

[0095] Step A2.4: Update threshold th;

[0096] Step A2.4.1: Calculate the minimum value of all m(n) min[m(n)]

[0097] Step A2.4.2: Calculate the maximum value of all m(n) max[m(n)]

[0098] Step A2.4.2: Obtain a new threshold th according to formula (6)

[0099] Step A2.4.3: Repeat step A2.2.

[0100] The method of the present application comprehensively considers multiple factors such as the number of event reports, the influence of events, the number of event mentions, etc., and designs a method for quantitatively expressing the intensity of hot events; secondly, on the basis of forming a time series curve of the intensity value of hot events, the hot event discovery is transformed into a curve jump point detection problem, and a double sliding window energy detection method is adopted. Two continuous sliding windows are set and the received energy of the windows is calculated respectively, and the energy ratio is used as the decision variable; thirdly, the adaptive threshold selection can automatically adjust the threshold value according to the curve envelope to improve the probability of jump point detection.

[0101] An embodiment of the present application also proposes a hot event discovery device based on an open database, which is characterized in that it includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, the steps of the hot event detection method based on the open database are implemented as described above.

[0102] The embodiment of the present application further proposes a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned hot event detection method based on an open database are implemented.

[0103] It should be noted that in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0104] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0106] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A hot event detection method based on an open database, characterized in that: include: Perform data screening based on the target database to obtain events of interest; Less than zero is the first type of event. The smaller the score, the worse the potential negative impact of the event. Greater than or equal to 0 is the second type of event. The larger the score, the better the potential impact. An event value is adapted for any event. Calculate the comprehensive intensity of any event based on the assigned event value and the numerical segment of the event mentioned in the article, and determine the event intensity curve within the specified time period of the target area based on the calculated comprehensive intensity; For the determined event intensity curve, a detection method based on a double sliding window is adopted to detect the jump points in the event intensity curve, so as to determine the hot events based on the jump points.

2. The hot event detection method based on an open database as claimed in claim 1, characterized in that: Data screening based on the target database to obtain events of interest include: Based on the target database, select the data of the required time period and clean the data according to the required fields; After cleaning, filter by fields associated with the target region to obtain events of interest.

3. The hot event detection method based on an open database as claimed in claim 1, characterized in that: The comprehensive intensity of any event is calculated based on the assigned event value and the numerical segment of the event mentioned in the article to meet the following requirements: CEI=abs(GoldsteinScale)×NumMentions Among them, GoldsteinScale is the assigned event value, and NumMentions is the number of event and article mentions.

4. The hot event detection method based on an open database as claimed in claim 3, characterized in that: According to the calculated comprehensive intensity, the event intensity curve within the specified time period of the target area is determined to include: In units of days, the event quantification standard LCEI for the target area is determined to meet: Among them, i represents the event ordinal number, M is the total number of events in the target area, and N is the total number of all events involving the target area in the news records of the day; Based on the calculated LCEI of each event, the event intensity curve within the specified time period of the target area is determined in chronological order.

5. The hot event detection method based on an open database as claimed in claim 4, characterized in that: For the determined event intensity curve, a detection method based on a double sliding window is used to detect the jump points in the event intensity curve, including: Based on the event intensity curve, two relatively static continuous sliding windows are set to process the input data in segments; The data in each segment is accumulated, and the energy ratio of the two windows is used as the decision statistic to perform jump point detection, where the energy of window a(n) and window b(n) satisfies: Among them, d is the event number in the window, and L is the window length.

6. The hot event detection method based on an open database as claimed in claim 5, characterized in that: For the determined event intensity curve, a detection method based on a double sliding window is used to detect the jump point in the event intensity curve, which also includes: By tracking the curve fluctuations, the adaptive detection threshold is estimated to meet the following requirements: Where m(n) is the energy ratio of the two windows; Using the adaptive detection threshold value, in the process of continuous sliding window sliding detection, if the energy ratios calculated for a specified number of points are all greater than the adaptive detection threshold value, then it is determined to be a jump point, otherwise it is a noise point; Remove noise points, update the adaptive detection threshold, and repeat the sliding window sliding detection jump point.

7. A hot event discovery device based on an open database, characterized in that: It comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the hot event detection method based on an open database as described in any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the hot event detection method based on an open database as described in any one of claims 1 to 6 are implemented.

Images