A Multidimensional Event Development Analysis Method Based on Big Data

Abstract

This invention discloses a multidimensional event development analysis method based on big data, comprising the following steps: collecting conflict event data; cleaning the data and automatically classifying the conflict events; dividing the dataset according to the conflicting events; filtering features through multidimensional correlation analysis to obtain a feature dataset; using the FP-growth algorithm to obtain frequent itemsets of the feature dataset; based on the obtained frequent itemsets, obtaining association rules with a lift greater than a certain threshold; and based on the association rules, obtaining multidimensional conflict development patterns, and controlling the development of conflicts in their early stages to prevent escalation. This invention, through the mining of frequent itemsets and association rules of conflict features, helps staff understand the patterns of conflict development in various fields, predict potential conflicts in a given event, greatly improves work efficiency, and saves human and material resources.

Description

Technical Field

[0001] This invention relates to the field of natural language processing research, specifically to a multi-dimensional event development analysis method based on big data. Background Technology

[0002] With the development of society and science and technology, communication between people has become more convenient, and information exchange has become more frequent. However, this has also made various social conflicts more prominent. If these conflicts are left unchecked, they may eventually evolve into larger social conflicts, seriously impacting social governance. Social managers, relying on manually perceived data, can only obtain some subjective patterns of conflict development, and this heavily depends on the professional knowledge of personnel. It requires summarizing by people with different backgrounds, leading to a significant waste of human and material resources, and ultimately failing to accurately understand the development patterns of conflicts in various fields. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention collects data on social conflict events and records their conflict levels; it utilizes the BERT model to classify conflict event texts and identify emotions, achieving accurate classification and reducing manual costs; it then constructs a new feature dataset by combining features in multiple dimensions and uses the FP-growth algorithm to efficiently select frequent itemsets; based on the frequent itemsets, it calculates the association rules between each feature or feature combination to obtain the conflict development law and can prevent the escalation of new conflict events based on the conflict development law. The specific technical solution of this invention is as follows:

[0004] A multi-dimensional event development analysis method based on big data includes the following steps:

[0005] Step 1: Collect the text content, event level, and event subject of each event, and assign a number to each event.

[0006] Preferably, the collected event levels are divided into four categories: A, B, C, and D, with A to D indicating progressively lower levels.

[0007] Step 2: Clean the event data; merge all event texts into an event corpus, set up training and test sets, continue training the model on the training set of this corpus using the pre-trained BERT-base-Chinese model, adjust the parameters using the test set, select the model with the highest score on the test set, save the best model parameters, and obtain the best language model in the training phase; based on the trained language model, classify each event into content category, sentiment level, and event nature.

[0008] Preferably, the training set and test set of the event corpus are set in a 7:3 ratio.

[0009] Step 3: Based on the data obtained in Steps 1 and 2, extract the event subject information, including name, age, gender, region, and event handling satisfaction. Age is binned. If the event subject is not a single person, record the event for each person and add event characteristics.

[0010] Preferably, the age-based binning operation rules are as follows: those under 25 years old are recorded as 1; those between 25 and 35 years old are recorded as 2; those between 35 and 45 years old are recorded as 3; those between 45 and 55 years old are recorded as 4; those between 55 and 65 years old are recorded as 5; and those over 65 years old are recorded as 6.

[0011] Step 4: For a single event involving a single event subject, there will be multiple records. Sort the records by time. For each record at a time point, construct the following features: event category, emotional level of the event subject in the record, satisfaction level of handling the event record, whether all previous events of this record have progressed across levels, and whether the event subject of this record is an individual or a group. After combining these features in multiple dimensions and performing correlation analysis with the event level, the final features are selected to obtain the feature dataset.

[0012] The specific method is as follows: construct a test statistic using the chi-square test, and calculate the value of the chi-square statistic for the characteristic data. By setting a significance level p, the chi-square statistic value that satisfies the significance level p is obtained. if This indicates that the feature or combination of features is related to the level of contradiction; features and combinations of features that are highly correlated with the level of contradiction are selected.

[0013] Preferably, the number of selected features and feature combinations is 10.

[0014] Step 5: Count the frequency of each feature in the feature dataset, set a minimum support threshold, and use the FP-growth algorithm to select frequent itemsets whose support is greater than the minimum support threshold. The support of feature {A, B} is defined as follows:

[0015] S(A,B)=N(A,B) / n, where N(A,B) is the number of times feature A and feature B appear in combination, and n is the total number of occurrences.

[0016] The specific process is as follows:

[0017] Features with a support greater than the minimum support threshold are filtered out and sorted according to frequency to obtain a new frequent feature dataset.

[0018] Construct an FP-tree based on the new frequent feature dataset. Read the dataset and insert it into the FP-tree. When inserting, insert the data into the FP-tree in the sorted order. The nodes that are first in the sorted order are root nodes, and the nodes that are last in the sorted order are leaf nodes. If there is a shared root node, increment the count of the corresponding shared root node by 1.

[0019] After insertion, if a new node appears, the node corresponding to the head pointer table will be linked to the new node through the node linked list until all data is inserted into the FP-tree, at which point the FP-tree is completed.

[0020] Based on the FP-tree, the conditional pattern base corresponding to the head pointer table item is found sequentially from the bottom item of the head pointer table upwards. Frequent itemsets of the head pointer table items are obtained by recursively mining from the conditional pattern base. Finally, frequent itemsets that satisfy the conditional pattern base with a number of items greater than the minimum support threshold are returned.

[0021] Step 6: Combine the obtained frequent feature terms into different sets, calculate the lift between each set, set a minimum lift threshold, and return the set with a lift greater than the minimum lift threshold to obtain the association rule;

[0022] The lift is defined as: lift from feature A to feature B = (confidence from feature A to feature B) / support of feature B, i.e., T(A,B) = C(A,B) / S(B).

[0023] The confidence level is defined as: confidence level from feature A to feature B = (support of features A and B) / support of feature A, i.e., C(A,B) = S(A,B) / S(A).

[0024] Step 7: Based on the association rules in Step 6, obtain the multidimensional event development pattern, and control the event in its early stages to prevent it from deteriorating.

[0025] Compared with existing technologies, the beneficial effects of this invention are as follows: by utilizing natural language processing technology, it enables automatic classification and emotion recognition of conflicting event texts; by mining frequent itemsets and association rules of conflicting features, it combines features in multiple dimensions, making it more efficient to find association rules, helping staff understand the development patterns of social conflicts in various fields, predicting potential conflicts in a certain event in advance, greatly improving work efficiency, and saving social resources such as manpower and material resources. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the FP-tree used to construct frequent itemsets in this invention. Detailed Implementation

[0027] To clarify the technical solution and working principle of the present invention, the embodiments of this disclosure will be described in further detail below. All the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be elaborated upon here.

[0028] The terms "step 1," "step 2," "step 3," and similar descriptions used in the specification and claims of this application are for distinguishing similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those described herein.

[0029] This disclosure provides a method for multi-dimensional event development analysis based on big data, specifically including the following steps:

[0030] Step 1: Collect data on conflicting events.

[0031] For contradictory events, the text content, event level, and event subject are collected for each event. The event level is divided into four categories: A, B, C, and D, with A to D indicating progressively lower levels. Each event is then numbered.

[0032] Step 2: Clean the data and automatically classify contradictory events.

[0033] Based on pre-set cleaning rules, conflicting event data is cleaned.

[0034] All event texts are merged into an event corpus, with the training and test sets set in a 7:3 ratio. The pre-trained BERT-base-Chinese model is then trained on the training set of this corpus. Parameters are adjusted using the test set, and the model with the highest score on the test set is selected and its optimal parameters are saved, resulting in the best language model for the training phase.

[0035] Based on the trained language model, each event is classified into content category, emotion level, and event nature (whether it is an extreme event). The content category includes 35 secondary tags, and the emotion level includes three types: neutral, positive, and negative.

[0036] Step 3: Divide the dataset according to the events reacted by each person.

[0037] First, based on the data obtained in steps 1 and 2, extract the event subject information, including name, age, gender, region, and event handling satisfaction. Age is binned as follows: under 25 years old is 1; 25-35 years old is 2; 35-45 years old is 3; 45-55 years old is 4; 55-65 years old is 5; and over 65 years old is 6. Regions are uniformly divided, for example, by street / town / district. If the event subject is not a single person, record the event for each person and add event characteristics.

[0038] Step 4: After filtering features through multidimensional correlation analysis, the feature dataset is obtained.

[0039] For a single event involving a single event subject, there will be multiple records. These records are sorted chronologically. For each record at a given time point, the following features are constructed: event category, the emotional level of the event subject in that record, the satisfaction level with the handling of that event record, whether all previous events in that record have progressed across different levels, and whether the event subject in this record is an individual or a group. These features are then combined in multiple dimensions and subjected to correlation analysis with the event level to select the final features. The specific operation is as follows:

[0040] For each feature or combination of features (multiple feature combinations form a new feature), perform a correlation analysis with the event level:

[0041] First, construct the test statistic; here, we choose the chi-square test. Then, calculate the value of the chi-square statistic based on the actual data. The chi-square statistic value that satisfies the pre-set significance level p is obtained by looking up the table. if The null hypothesis is rejected, and an alternative hypothesis is selected, indicating that the feature or combination of features is related to the level of contradiction. Features and combinations of features with the highest correlation to the level of contradiction are selected. Preferably, this patent selects 10 features and combinations of features, as detailed below:

[0042] Feature 1: The category of contradictory events;

[0043] Feature 2: Abnormal emotions;

[0044] Feature 3: Dissatisfaction with the handling of conflicting events;

[0045] Feature 4: Extreme events;

[0046] Feature 5: Dissatisfaction with the handling of extreme and contradictory events;

[0047] Feature 6: Extreme events + abnormal emotions;

[0048] Special criterion 7: Development beyond one's level + abnormal emotions;

[0049] Special Principle 8: Cross-level development + unsatisfactory handling of conflict incidents;

[0050] Feature 9: Extreme events + abnormal emotions + unsatisfactory handling of conflicting events;

[0051] Feature 10: Abnormal emotions + unsatisfactory handling of conflicting events + development across levels.

[0052] Step 5: Obtain the frequent itemsets of the feature dataset.

[0053] Based on the feature dataset constructed above, the FP-growth algorithm is used to select frequent itemsets with support greater than a certain threshold. The support of feature {A, B} is defined as follows:

[0054] S(A,B)=N(A,B) / n, where N(A,B) is the number of times feature A and feature B appear in combination, and n is the total number of occurrences.

[0055] A frequent itemset is defined as a set whose support is greater than the minimum support threshold.

[0056] The specific process of the FP-growth algorithm is as follows: First, count the frequency of each feature in the feature dataset, set a minimum support threshold, and filter out features that exceed the minimum support threshold. Sort the features by frequency to obtain a new frequent feature dataset. Second, construct an FP-tree based on the new frequent feature dataset. Read the dataset and insert it into the FP-tree in the sorted order. The node at the beginning of the sorted list becomes the root node, and the nodes at the end become leaf nodes. If there is a shared root node, its count is incremented by 1. After insertion, if a new node appears, the node corresponding to the head pointer table will be linked to the new node through the node linked list. The FP-tree is completed after all data has been inserted. A partial FP-tree diagram is shown below. Figure 1 As shown.

[0057] Based on this FP-tree, we can quickly find frequent itemsets and their support. The specific steps are as follows: Starting from the bottom of the head pointer table, find the conditional pattern base corresponding to each item in the head pointer table. Recursively mine the frequent itemsets of the head pointer table items from the conditional pattern base. Finally, return the frequent itemsets that satisfy the condition that have more than a certain number of items.

[0058] Step 6: Based on the frequent itemsets obtained above, obtain association rules with lift greater than a certain threshold.

[0059] The lift is defined as: lift from feature A to feature B = (confidence from feature A to feature B) / support of feature B, i.e.

[0060] T(A,B)=C(A,B) / S(B)

[0061] The confidence level is defined as: Confidence from feature A to feature B = (Support of features A and B) / Support of feature A, i.e.

[0062] C(A,B)=S(A,B) / S(A)

[0063] Confidence can be understood as the probability that feature B will appear when feature A appears, while lift can be understood as the increase in the probability of feature B appearing when feature A appears.

[0064] Specifically, the obtained frequent features are combined into different sets, the lift between each set is calculated, and the set with a lift greater than a certain threshold is returned.

[0065] Step 7: Based on the association rules in Step 6, obtain the multidimensional event development pattern, and control the event in its early stages to prevent it from deteriorating.

[0066] Specifically, when a pattern similar to the following is obtained: {Urban and rural construction event, abnormal emotions, and escalation of conflict across levels}----->{Conflict level B}, the confidence level is 0.7; this indicates that when abnormal emotions and escalation of conflict across levels occur in an urban and rural construction event, there is a 70% probability that the conflict level will be B; when a subsequent urban and rural construction event occurs, accompanied by abnormal emotions of the conflicting parties and escalation of conflict across levels, it is necessary to pay closer attention to it and take corresponding measures to prevent the conflict level from gradually deteriorating.

[0067] The present invention has been described above by way of example. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other situations without improvement or equivalent substitution, are all within the protection scope of the present invention.

Claims

1. A big data based multi-dimensional event development analysis method, characterized in that, Comprise the following steps: Step 1, collect the text content of each event, event level, event subject, number each event; Step 2, clean the event data; all event texts are combined into an event corpus, a training set and a test set are set, the pre-trained BERT-base-Chinese model is used to continue training the model on the training set of the corpus, the parameters are adjusted using the test set, the model with the highest score on the test set is selected, the best model parameters are saved, and the best language model in the training stage is obtained; According to the trained language model, each event is divided into content category, emotion level and event nature; Step 3, according to the data obtained in steps 1 and 2, the event subject information is extracted: including name, age, gender, region, event handling satisfaction; Wherein the age is binned; If the event subject is not a single person, record the event for each person and add the characteristics of the event; Step 4, for a single event of a single event subject, there will be multiple records, the records are sorted by time, and for each time point record, the following features are constructed: event category, emotion level of event subject in the record, handling satisfaction of the event record, whether all previous event records of the record develop across levels, whether the event subject of the record is single or group; After multi-dimensional combination of these features and correlation analysis of event level, the final features are screened to obtain a feature data set; Step 5, count the number of times each feature appears in the feature data set, set a minimum support threshold, and use the FP-growth algorithm to select frequent item sets with support greater than the minimum support threshold; Wherein, the support of feature {A, B} is defined as: S(A, B) = N(A, B) / n, wherein N(A, B) is the number of times the combination of features A and B appears, and n is the total number of times; Step 6, combine the obtained frequent feature items into different sets, calculate the lift degree between each set respectively, set a minimum lift threshold, return the sets with lift degree greater than the minimum lift threshold, and obtain the association rules; Wherein, the lift degree is defined as: the lift degree of feature A to feature B = (the confidence of feature A to feature B) / the support of feature B, that is, T(A, B) = C(A, B) / S(B); Wherein, the confidence is defined as: the confidence of feature A to feature B = (the support of feature A, B) / the support of feature A, that is, C(A, B) = S(A, B) / S(A); Step 7, according to the association rules of step 6, the multi-dimensional event development rule is obtained, and the event development rule is used to control the early stage of event development to prevent deterioration.

2. The method of claim 1, wherein, The event level collected in step 1 is divided into four categories: A, B, C and D, which represent lower and lower levels from A to D.

3. The method of claim 2, wherein, In step 2, the training set and test set of the event corpus are set in a ratio of 7:

3. 4.The method of claim 2, wherein, In step 3, the age binning operation rule is: age below 25 years old, recorded as 1; Age between 25 and 35 years old, recorded as 2; Age between 35 and 45 years old, recorded as 3; Age between 45 and 55 years old, recorded as 4; Age between 55 and 65 years old, recorded as 5; Age is greater than 65 years old, recorded as 6.

5. The method of claim 2, wherein, The method for screening the final features in step 4 is: Using chi-square test to construct test statistics, calculate the value of chi-square statistics of feature data , set the significance level p, get the chi-square statistics value that satisfies the significance level p , if > , it means that the feature or feature combination is related to the contradiction level; screen out the top features and feature combinations related to the contradiction level.

6. The method of claim 5, wherein, The number of selected features and feature combinations is 10.

7. The method of claim 6, wherein, The specific process for selecting frequent item sets in step 5 using the FP-growth algorithm is: Screen out features greater than the minimum support threshold, and get a new frequent feature dataset according to the number of sorting; According to the new frequent feature dataset, build an FP tree, read the dataset, and insert the FP tree. When inserting, follow the sorted order, insert into the FP tree, and the nodes at the front are the root nodes, while the nodes at the back are the leaf nodes. If there are common root nodes, the corresponding common root nodes count is incremented by 1; After insertion, if new nodes appear, the corresponding nodes of the head pointer table will be linked to the new nodes through the node chain table until all data is inserted into the FP tree, and the FP tree is built. According to the FP tree, find the conditional pattern base corresponding to the head pointer table item from the bottom item of the head pointer table to the top, recursively mine the frequent item set of the head pointer table item from the conditional pattern base, and finally return the frequent item set that satisfies the item number greater than the minimum support threshold.

Images