A conflict risk level classification method based on distributed big data model
Through the distributed big data model, the gradients are calculated on multiple working machines and aggregated, the problem of time and difficulty of assessment of contradiction risk in the existing technology is solved, efficient classification and automatic assessment of contradiction risk are achieved, and manpower and material consumption is reduced.
Patent Information
- Application Number
- CN202211535702.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-01
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-12-01
AI Technical Summary
The existing technology is difficult to accurately assess the level of conflict risk, which increases the difficulty of social governance and the time-consuming analysis of big data, which increases the burden on staff.
Adopting the contradictory risk level classification method based on the distributed big data model, by dividing the data into sub-data sets and calculating the gradient on multiple workers, reducing the model training time, and iterating with the host aggregation gradient to achieve effective communication loss function optimization , reduce manpower and material consumption.
It greatly reduces the training time of big data model, improves the efficiency of conflict risk prevention and control, reduces the burden on staff, and realizes automatic classification of conflict risks and risk level assessment.
Smart Images

Figure CN116050818B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology and machine learning, and in particular to a conflict risk level classification method based on a distributed big data model. Background Art
[0002] With the development of society and science and technology, communication between people has become more convenient and information exchange has become more frequent. However, this has also made various social conflicts more prominent. If these conflicts are allowed to develop, they may eventually evolve into larger social conflicts and have a serious impact on social governance. It is difficult for social managers to obtain accurate risk levels of conflict events through manual perception of data, which greatly increases the difficulty of conflict risk governance. With the increase in data volume, ordinary big data analysis technology consumes a lot of time.
[0003] The present invention provides a conflict risk level classification method based on a distributed big data model, the purpose of which is to reduce the manpower and material resources consumed in prevention and control work, save time for big data model training, and reduce the burden on staff. Summary of the invention
[0004] In order to overcome the shortcomings of the prior art, the present invention provides a conflict risk level classification method based on a distributed big data model, which divides the huge amount of data into sub-data sets with appropriate data in each group, distributes each sub-data set to different sub-machines, calculates the gradient on each sub-machine and transmits it to the host, saves the time of big data model training, and reduces the burden on the staff. The technical solution of the present invention is as follows:
[0005] A method for classifying conflict risk levels based on a distributed big data model, the method comprising the following steps:
[0006] Step 1: Collect conflict event data and extract event feature labels for each conflict event’s appeal record;
[0007] Step 2: For the binary classification extraction problem of multiple feature labels of conflict appeal events, convert it into a single-label multi-classification problem;
[0008] Step 3: divide the conflict appeal event data obtained in step 2 and distribute it to m work machines. Select one of the work machines as the host, denoted as w1, and the other work machines are denoted as: w2, w3, ... w m ;
[0009] Step 4: Select the vector machine model as the classification model, and use the data assigned to the host to train the classification model to obtain the initial model parameters β 0 The host passes the initial value parameters to other working machines. j(j=2,3,…,m);
[0010] Step 5: For each worker machine w j , based on the locally stored data and the model parameters β obtained from the host 0 Computing Gradients j=2,3……m,the calculation formula is: Where n is the number of samples, (X i ,y i ) is the feature and label of the i-th sample, I() is the indicative function;
[0011] The local gradient vector Take the absolute value of each element of and retain the first K elements with the largest absolute value, and set the other elements to 0, so as to obtain the sparse TopK gradient vector Finally, we get the gradient error feedback vector required for the next iteration.
[0012] Step 6: The host receives the gradient vectors of each worker after the sparsification and error feedback operations. Aggregate it to get the aggregate gradient of the working machine The host performs the first iteration and obtains the effective communication loss function The communication-efficient loss function used in the first iteration The form is as follows:
[0013]
[0014] in Refers to the original loss function on the host, and the loss function that makes the communication effective is solved on the host The parameter β that reaches the minimum 1 ;
[0015] Step 7: 1 As the new initial estimator of the parameter, β 1 Transmitted to each sub-machine, the j-th sub-machine uses β 1 Compute local gradients Using the error feedback vector Get the new gradient vector The new gradient vector is obtained by TopK sparsification And calculate the error feedback vector Will Passed to the host, the host aggregates to get the aggregated gradient of the worker machine The communication effective loss function Replace the original loss function L, and use the communication-effective loss function for subsequent iterations The form is as follows:
[0016]
[0017] Solve the parameter β that minimizes the loss function on the host 2 ;
[0018] Step 8, loop step 7 to get β t , t=3,4,5,…, until β t No more updates, β t are the final model parameters;
[0019] Step 9: For new conflict appeal event records, the model parameter β obtained in step 8 is t , classify the events and obtain the event features of the appeal record;
[0020] Step 10: for conflict appeal events classified by event characteristics, count the number of occurrences of each characteristic, calculate the risk value of each characteristic, and obtain the total risk value by weighted average of each characteristic risk. A total risk value of 85 points or more indicates high risk, 60 to 85 points indicates moderate risk, 50 to 60 points or less indicates low risk, and 50 points or less indicates no risk;
[0021] Step 11, for the divided conflict event data and risk level labels, set a level adjustment button, and let experts decide whether to adjust to the current risk level based on industry experience to obtain the final event risk level.
[0022] Preferably, the following feature labels are extracted in step 1: whether it is an extreme event, whether it is a negative public opinion event, and whether it is an extreme behavior event.
[0023] Preferably, the labels in step 2 are classified as follows: {extreme events}, {radical behavior events}, {negative public opinion events}, {extreme events, radical behavior events}, {extreme events, negative public opinion events}, {radical behavior events, negative public opinion events}, {extreme events, radical behavior events, negative public opinion events}.
[0024] Preferably, the amount of data obtained by each working machine in step 3 is the same and there is no duplicate data.
[0025] Preferably, the classification model may also select models such as logistic regression and neural network.
[0026] Preferably, the specific calculation formula for calculating the risk value of the feature in step 10 is as follows:
[0027] score 各特征风险 =min(95,2x 3 -20x2 +70x)*e -0.01t
[0028] Where x represents the number of events for each feature, and t represents the number of days since the event.
[0029] Compared with the prior art, the method of the present invention has the following beneficial effects: by dividing the data set, it is only necessary to calculate the gradient on each sub-machine and pass it to the host, and the host performs model training based on the locally stored data. This is a communication-effective big data analysis method. According to the trained model, subsequent conflict risk events can be automatically classified, which greatly reduces the expenditure of manpower and material resources and improves the efficiency of subsequent conflict risk prevention and control. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 This is a model diagram of the present invention. DETAILED DESCRIPTION
[0031] In order to illustrate the technical solution and working principle of the present invention, the embodiments of the present invention will be described in further detail below. All the above optional technical solutions can be combined in any way to form optional embodiments of the present invention, which will not be described one by one here.
[0032] The terms "step 1", "step 2", "step 3", and the like in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those described herein.
[0033] The present disclosure provides a method for classifying conflict risk levels based on a distributed big data model, the method comprising the following steps:
[0034] Step 1: Collect conflict event data and extract event feature labels for the appeal records of each conflict event. Preferably, we extract the following feature labels of the conflict event: whether it is an extreme event, whether it is a negative public opinion event, and whether it is an extreme behavior event.
[0035] Step 2: For the binary classification problem of multi-feature labels of conflict appeal events, convert it into a single-label multi-classification problem. The label classification in step 1 is as follows: {extreme events}, {excessive behavior events}, {negative public opinion events}, {extreme events, excessive behavior events}, {extreme events, negative public opinion events}, {excessive behavior events, negative public opinion events}, {extreme events, excessive behavior events, negative public opinion events}. Next, natural language processing technology will be used and the following distributed communication effective settings will be adopted to train the model.
[0036] Step 3: divide the conflict appeal event data obtained in step 2 and distribute it to m work machines. Preferably, the amount of data obtained by each work machine is the same and there is no duplicate data. Select one of the work machines as the host, denoted as w1, and the other work machines are denoted as: w2, w3, ... w m ,Subsequent parameter transfer and model training will be completed on the host.
[0037] Step 4: Select a classification model and use the data assigned to the host to train the classification model to obtain the initial model parameters β 0 The host passes the initial value parameters to other working machines. j (j=2,3,…,m).
[0038] Preferably, the classification model can be selected from support vector machines, logistic regression, neural networks, etc.
[0039] Step 5: For each worker machine w j , based on the locally stored data and the model parameters β obtained from the host 0 Computing Gradients j=2,3……m,transfer it to the host and obtain the gradient error feedback vector required for the next iteration
[0040] First, according to the classification model selected in step 4, the gradient calculation formula of the corresponding loss function L is obtained:
[0041] For example, when the support vector machine model is selected, the gradient calculation formula is:
[0042]
[0043] Where n is the number of samples, (X i ,y i ) is the feature and label of the i-th sample, and I() is the indicative function.
[0044] Secondly, for each worker machine w j , based on the locally stored data, the local gradient vector is calculated Then the local gradient vector Take the absolute value of each element of and retain the first K elements with the largest absolute value, and set the other elements to 0, so as to obtain the sparse TopK gradient vector And get the gradient error feedback vector of the working machine
[0045] Step 6: The host receives the gradient vectors of each worker after the sparsification and error feedback operations. Aggregate it to get the aggregate gradient of the working machine The host performs the first iteration and obtains the effective communication loss function The communication-efficient loss function used in the first iteration The form is as follows:
[0046]
[0047] in Refers to the original loss function on the host, and the loss function that makes the communication effective is solved on the host The parameter β that reaches the minimum 1 .
[0048] Step 7: 1 As the new initial estimator of the parameter, β 1 Transmitted to each sub-machine, the j-th sub-machine uses β 1 Compute local gradients Using the error feedback vector Get the new gradient vector The new gradient vector is obtained by TopK sparsification And calculate the error feedback vector Will Passed to the host, the host aggregates to get the aggregated gradient of the worker machine Finally, we will communicate the effective loss function Replace the original loss function L, specifically, the communication-effective loss function used in subsequent iterations The form is as follows:
[0049]
[0050] Solve the parameter β that minimizes the loss function on the host 2 .
[0051] Step 8, loop step 7 to get β t , t=3,4,5,…, until β t No more updates. t are the final model parameters.
[0052] Step 9: For the new conflict appeal event record, classify it according to the classification model parameters obtained in step 8 to obtain the event features of the appeal record.
[0053] Step 10: After obtaining the event classification label, the new conflict appeal event record is classified into risk level according to the rules.
[0054] First, according to the new conflict appeal event record and risk level classification rules, each event is divided into four levels: high-risk event, medium-risk event, low-risk event, and no-risk event. Specifically, the risk level classification rules are as follows:
[0055] We obtain the characteristics of each conflict event: extreme events, negative public opinion events, and radical behavior events. For conflict events, we count the number of occurrences of each feature and calculate the risk value of each feature. By fitting the existing data, we obtain a specific calculation formula as follows:
[0056] score 各特征风险 =min(95,2x 3 -20x 2 +70x)*e -0.01t
[0057] Where x represents the number of events for each feature (for example, if the event is {extreme event, negative public opinion event}, then x is 2), and t represents the number of days since the event;
[0058] The total risk value is obtained by weighted averaging each characteristic risk. A total risk value above 85 points indicates high risk, 60 to 85 points indicates moderate risk, 50 to 60 points or less indicates low risk, and below 50 points indicates no risk.
[0059] Step 11, for the divided conflict event data and risk level labels, set the level adjustment button (confirm adjustment, do not adjust for now), and let the experts decide whether to adjust to the current risk level based on industry experience to obtain the final event risk level.
[0060] The present invention is described above by way of example. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. Any non-substantial improvements made by adopting the method concept and technical solution of the present invention; or directly applying the above-mentioned concept and technical solution of the present invention to other occasions without improvement or equivalent replacement, are all within the protection scope of the present invention.
Claims
1. A method for classifying contradiction risk levels based on a distributed big data model, characterized in that: The method comprises the following steps: Step 1: Collect conflict event data and extract event feature labels for each conflict event’s appeal record; Step 2: For the binary classification extraction problem of multiple feature labels of conflict appeal events, convert it into a single-label multi-classification problem; Step 3: divide the conflict appeal event data obtained in step 2 and distribute it to m work machines. Select one of the work machines as the host, denoted as w1, and the other work machines are denoted as: w2, w3, .....w m ; Step 4: Select the vector machine model as the classification model, and use the data assigned to the host to train the classification model to obtain the initial model parameters β 0 The host passes the initial value parameters to other working machines w j (j=2, 3, ..., m); Step 5: For each worker machine w j , based on the locally stored data and the model parameters β obtained from the host 0 Computing Gradients The calculation formula of the vector machine model is: Where n is the number of samples, (X i ,y i ) is the feature and label of the i-th sample, I() is the indicative function; The local gradient vector Take the absolute value of each element of and retain the first K elements with the largest absolute value, and set the other elements to 0, so as to obtain the sparse TopK gradient vector Finally, we get the gradient error feedback vector required for the next iteration. Step 6: The host receives the gradient vectors of each worker after the sparsification and error feedback operations. Aggregate it to get the aggregate gradient of the working machine The host performs the first iteration and obtains the effective communication loss function The communication-efficient loss function used in the first iteration The form is as follows: in Refers to the original loss function on the host, and the loss function that makes the communication effective is solved on the host The parameter β that reaches the minimum 1 ; Step 7: 1 As the new initial estimator of the parameter, β 1 Transmitted to each sub-machine, the j-th sub-machine uses β 1 Compute local gradients Using the error feedback vector Get the new gradient vector The new gradient vector is obtained by TopK sparsification And calculate the error feedback vector Will Passed to the host, the host aggregates to get the aggregated gradient of the worker machine The communication effective loss function Replace the original loss function L, and use the communication-effective loss function for subsequent iterations The form is as follows: Solve the parameter β that minimizes the loss function on the host 2 ; Step 8: loop step 7 to get β t , t=3,4,5,..., until β t No more updates, β t are the final model parameters; Step 9: For new conflict appeal event records, the model parameter β obtained in step 8 is t , classify the events and obtain the event features of the appeal record; Step 10: for conflict appeal events classified by event characteristics, count the number of occurrences of each characteristic, calculate the risk value of each characteristic, and obtain the total risk value by weighted average of each characteristic risk. A total risk value of 85 points or more indicates high risk, 60 to 85 points indicates moderate risk, 50 to 60 points or less indicates low risk, and 50 points or less indicates no risk; Step 11, for the divided conflict event data and risk level labels, set a level adjustment button, and let experts decide whether to adjust to the current risk level based on industry experience to obtain the final event risk level.
2. According to the method for classifying contradiction risk levels based on a distributed big data model according to claim 1, it is characterized in that: In step 1, the following feature labels are extracted: whether it is an extreme event, whether it is a negative public opinion event, and whether it is an extreme behavior event.
3. According to claim 2, a method for classifying contradiction risk levels based on a distributed big data model is characterized in that: The labels in step 2 are classified as follows: {extreme events}, {radical behavior events}, {negative public opinion events}, {extreme events, radical behavior events}, {extreme events, negative public opinion events}, {radical behavior events, negative public opinion events}, {extreme events, radical behavior events, negative public opinion events}.
4. The method for classifying contradiction risk levels based on a distributed big data model according to claim 3 is characterized in that: In step 3, each worker machine obtains the same amount of data and there is no duplicate data.
5. The conflict risk level classification method based on a distributed big data model according to any one of claims 1 to 4, characterized in that: The specific calculation formula for calculating the risk value of the feature in step 10 is as follows: score 各特征风险 =min(95,2x 3 -20x 2 +70x)*e -0.01t Where x represents the number of events for each feature, and t represents the number of days since the event.
Citation Information
Patent Citations
Road traffic accident occurrence risk determination method and system
CN111126868A
Deep learning method for recognizing cause-effect relationship of contradictory dispute event
CN111709244A
SA-ISAR imaging and self-focusing method based on CV-ADMMN
CN112099008A
Social contradiction index prediction method based on big data
CN113822498A
Image classification-oriented multi-parameter adaptive heterogeneous parallel computing method
CN114140641A
