Event prediction method and device based on big data, equipment and storage medium

Abstract

The invention discloses a big data-based event prediction method, device and equipment and a storage medium, and the method comprises the steps: collecting an initial data set, carrying out the preprocessing, predicting a situation value set and a situation variable quantity through a situation evaluation model and a situation prediction model, further calculating an event state transition probability, and determining an event state. And finally, performing safety early warning according to the event state and the event importance. The method solves the problems that an existing event prediction method only pays attention to the event situation and ignores the situation change trend, and the prediction precision is low and the feature selection is unreasonable when large-scale and multi-dimensional data are processed.

Description

Technical Field

[0001] The present invention relates to the field of event prediction technology, and in particular to an event prediction method, device, equipment and storage medium based on big data. Background Art

[0002] In the context of big data, event prediction has become an important tool for decision support in many fields (such as finance, transportation, medical care, etc.). The accuracy of event prediction not only depends on advanced algorithm models, but also highly depends on the quality of input data and the rationality of feature selection.

[0003] However, existing event prediction methods often face the following challenges when processing large-scale, multi-dimensional data. On the one hand, traditional event prediction mainly focuses on event status, but not on the trend of situation changes, resulting in low prediction accuracy. On the other hand, the large data sets used for prediction contain a large number of features, many of which do not contribute substantially to the prediction target and may even introduce noise, affecting the prediction accuracy. At the same time, traditional feature selection methods mostly rely on expert experience or simple statistical methods, making it difficult to comprehensively and objectively evaluate the importance of each feature to the prediction target. As the environment and time change, the importance of certain features will also change, and existing methods often find it difficult to capture such dynamic changes. Summary of the invention

[0004] In view of the defects in the prior art, the present invention provides an event prediction method, device, equipment and storage medium based on big data.

[0005] In a first aspect, the present invention provides an event prediction method based on big data, comprising:

[0006] Collect the initial data set D initial , the initial data set includes at least one historical data related to the event to be predicted;

[0007] For the initial data set D initial Preprocessing is performed and the situation assessment model f(D preprocessed ) predicts the set of situation values ​​at the past k time points {U history (t-1),...,U history (tk)}, using the situation prediction model g(θ,{U history (t-1),...,U history (tk)}) predict the situation change ΔU at time t;

[0008] Calculate the probability P of the event to be predicted transferring from the current state to other states under the state change amount trans (E i →E j |ΔU); according to the probability Ptrans (E i →E j |ΔU), situation change ΔU pred (t) determine the status of the incident;

[0009] A security warning is issued based on the event status and event importance.

[0010] Preferably, the preprocessing includes data clarity and formatting uniformity.

[0011] 4. Preferably, the training data acquisition of the situation assessment model includes:

[0012] Extract the initial feature set F Ei , for each initial feature set F Ei The feature f in i ∈F Ei , using the high confidence feature set R Ci The probability distribution P of similar features to history hist (F Ei ∣R Ci ), calculate the feature credibility according to the credibility evaluation formula The credibility evaluation formula is:

[0013]

[0014] In the formula, n is the total number of features, sim(g) is the similarity function;

[0015] According to the calculated feature credibility, a threshold ξ is set threshold , select features with a credibility higher than the threshold to construct the final feature set F for training final .

[0016] Preferably, the high-confidence feature set R high The construction method is:

[0017] Determine the situation assessment target and the feature set to be screened, invite multiple field experts to score the relative importance of each feature in the feature set to be screened for the situation assessment target, and construct an original judgment matrix;

[0018] Extract the score from the original judgment matrix, construct a feature weight data point set, set multiple possible classification data K, use a clustering algorithm to perform negative energy on the feature weight data point set, and obtain classification clusters under different preset classification data;

[0019] Calculate the distance between the cluster center points of two classification clusters to obtain a distance value, and determine a reasonable number of classifications and a corresponding set of classification data points based on a comparison result between the distance value and a preset distance;

[0020] Calculate the cluster center points of the classified data point set, and perform elimination and update processing based on the distance data set to eliminate outliers or abnormal data points to obtain the target cluster set;

[0021] The features occupying multiple target cluster sets are selected as high-confidence features, thereby obtaining a high-confidence feature set.

[0022] Preferably, the similarity function is calculated using cosine similarity or Pearson correlation coefficient.

[0023] Preferably, the security warning is performed according to the event status and event importance, including:

[0024] Determine the warning level according to the event status and event importance,

[0025] An early warning is issued according to the early warning level.

[0026] In a second aspect, the present invention provides an event prediction system based on big data, comprising:

[0027] Data acquisition module, used to collect the initial data set D initial , the initial data set includes at least one historical data related to the event to be predicted;

[0028] Situation awareness module, used for the initial data set D initial Preprocessing is performed and the situation assessment model f(D preprocessed ) predicts the set of situation values ​​at the past k time points {U history (t-1),...,U history (tk)}, using the situation prediction model g(θ,{U history (t-1),...,U history (tk)}) predict the situation change ΔU at time t;

[0029] The state determination module is used to calculate the probability P of the event to be predicted to transfer from the current state to other states under the state change amount. trans (E i →E j |ΔU); according to the probability P trans (E i →E j |ΔU), situation change ΔU pred (t) determine the status of the incident;

[0030] The security warning module is used to issue a security warning based on the event status and event importance.

[0031] In a third aspect, the present invention provides a computer device, comprising a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute any one of the above methods.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, it can implement any of the methods described above.

[0033] The beneficial effects of the present invention are: improving the accuracy and precision of event prediction, optimizing the feature selection process, enhancing the robustness and adaptability of the model, and providing a comprehensive safety warning mechanism. These advantages make the present invention have significant application value and market prospects in the field of event prediction technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0035] Figure 1 A flowchart of an event prediction method based on big data is provided for the present invention;

[0036] Figure 2 A system block diagram of an event prediction system based on big data is provided for the present invention;

[0037] Figure 3 A structural schematic diagram of a computer device is provided for the present invention. DETAILED DESCRIPTION

[0038] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0039] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0040] Embodiment 1

[0041] like Figure 1 As shown, the present invention provides an event prediction method based on big data, comprising:

[0042] Step 1: Collect the initial data set D initial, the initial data set includes at least one historical data related to the event to be predicted.

[0043] Specifically, historical data related to the event to be predicted is collected through data warehouses, data lakes or real-time data streams. These data may include various logs, transaction records, sensor data, etc.

[0044] Step 2: For the initial data set D initial Preprocessing is performed and the situation assessment model f(D preprocessed ) predicts the set of situation values ​​at the past k time points {U history (t-1),...,U history (tk)}, using the situation prediction model g(θ,{U history (t-1),...,U history (tk)}) predicts the situation change ΔU at time t.

[0045] In this embodiment, the preprocessing includes data clarification and formatting unification.

[0046] Data clarity includes: Duplicate data processing: Identify and delete duplicate data through uniqueness verification, hash algorithm and other technical means. Missing value processing: Fill missing values ​​(such as using mean, median, mode, or using prediction model) or delete them. Outlier detection and processing: Detect and process outliers using statistical methods (such as Z-score, IQR range) or machine learning algorithms (such as isolation forest).

[0047] Formatting unification includes: Data standardization: converting different types of data into a unified format, such as unifying the date and time format, converting text data into numerical representation (such as one-hot encoding, label encoding). Data normalization: scaling the data, such as minimum-maximum normalization and Z-score normalization, to reduce the impact of different dimensions and ranges on subsequent analysis.

[0048] In this embodiment, the training data acquisition of the situation assessment model includes:

[0049] Extract the initial feature set F Ei , for each initial feature set F Ei The feature f in i ∈F Ei , using the high confidence feature set R Ci The probability distribution P of similar features to history hist (F Ei ∣R Ci ), calculate the feature credibility according to the credibility evaluation formula The credibility evaluation formula is:

[0050]

[0051] In the formula, n is the total number of features, sim(g) is the similarity function;

[0052] According to the calculated feature credibility, a threshold ξ is set threshold , select features with a credibility higher than the threshold to construct the final feature set F for training final .

[0053] It should be noted that as new data continues to accumulate, the high-confidence feature set and credibility assessment formula should be reviewed and adjusted regularly to ensure that the model can continue to adapt to new environments and scenarios. The above method can effectively reduce the amount of data, reduce model complexity and robustness, and improve event prediction efficiency and accuracy.

[0054] In this embodiment, the high-confidence feature set R high The construction method is as follows: determine the situation assessment target and the feature set to be screened, invite multiple field experts to score the relative importance of each feature in the combination of the features to be screened for the situation assessment target, and construct an original judgment matrix; extract the scores from the original judgment matrix to construct a feature weight data point set, set multiple possible classification data K, use a clustering algorithm to perform negative energy on the feature weight data point set, and obtain classification clusters under different preset classification data; calculate the distance between the cluster center points of two classification clusters to obtain a distance value, and determine a reasonable number of classifications and a corresponding classification data point set based on the comparison result of the distance value and the preset distance; calculate the cluster center point of the classification data point set, and perform elimination and update processing based on the distance data set to eliminate outliers or abnormal data points to obtain a target cluster set; select features occupying multiple target cluster sets as high-confidence features, and then obtain a high-confidence feature set.

[0055] In this embodiment, the similarity function is calculated using cosine similarity or Pearson correlation coefficient. Of course, other similarity calculation methods may also be used, which are not limited in this embodiment.

[0056] First, clarify the goal of situation assessment, such as financial fraud detection, market trend prediction, etc. According to the assessment goal, collect and organize relevant feature sets, which may be extracted from various data sources, such as log data, transaction records, user behavior data, etc. Then invite multiple field experts (such as financial analysts, market experts, etc.) to score the relative importance of each feature in the selected feature set; each expert makes a subjective assessment of the importance of the feature for the situation assessment goal based on his or her own experience and knowledge, and constructs an original judgment matrix; extracts the score values ​​from the original judgment matrix, which represent the expert's judgment on the importance of the feature; integrates the score values ​​of all experts to construct a feature weight data point set, each data point represents the importance score of a feature in the eyes of a certain expert; set multiple possible classification numbers K (such as 2, 3, 4, etc.) to represent the assumed feature importance classification level. Use clustering algorithms (such as K-means, hierarchical clustering, etc.) to classify the feature weight data point set to obtain classification clusters under different preset classification numbers. In the clustering process, the data points are assigned to the nearest cluster by calculating the Euclidean distance or other distance measurement between the score point and the cluster center; the distance between the cluster centers of two adjacent classification clusters is calculated to obtain a series of distance values; a preset distance threshold (or acceptance threshold) is set, and the number of classifications that are reasonable is determined based on the comparison between the distance value and the threshold. Usually, the number of classifications with the largest distance between classification clusters and the highest proportion of majority class data points is selected as the final reasonable number of classifications; the cluster center of the classification data point set is calculated, and the distance data set between each data point and the cluster center is calculated based on the cluster center; the distance data set is updated by elimination, usually by calculating Z-Score or median absolute deviation (MAD) and other methods to identify and eliminate outliers or abnormal data points. The cluster set obtained after elimination is purer and represents the distribution of feature importance with a higher expert consensus; features that appear in multiple target clusters are selected, and these features are considered important in different classification situations; these features are used as high-confidence features to construct a high-confidence feature set.

[0057] The above method, by screening out high-confidence feature training models, can more accurately capture the key factors of situation changes, thereby improving the accuracy of prediction and evaluation.

[0058] Step 3: Calculate the probability P of the event to be predicted transferring from the current state to other states under the state change amount trans (E i →E j |ΔU); according to the probability P trans (E i →E j |ΔU), situation change ΔU pred (t) Determine the status of the event.

[0059] The above method combines historical data and real-time data to dynamically capture the changing trends in the data. Based on the changing trends, it can accurately predict the event status and provide strong support for event warning.

[0060] Step 4: Issue a security warning based on the event status and event importance.

[0061] In this embodiment, performing a security warning according to the event status and the event importance includes: determining a warning level according to the event status and the event importance, and performing a warning according to the warning level.

[0062] The event status refers to the current progress of the security incident, including the various stages of the incident, such as the occurrence, continuation, expansion or end. After obtaining the evaluation results of the event status and the importance of the event, the current event is classified into the corresponding warning level according to the preset warning level classification standards. The warning level is usually divided into multiple levels, such as low, medium, high, emergency, etc., and each level corresponds to different response strategies and urgency.

[0063] It should be noted that the corresponding response measures are initiated according to the warning level. For example, for low-level warnings, only routine monitoring and recording may be required; while for high-level or emergency warnings, emergency plans need to be immediately activated and relevant resources need to be mobilized for rapid response and processing.

[0064] This embodiment comprehensively considers the event state transition, the event situation change, and the event importance to conduct a comprehensive assessment of the event risk, thereby improving the warning accuracy.

[0065] In summary, the embodiments of the present invention provide

[0066] Embodiment 2

[0067] like Figure 2 As shown, the present invention provides an event prediction system based on big data, comprising:

[0068] Data acquisition module, used to collect the initial data set D initial , the initial data set includes at least one historical data related to the event to be predicted;

[0069] Situation awareness module, used for the initial data set D initial Preprocessing is performed and the situation assessment model f(D preprocessed ) predicts the set of situation values ​​at the past k time points {U history (t-1),...,U history (tk)}, using the situation prediction model g(θ,{U history (t-1),...,U history(tk)}) predict the situation change ΔU at time t;

[0070] The state determination module is used to calculate the probability P of the event to be predicted to transfer from the current state to other states under the state change amount. trans (E i →E j |ΔU); according to the probability P trans (E i →E j |ΔU), situation change ΔU pred (t) determine the status of the incident;

[0071] The security warning module is used to issue a security warning based on the event status and event importance.

[0072] It should be understood that the event prediction system based on big data provided in the embodiment of the present invention and the event prediction method based on big data provided in the aforementioned embodiment are based on the same inventive concept. Therefore, for more specific working processes of each module in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiment, which will not be repeated here.

[0073] Embodiment 3

[0074] The above-mentioned remote control vehicle system can be implemented in the form of a computer program. The computer program can be Figure 3 Runs on the computer device shown.

[0075] See also Figure 3 , Figure 3 It is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device may be a terminal, wherein the terminal may be an electronic device with communication function such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The computer device includes a processor, a memory, and an interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system and a computer program. When the computer program is executed, the processor may execute a remote vehicle control method. The processor is used to provide computing and control capabilities to support the operation of the entire computer device. The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium, and when the computer program is executed by the processor, the processor may execute a remote vehicle control method.

[0076] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0077] Embodiment 4

[0078] It is understood by those skilled in the art that all or part of the processes in the method for implementing the above embodiment can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a storage medium, which is a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0079] Therefore, the present invention also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program. The computer-readable storage medium can be the memory of the device provided in the aforementioned embodiment, such as the hard disk or memory of the device. The computer-readable storage medium can also be an external memory of the device, such as a plug-in hard disk equipped on the identification device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium can also include both the internal memory of the identification device and the external memory. The computer-readable storage medium is used to store the computer program and other programs and data. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0080] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. An event prediction method based on big data, characterized in that: include: Collect the initial data set D initial , the initial data set includes at least one historical data related to the event to be predicted; For the initial data set D initial Preprocessing is performed and the situation assessment model f(D preprocessed ) predicts the set of situation values ​​at the past k time points {U history (t-1),...,U history (tk)}, using the situation prediction model g(θ,{U history (t-1),...,U history (tk)}) predict the situation change ΔU at time t; Calculate the probability P of the event to be predicted transferring from the current state to other states under the state change amount trans (E i →E j |ΔU); according to the probability P trans (E i →E j |ΔU), situation change ΔU pred (t) determine the status of the incident; A security warning is issued based on the event status and event importance.

2. The method according to claim 1, characterized in that The pre-processing includes data clarification and formatting uniformity.

3. The method according to claim 2, characterized in that The training data acquisition of the situation assessment model includes: Extract the initial feature set F Ei , for each initial feature set F Ei The feature f in i ∈F Ei , using the high confidence feature set R Ci The probability distribution P of similar features to history hist (F Ei ∣R Ci ), calculate the feature credibility according to the credibility evaluation formula The credibility evaluation formula is: In the formula, n is the total number of features, sim(g) is the similarity function; According to the calculated feature credibility, a threshold ξ is set threshold , select features with a credibility higher than the threshold to construct the final feature set F for training final .

4. The method according to claim 3, characterized in that The high-confidence feature set R high The construction method is: Determine the situation assessment target and the feature set to be screened, invite multiple field experts to score the relative importance of each feature in the feature set to be screened for the situation assessment target, and construct an original judgment matrix; Extract the score from the original judgment matrix, construct a feature weight data point set, set multiple possible classification data K, use a clustering algorithm to perform negative energy on the feature weight data point set, and obtain classification clusters under different preset classification data; Calculate the distance between the cluster center points of two classification clusters to obtain a distance value, and determine a reasonable number of classifications and a corresponding set of classification data points based on a comparison result between the distance value and a preset distance; Calculate the cluster center points of the classified data point set, and perform elimination and update processing based on the distance data set to eliminate outliers or abnormal data points to obtain the target cluster set; The features occupying multiple target cluster sets are selected as high-confidence features, thereby obtaining a high-confidence feature set.

5. The method according to claim 3, characterized in that: The similarity function is calculated using cosine similarity or Pearson correlation coefficient.

6. The method according to claim 1, characterized in that Provide security warnings based on the event status and event importance, including: Determine the warning level according to the event status and event importance, An early warning is issued according to the early warning level.

7. The event prediction system based on big data is characterized by: include: Data acquisition module, used to collect the initial data set D initial , the initial data set includes at least one historical data related to the event to be predicted; Situation awareness module, used for the initial data set D initial Preprocessing is performed and the situation assessment model f(D preprocessed ) predicts the set of situation values ​​at the past k time points {U history (t-1),...,U history (tk)}, using the situation prediction model g(θ,{U history (t-1),...,U history (tk)}) predict the situation change ΔU at time t; The state determination module is used to calculate the probability P of the event to be predicted to transfer from the current state to other states under the state change amount. trans (E i →E j |ΔU); according to the probability P trans (E i →E j |ΔU), situation change ΔUpred(t) determines the event state; The security warning module is used to issue a security warning based on the event status and event importance.

8. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to execute the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 can be implemented.

Images