A threat scenario analysis method and system based on big data

Abstract

The application relates to a big data-based threat scenario analysis method and system. The method comprises the following steps: forming an event rule according to the characteristics of a security threat scenario of log data; importing the event rule into a complex event processing rule engine; and utilizing the complex event processing rule engine to perform threat recognition on the log data to be recognized by using a complex event processing technology, so as to obtain threat alarm data. The application can discover threat information from a large amount of data and timely alarm, has the capability of efficiently processing a large amount of data, and realizes quality improvement and efficiency increase of operation and maintenance, management and decision-making.

Description

Technical Field

[0001] This invention relates to the field of data processing, and in particular to a method and system for threat scenario analysis based on big data. Background Technology

[0002] In the era of big data, data security incidents are emerging one after another, and data theft, tampering, and leakage of personal privacy have become a focus of social concern. Currently, an effective approach is to analyze the characteristics of data security threat scenarios to form event rules, and then use a rule engine to output event alerts. However, real-world environments often present complex attacks. For example, network security device alerts are numerous, scattered, and disorganized, making it difficult to identify security vulnerabilities, and there are many false alarms. Operations and maintenance personnel often act as firefighters, lacking efficient security maintenance methods, and in most cases, they only become aware of security incidents after they have occurred, resulting in a very passive situation. Furthermore, simplistic rules, shallow matching depth, and lack of correlation lead to a large number of missed attack behaviors and inaccurate alerts.

[0003] Based on the above problems, there is an urgent need for a data analysis method or system. Summary of the Invention

[0004] The purpose of this invention is to provide a threat scenario analysis method and system based on big data, which can discover threat information from a large amount of data and issue timely alerts, achieve the ability to process massive amounts of data efficiently, and improve the quality and efficiency of operation, maintenance, management, and decision-making.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A threat scenario analysis method based on big data includes:

[0007] Event rules are formed based on the characteristics of security threat scenarios in log data;

[0008] Import event rules into the complex event handling rule engine;

[0009] The complex event processing rule engine uses complex event processing technology to identify threats in the log data to be identified, and obtains threat alert data.

[0010] Optionally, the step of forming event rules based on the characteristics of security threat scenarios in log data specifically includes:

[0011] Obtain log data from different sources; the log data from different sources includes: server log data, network device log data, security device log data, and traffic log data;

[0012] Normalize log data from different sources and add log type labels;

[0013] Determine the characteristics of threat events based on log data after adding log type tags;

[0014] Based on the characteristics of threat events, a time-series model of threat rules is established and linked with a threat risk attack model to determine the characteristics of security threat scenarios and form event rules.

[0015] Optionally, the step of establishing a threat rule time series model based on the performance characteristics of threat events and associating it with a threat risk attack model to determine the characteristics of security threat scenarios and form event rules specifically includes:

[0016] Establish time windows and triggering conditions based on the threat rule time series model and the associated threat risk attack model.

[0017] Optionally, importing event rules into the complex event processing rule engine specifically includes:

[0018] Transform event rules into an event handling language;

[0019] Import the transformed event rules into the complex event handling rule engine.

[0020] Optionally, the complex event processing rule engine includes: real-time complex event processing and offline complex event processing.

[0021] A threat scenario analysis system based on big data includes:

[0022] The event rule generation module is used to generate event rules based on the characteristics of security threat scenarios in log data;

[0023] The event rule import module is used to import event rules into the complex event processing rule engine;

[0024] The threat alert data determination module is used by the complex event processing rule engine to identify threats in the log data to be identified using complex event processing technology, and to obtain threat alert data.

[0025] A threat scenario analysis system based on big data includes: at least one processor, at least one memory, and computer program instructions stored in the memory. When the computer program instructions are executed by the processor, a threat scenario analysis method based on big data as described above is implemented.

[0026] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0027] This invention provides a threat scenario analysis method and system based on big data. It forms event rules based on the characteristics of security threat scenarios in log data, uses a Complex Event Processing (CEP) rule engine to perform deep correlation analysis on standardized logs, generates security alert events, and pushes them to security operations personnel. Combined with a workflow engine, it completes closed-loop management of the entire security event operation, from protection to detection to response. Utilizing big data analytics and artificial intelligence technologies, and combining threat risk attack and risk analysis models, it performs fusion analysis on different data to discover security events such as configuration violations, network security, data security, and abnormal behavior. Simultaneously, it uses real-time and offline complex event processing to perform multi-dimensional data analysis, calculation, and statistics, calculating an overall risk index to provide data support for the data presentation layer. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of a threat scenario analysis method based on big data provided by the present invention;

[0030] Figure 2 This is a schematic diagram of the overall process of a threat scenario analysis method based on big data provided by the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] The purpose of this invention is to provide a threat scenario analysis method and system based on big data, which can discover threat information from a large amount of data and issue timely alerts, achieve the ability to process massive amounts of data efficiently, and improve the quality and efficiency of operation, maintenance, management, and decision-making.

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1This is a schematic diagram of a threat scenario analysis method based on big data provided by the present invention. Figure 2 This is a schematic diagram illustrating the overall process of a big data-based threat scenario analysis method provided by the present invention, as shown below. Figure 1 and Figure 2 As shown, the present invention provides a threat scenario analysis method based on big data, comprising:

[0035] S101, Formulate event rules based on the characteristics of security threat scenarios in log data;

[0036] S101 specifically includes:

[0037] Obtain log data from different sources; the log data from different sources includes: server log data, network device log data, security device log data, and traffic log data;

[0038] Log data from different sources is normalized and labeled with log type tags. These tags include, but are not limited to, linux-message, linux-secure, windows-secure, net-switch, net-router, safe-apt, safe-ids, safe-ips, safe-waf, safe-firewall, safe-avs, safe-avfw, safe-blj, mw-tomcat-access, mw-tomcat-error, mw-apache-access, mw-apache-error, mw-nginx-access, mw-nginx-error, db-mysql, db-oracle, bns, flow, and other. The data is stored in the data engine and cache message queue.

[0039] The performance characteristics of threat events are determined based on log data after adding log type tags; a fuzzy mathematical model is used to fuse the performance characteristics (field keywords), popularity characteristics, traffic characteristics, and port protocol mismatch characteristics of the threat to be identified, to obtain a comprehensive feature value of the threat to be identified; a set of influencing factors A = {A1, A2, A3, ... A...} is defined. n}, A = {Performance characteristics (field keywords), popularity characteristics, traffic characteristics, port protocol mismatch characteristics};

[0040] The evaluation set used is defined as W = {W1, W2, W3, ..., W...}. m};

[0041] Define a single factorial: f1: A→(W), Ai|→f1(A i )=(a i ,1,a i ,2,...,a i ,m)∈(W), where a i,j In this context, i and j satisfy 1≤i≤n and 1≤j≤m respectively, and their values ​​represent A. i In W m’ The evaluation value of the factor, and a i,1 +a i,2 +a i,3 +...+a i,m =1, and through the intermediate fuzzy mapping, the fuzzy relationship is obtained.

[0042] A predefined weight matrix Z is defined; Z = [Z1, Z2, ..., Zn], where z1 + z2 + ... + zn = ... n =1, where the value of each element in the set represents the importance of each influencing factor in the set. R and Z are combined using a maximum-minimum operation to obtain the comprehensive value of feature extraction.

[0043] Based on the characteristics of threat events, a threat rule time-series model is established and linked with the threat risk attack model to determine the characteristics of security threat scenarios and form event rules. The threat rule time-series model is the first step to generate key features, then the second step generates key actions, and finally, key data is generated.

[0044] The Threat Risk Attack Model describes the actions an adversary takes to achieve its tactical objectives. Within each tactical category, a limited number of actions can accomplish the tactic's goal. Throughout their compromised operations, the adversary continuously decides which techniques to use based on knowledge, acquired information about the target environment, information needed for future actions, and currently available capabilities. The techniques are described in a way that is independent of specific adversary malware and tools. The advantage of this approach is that it encompasses the behavior exhibited by the adversary through interactions with remote access tools, scripts, or command-line interfaces. The associated Threat Risk Attack Model includes 12 tactics as follows:

[0045] (1) Initial Access

[0046] Initial access is an attacker's foothold in an enterprise environment, and attackers will use different techniques to achieve this goal.

[0047] (2) Execution

[0048] Regardless of the methods attackers use, only by "executing" their tactics can they ultimately achieve their attack objectives. Malware or code must be executed, giving security personnel the opportunity to block or detect it. However, not all malicious code is easily detected. Attackers may meticulously package it, using obfuscation or even automatic backups to hide it. In cases where machines cannot scan it, manual intervention from administrators is necessary.

[0049] (3)Persistence

[0050] For ransomware, most attackers' survival time depends solely on when they are detected by the system. Even after an attacker successfully "persistes," and even if system administrators take measures such as restarting or changing credentials, persistence can still allow the computer to be reinfected or maintain its existing connections. Examples include "modifying the registry, boot folders, and IFEO (Initial Image Execution)."

[0051] (4) Privilege escalation

[0052] Not every attacker can use an administrator account to launch an attack. Successful privilege escalation often signifies a stage victory in an intrusion attack. Exploiting system vulnerabilities to gain root-level access is one of the core objectives of attackers.

[0053] (5) Defense Bypass

[0054] "Defense bypass" tactics include techniques that allow malicious code to circumvent defenses, rendering them ineffective or even bypassing whitelisting mechanisms. Examples include modifying registry keys, deleting core files, and disabling security tools. To counter this technique, defenders can monitor for unusual changes on terminals and collect logs from critical systems, making it impossible for intrusions to bypass the defenses.

[0055] (6) Credential Access

[0056] "Credential access" is also a common tactic among attackers, as "credentials" themselves are a key target. With credentials, attackers can save significant attack costs while reducing the risk of detection. This is because even the strongest fortresses are often easily breached from within.

[0057] (7) Discovery

[0058] In enterprise environments, the normal operation of business inevitably exposes valuable information, which is often a target for attackers. Containerized enterprise environments make this tactic even more difficult to defend against, as evidenced by the frequent reports of user privacy breaches.

[0059] (8) Lateral movement

[0060] After gaining access to a system, attackers typically attempt to "move laterally" within the network, whether to gather information or to find entry points for their next attack. Attackers usually begin by establishing a foothold and then start making every possible move across various systems to gain better access privileges and ultimately control the entire network.

[0061] (9) Collection

[0062] "Collection" tactics are techniques used by attackers to discover and collect data needed to achieve their goals. We can use whitelisting mechanisms to defend against such anomalous behavior.

[0063] (10) Command and Control

[0064] Most malware possesses a degree of command and control. Attackers can use this control to compromise data and manipulate malicious code. For each type of command and control, the attacker gains access to the network from a remote location. Therefore, real-time monitoring, command, and control are crucial for countering these techniques.

[0065] (11) Data leakage

[0066] Once attackers gain access, they search for relevant data and begin data infiltration. However, not all malware reaches this stage. For example, ransomware typically has no interest in gradually leaking data. Similar to "collection" tactics, whitelisting mechanisms can effectively address this situation.

[0067] (12) Impact

[0068] The techniques used for "impact" include, but are not limited to, data corruption or tampering, where attackers attempt to manipulate, interfere with, or disrupt a company's systems and data. In extreme cases, business processes may appear to be functioning correctly on the surface, but have actually been secretly altered.

[0069] Threat risk attack model tactical collision, illustrated with the following example:

[0070] Behavior: Executed remotely using a scheduled task via schtasks.exe.

[0071] Threat risk attack model tactical collision requirements:

[0072] 1. Provide SMB (Server Message Block - Windows File Sharing) credentials or existing domain permissions to access the remote system.

[0073] 2. Able to move a file to a remote system so that it can be executed according to a scheduled task.

[0074] 3. Allows schtasks.exe to run on the local system. By default, any user can run schtasks.exe.

[0075] 4. Management access to remote systems, scheduling tasks via remote procedure calls.

[0076] Threat and Risk Attack Model Tactical Collision Reasons:

[0077] 1. Invoke schtasks.exe in the command line interface, with parameters to execute the file on the remote system.

[0078] Threat and Risk Attack Model Tactical Collision Effect:

[0079] The 1.exe process is launched on the local system.

[0080] 2. Establish an RPC connection from the local system to the target system.

[0081] 3. The entry for this task is located in the "%SystemRoot%\Tasks\M" directory of the remote system.

[0082] 4. Files on the remote system are executed as child processes of taskeng.exe at a specified time.

[0083] 5. Subsequent system changes were caused by the execution of binary files or scripts. For example, if the program is a remote access tool, the resulting process might attempt to open a network connection.

[0084] Establish time windows and triggering conditions based on the threat rule time series model and the associated threat risk attack model.

[0085] S102, import the event rules into the complex event processing rule engine;

[0086] S102 specifically includes:

[0087] Transform event rules into an event handling language;

[0088] Import the transformed event rules into the complex event handling rule engine.

[0089] The complex event processing rule engine includes: real-time complex event processing and offline complex event processing.

[0090] Real-time complex event processing acquires real-time log data from device operation, uses the CEP engine to detect abnormal access behavior, and generates security events, thereby achieving real-time security protection.

[0091] Offline complex event processing employs a minimum log source bidirectional association algorithm to search from any stage, combining association conditions to iteratively search and complete aggregate association analysis in an in-memory database, thereby discovering threat events.

[0092] S103, the complex event processing rule engine uses complex event processing technology to identify threats in the log data to be identified, and obtains threat alarm data.

[0093] The steps for implementing complex event handling (CEP) rule engine are as follows:

[0094] Step 1: Based on the threat rules, summarize the event detection logic and convert it into an event handling language (EPL);

[0095] Step 1.1: Issue an alarm for detected security events.

[0096] Step 1.2: Organize the detected security event logic, form an event description process, and add it to the rule base.

[0097] Step 2: Integrate the EPL into the CEP rule engine and execute it;

[0098] Step 2.1: Load the EPL into the CEP rule engine. Through parsing and verifying the EPL and other processes, a physical execution flowchart is finally obtained and distributed to each worker node in the cluster. Real-time log streams are detected according to event matching rules to form a complete process of rules, monitoring and alarms.

[0099] Step 3: Load different data sources based on the different data model types;

[0100] Step 3.1: Using streaming analytics, data is monitored from a Kafka queue, and data is read in real time for analysis and matching with threat rule models;

[0101] Step 3.2: Static analysis data comes from big data storage components (HDFS, Elasticsearch, etc.), and data is read and analyzed periodically to match threat rule models;

[0102] Step 4: Correlate the threat risk attack model to identify the stage at which the threat event exists;

[0103] Step 5: Generate a security incident and link it to the original data and the asset information involved;

[0104] Step 6: Closed-loop handling of security incidents, completing the closed-loop management of the entire security incident operation from protection to detection to response.

[0105] To address the above methods, this invention provides a threat scenario analysis system based on big data, comprising:

[0106] The event rule generation module is used to generate event rules based on the characteristics of security threat scenarios in log data;

[0107] The event rule import module is used to import event rules into the complex event processing rule engine;

[0108] The threat alert data determination module is used by the complex event processing rule engine to identify threats in the log data to be identified using complex event processing technology, and to obtain threat alert data.

[0109] In order to execute the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, the present invention also provides a threat scenario analysis system based on big data, including: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, a threat scenario analysis method based on big data as described above is implemented.

[0110] This invention employs complex event processing technology to analyze and process various types of data through correlation, time-series analysis, and aggregation. It receives security logs, behavior logs, and data access and transmission logs from different devices, including servers, network devices, and security devices. After standardization through platform parsing, enhancement, and normalization, the standardized logs are used with the CEP rule engine to perform deep correlation analysis, generate security alert events, and push them to security operations personnel for work order processing. This completes the closed-loop management of the entire security event operation, from protection to detection to response.

[0111] This invention relies on a big data analysis platform, utilizes big data analysis and artificial intelligence technologies, and combines threat risk attack and risk analysis models to perform integrated analysis on different data, discover security events such as configuration violations, network security, data security, and abnormal behavior. At the same time, it uses real-time complex event processing and offline complex event processing to perform multi-dimensional analysis, calculation and statistics of data, calculate the overall risk index, and provide data support for the data display layer.

[0112] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0113] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A threat scenario analysis method based on big data, characterized in that, include: Event rules are formed based on the characteristics of security threat scenarios in log data; Import event rules into the complex event handling rule engine; The complex event processing rule engine uses complex event processing technology to identify threats in the log data to be identified, and obtains threat alert data. The process of forming event rules based on the characteristics of security threat scenarios in log data specifically includes: Obtain log data from different sources; the log data from different sources includes: server log data, network device log data, security device log data, and traffic log data; Normalize log data from different sources and add log type labels; The characteristics of threat events are determined based on log data after adding log type tags; a fuzzy mathematical model is used to fuse influencing factors to obtain the characteristics; the influencing factors include: field keywords, popularity characteristics, traffic characteristics, and port protocol mismatch characteristics; Based on the characteristics of threat events, a time-series model of threat rules is established and linked with a threat risk attack model to determine the characteristics of security threat scenarios and form event rules.

2. The threat scenario analysis method based on big data according to claim 1, characterized in that, The process of establishing a time-series model of threat rules based on the performance characteristics of threat events, and linking it with a threat risk attack model to determine the characteristics of security threat scenarios and form event rules, specifically includes: Establish time windows and triggering conditions based on the threat rule time series model and the associated threat risk attack model.

3. The threat scenario analysis method based on big data according to claim 1, characterized in that, The process of importing event rules into a complex event processing rule engine specifically includes: Transform event rules into an event handling language; Import the transformed event rules into the complex event handling rule engine.

4. The threat scenario analysis method based on big data according to claim 1, characterized in that, The complex event processing rule engine includes: real-time complex event processing and offline complex event processing.

5. A threat scenario analysis system based on big data, used to implement the threat scenario analysis method based on big data as described in any one of claims 1-4, characterized in that, include: The event rule generation module is used to generate event rules based on the characteristics of security threat scenarios in log data; The event rule import module is used to import event rules into the complex event processing rule engine; The threat alert data determination module is used by the complex event processing rule engine to identify threats in the log data to be identified using complex event processing technology, and to obtain threat alert data.

6. A threat scenario analysis system based on big data, characterized in that, include: The system comprises at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, the system implements a big data-based threat scenario analysis method as described in any one of claims 1-4.

Images