Method and device for automatically generating threat intelligence

Abstract

The invention discloses a threat intelligence automatic generation method and device. The method comprises the following steps: acquiring safety behavior data in real time; performing feature extraction on the security behavior data to obtain target behavior features; comparing the target behavior characteristics with a boundary threshold range of a pre-trained normal baseline model; the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction; inputting the target behavior characteristics into a pre-trained anomaly detection model to obtain a current anomaly score; the anomaly detection model is obtained by constructing a random isolation tree based on a historical feature matrix obtained based on historical behavior features and training based on the random isolation tree; if the current exception score is greater than a preset exception sample threshold value, judging the security behavior data as first exception data; and the first abnormal data is processed to obtain threat intelligence, so that the potential abnormal behaviors are accurately identified, and the threat detection efficiency and response capability in a complex environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data analysis technology, and in particular to a method and device for automatically generating threat intelligence. Background Art

[0002] As cyberattacks become increasingly diverse and complex, traditional security protection methods that rely on manual analysis are no longer sufficient to meet the rapidly evolving cybersecurity landscape. To address this ever-changing threat landscape, a technical solution is urgently needed that can efficiently, intelligently, and dynamically identify potential threats and facilitate rapid response and disposal.

[0003] Threat intelligence refers to the collection, analysis, and sharing of information about potential or existing attacks in network environments, helping organizations identify early signs of an attack, analyze attacker behavior patterns, and implement proactive protective measures. This intelligence typically includes attacker identity, attack methods, targets, active time periods, and behavioral trends. In practical applications, one of the core functions of threat intelligence is to extract and organize indicators of compromise (IoCs) for system detection.

[0004] IoCs (Indicators of Compromise) are key features that can be used to determine whether a system has been attacked or compromised. These include the hash value of a malicious file, the IP address used by the attacker, the domain name used to communicate with the remote server, or the sender address of a phishing email. For example, the SHA-256 hash value of a piece of malware can serve as its unique identifier; the presence of a file corresponding to this hash value in the system may indicate a compromise. Similarly, the presence of connections to known malicious IP addresses or domains in communication logs also indicates a security risk.

[0005] However, over time, attackers' strategies and behavior patterns continue to evolve, leading to "data drift" in network behavior characteristics. This means that existing threat identification models may no longer be adapted to new attack patterns. Therefore, key technical challenges in network security are urgently needed: how to promptly identify and adapt to data drift, accurately identify abnormal behavior from massive amounts of distributed data, and dynamically adjust detection models and efficiently generate indicators of compromise (IoCs). To this end, there is an urgent need to build an intelligent security defense mechanism with real-time, adaptive, and automated analysis capabilities to significantly improve threat identification accuracy and response efficiency.

[0006] This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section. Summary of the Invention

[0007] The embodiment of the present invention provides a method for automatically generating threat intelligence, which is used to accurately identify potential abnormal behaviors and generate structured threat intelligence, thereby improving the threat detection efficiency and response capabilities in complex environments.

[0008] Methods for automated threat intelligence generation include:

[0009] Real-time acquisition of security behavior data; the security behavior data includes: network traffic monitoring data, user behavior logs and terminal security events;

[0010] Extracting features from the security behavior data to obtain target behavior features;

[0011] Comparing the target behavior feature with a boundary threshold range of a pre-trained normal baseline model; wherein the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction;

[0012] Inputting the target behavior features into a pre-trained anomaly detection model to obtain a current anomaly score; wherein the anomaly detection model is constructed by constructing a random isolation tree based on a historical feature matrix obtained based on the historical behavior features, and is trained based on the random isolation tree;

[0013] If the current anomaly score is greater than a preset abnormal sample threshold, the safety behavior data is determined to be first abnormal data;

[0014] The first abnormal data is processed to obtain threat intelligence.

[0015] In one embodiment, comparing the target behavior feature with a boundary threshold range of a pre-trained normal baseline model includes:

[0016] If the target behavior feature exceeds the boundary threshold range, the safe behavior data is determined to be second abnormal data.

[0017] In one embodiment, processing the first abnormal data to obtain threat intelligence includes:

[0018] Data processing is performed on the first abnormal data and the second abnormal data to obtain threat intelligence.

[0019] In one embodiment, the real-time acquisition of safety behavior data includes:

[0020] Creating corresponding topics for the network traffic monitoring data, the user behavior log, and the terminal security event in a distributed data collection system;

[0021] Data producers and data consumers are configured in a distributed data collection system so that the data consumers receive the security behavior data corresponding to the topic in real time.

[0022] In one embodiment, the step of establishing the normal baseline model includes:

[0023] Performing feature extraction on the historical network traffic monitoring data and the historical user behavior logs to obtain behavior feature data;

[0024] Determine a boundary threshold range of abnormal values based on the behavior characteristic data.

[0025] In one embodiment, determining a boundary threshold range of an abnormal value based on the behavioral characteristic data includes:

[0026] determining a first quartile and a third quartile based on the behavioral characteristic data;

[0027] determining an interquartile range based on the first quartile and the third quartile;

[0028] An upper boundary threshold and a lower boundary threshold of an outlier are determined according to the first quartile, the third quartile, and the interquartile range.

[0029] In one embodiment, the method further comprises:

[0030] Determine updated model parameters based on current model parameters, a learning rate, true observation values corresponding to the safety behavior data, and the first abnormal data to update the normal baseline model.

[0031] In one embodiment, the training step of the anomaly detection model includes:

[0032] Extract historical behavior features from historical security behavior data and construct a historical feature matrix.

[0033] constructing a plurality of random isolation trees according to the historical feature matrix;

[0034] Determining an average path length of each historical behavior feature in the historical feature matrix in a plurality of the random isolation trees;

[0035] The average path length is converted into a historical anomaly score.

[0036] In one embodiment, constructing a plurality of random isolation trees according to the historical feature matrix includes:

[0037] Randomly select a segmentation feature in the historical feature matrix;

[0038] Randomly select a segmentation point from the value range corresponding to the segmentation feature;

[0039] Dividing the historical feature matrix according to the split point to construct a left leaf node and a right leaf node of a random isolation tree;

[0040] The above steps are iteratively performed until a preset termination condition is met, and then the random isolation tree construction is stopped.

[0041] In one embodiment, the processing the first abnormal data and the second abnormal data to obtain threat intelligence includes:

[0042] Performing field extraction on the first abnormal data and the second abnormal data to obtain a threat information field;

[0043] Determine the threat information field based on preset rules to generate a threat indicator;

[0044] The threat indicators are integrated with the security behavior data to obtain threat intelligence.

[0045] An embodiment of the present invention also provides a device for automatically generating threat intelligence, which is used to accurately identify potential abnormal behaviors and generate structured threat intelligence, thereby improving threat detection efficiency and response capabilities in complex environments.

[0046] The threat intelligence automated generation device includes:

[0047] A data acquisition module is used to acquire security behavior data in real time; the security behavior data includes: network traffic monitoring data, user behavior logs and terminal security events;

[0048] A feature extraction module is used to extract features from the security behavior data to obtain target behavior features;

[0049] A threshold comparison module, configured to compare the target behavior feature with a boundary threshold range of a pre-trained normal baseline model; wherein the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction;

[0050] An anomaly detection module, configured to input the target behavior features into a pre-trained anomaly detection model to obtain a current anomaly score; wherein the anomaly detection model is constructed by constructing a random isolation tree based on a historical feature matrix obtained based on the historical behavior features, and is trained based on the random isolation tree;

[0051] a data determination module, configured to determine the safety behavior data as first abnormal data if the current abnormal score is greater than a preset abnormal sample threshold;

[0052] The threat intelligence generation module is used to process the first abnormal data to obtain threat intelligence.

[0053] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for automatically generating threat intelligence when executing the computer program.

[0054] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned method for automatically generating threat intelligence when executed by a processor.

[0055] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for automatically generating threat intelligence.

[0056] The method and device for automatically generating threat intelligence provided by the embodiment of the present invention ensure the comprehensiveness and real-time nature of data analysis by acquiring multi-source security behavior data such as network traffic monitoring data, user behavior logs, and terminal security events in real time. On this basis, the target behavior features are extracted through feature extraction technology, which improves the efficiency and accuracy of subsequent model processing. By comparing the extracted target behavior features with the threshold range of the normal baseline model, abnormal data that deviates from the normal baseline can be effectively identified. The security behavior data is input into the anomaly detection model, and the anomaly detection model is constructed through the random forest algorithm to obtain the current abnormal behavior score, and the preset threshold is combined for judgment, so as to accurately identify high-risk security behavior data. The identified abnormal data is structured to generate threat intelligence, thereby achieving a rapid response to potential network attacks. The present invention not only improves the accuracy and response speed of threat identification, but also enhances the threat monitoring capability of the network security system in a complex environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0058] Figure 1 Schematic diagram of a process for automatically generating threat intelligence according to an embodiment of the present invention;

[0059] Figure 2 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0060] Figure 3 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0061] Figure 4 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0062] Figure 5 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0063] Figure 6 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0064] Figure 7 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0065] Figure 8 Schematic diagram of a flow chart of a method for automatically generating threat intelligence in another embodiment of the present invention;

[0066] Figure 9 This is a schematic diagram of the structure of a device for automatically generating threat intelligence in another embodiment of the present invention;

[0067] Figure 10 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0069] The information collected in the technical solution of this application is information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0070] Provide users with corresponding operation entrances for them to choose to agree or reject the automated decision-making results; if the user chooses to reject, they will enter the expert decision-making process.

[0071] In order to improve the ability to identify and respond to potential security threats in a complex and changing network environment, and to solve the problems of traditional threat intelligence and threat indicator (Indicators of Compromise, IoC) generation methods relying on manual judgment, delayed response, high false alarm rate and difficulty in adapting to data drift, this application provides a method for automatically generating threat intelligence. This method constructs a complete data processing and analysis process. First, it uses a distributed system to collect multi-source data in real time, and improves data quality through data cleaning and standardization. Subsequently, a normal behavior baseline model that can be dynamically updated is established based on historical security behavior data, and a machine learning algorithm is introduced to construct an anomaly detection model to identify potential threats that deviate from normal behavior. After detecting abnormal behavior, the system can automatically generate corresponding threat indicators, integrate them with existing threat intelligence, and finally output a visual analysis report for reference and analysis by the security team. This method effectively improves the accuracy and response efficiency of threat identification, enhances the monitoring and defense capabilities of complex attack behaviors, and significantly improves the intelligence level of network security systems in dynamic environments.

[0072] like Figure 1 As shown, the method for automatically generating threat intelligence includes: steps 101 to 106.

[0073] Step 101: Acquire security behavior data in real time, including network traffic monitoring data, user behavior logs, and terminal security events.

[0074] Step 102: Extract features from the security behavior data to obtain target behavior features.

[0075] Step 103: Compare the target behavior feature with the boundary threshold range of the pre-trained normal baseline model, wherein the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction.

[0076] Step 104: Input the security behavior data into a pre-trained anomaly detection model to obtain a current anomaly score; wherein the anomaly detection model is constructed based on a historical feature matrix obtained from a feature vector set and trained based on the random isolation tree.

[0077] Step 105: If the current anomaly score is greater than a preset anomaly threshold, the safety behavior data is determined to be abnormal data.

[0078] Step 106: Process the abnormal data to obtain threat intelligence.

[0079] In an embodiment of the present invention, by acquiring multi-source security behavior data such as network traffic monitoring data, user behavior logs, and terminal security events in real time, the comprehensiveness and real-time nature of data analysis are ensured. On this basis, the target behavior features are extracted through feature extraction technology, which improves the efficiency and accuracy of subsequent model processing. By comparing the extracted target behavior features with the threshold range of the normal baseline model, abnormal data that deviates from the normal baseline can be effectively identified. The security behavior data is input into the anomaly detection model, and the anomaly detection model is constructed through the random forest algorithm to obtain the current abnormal behavior score, and the preset threshold is combined for judgment, so as to accurately identify high-risk security behavior data. The identified abnormal data is structured to generate threat intelligence, thereby achieving a rapid response to potential network attacks. The present invention not only improves the accuracy and response speed of threat identification, but also enhances the threat monitoring capability of the network security system in complex environments.

[0080] like Figure 1 As shown, each step is explained in detail below. The execution subject of this application is the threat intelligence automatic generation system.

[0081] Step 101: Acquire security behavior data in real time, including network traffic monitoring data, user behavior logs, and terminal security events.

[0082] Specifically, key security behavior data such as network traffic monitoring data, user behavior logs, and endpoint security events are collected in real time through distributed data collection systems (such as Apache Kafka, Apache Flume, or Apache NiFi).

[0083] By setting up a network probe or using a network monitoring tool to capture network data packets and connection behavior, you can obtain network traffic monitoring data. Network monitoring tools include Wireshark or NetFlow. Network traffic monitoring data includes source IP address, destination IP address, source port number, destination port number, transport layer protocol type, total number of bytes sent by the client during communication, and total number of bytes received by the client during communication. Network traffic monitoring data can reflect the communication behavior between hosts on the network.

[0084] User behavior logs are generated by extracting data related to user behavior from operating systems, identity authentication services (such as Active Directory), business systems, or application logs. Application logs include Windows EventLog, Linux auditd, and web application logs. User behavior logs contain the user's unique identifier, the time the behavior event occurred, the type of user operation, the source IP address of the user operation, and the user's terminal device number. User behavior logs are used to record interactions between users and computer resources.

[0085] By deploying an EDR (Endpoint Detection and Response) system, you can monitor and capture various endpoint security events in real time on every endpoint device (such as an employee's computer or server). Examples of EDR systems include CrowdStrike Falcon and VMware Carbon Black. Endpoint security events include the unique identifier of the host device, the time the security event occurred, the type of security event, the hash value of the suspicious file, and the path to the suspicious file.

[0086] Since different data sources output data in different formats (such as JSON format, XML format, CSV format, or Syslog format, etc.), by determining the data format used by each data source and standardizing it into a unified data format (such as JSON format), the field names, timestamp formats, and character encodings of different data sources can be guaranteed to be consistent.

[0087] In one embodiment, Figure 2 As shown, step 101 includes steps 201 to 202.

[0088] Step 201: Create corresponding topics for network traffic monitoring data, user behavior logs, and terminal security events in a distributed data collection system.

[0089] Specifically, in the Apache Kafka system, corresponding topics are created for each type of security behavior data through the command line or management interface, which can realize the classification and management of different data sources.

[0090] For example, if the data source type is network traffic monitoring data, the corresponding Apache Kafka system topic name is network-traffic; if the data source type is user behavior log, the corresponding Apache Kafka system topic name is user-behavior; if the data source type is terminal security events, the corresponding Apache Kafka system topic name is endpoint-security.

[0091] Step 202: Configure data producers and data consumers in the distributed data collection system so that data consumers receive security behavior data corresponding to the topic in real time.

[0092] Specifically, first, in the Apache Kafka system, initialize the data producer (Producer). The data producer is the data input end of the Apache Kafka system, which is used to send the collected network traffic monitoring data, user behavior logs, terminal security events and other security behavior data to the created topics.

[0093] The steps of initializing the data producer include: setting the access address of the Apache Kafka system, and performing format conversion operations on the above-mentioned security behavior data (such as uniform conversion into JSON format).

[0094] After initializing the data producer, different types of security behavior data are routed to corresponding topics. For example, network traffic monitoring data is sent to the network-traffic topic, user behavior logs are sent to the user-behavior topic, and endpoint security events are sent to the endpoint-security topic. Through this approach, the Apache Kafka system can implement categorized transmission of different types of security behavior data, ensuring that the Apache Kafka system's message channels continuously receive the latest data from various data sources.

[0095] After configuring the data producer, further configure a data consumer in the Apache Kafka system to receive and aggregate security behavior data from various topics in the system in real time. The data consumer continuously listens to specified topics, receiving real-time security behavior data corresponding to the topic and storing it in a backend database (such as Elasticsearch or Hadoop HDFS).

[0096] First, in the Apache Kafka system, initialize a data consumer and subscribe to the specified topics. For example, subscribe to the network-traffic topic, the user-behavior topic, and the endpoint-security topic. The network-traffic topic receives network traffic monitoring data, the user-behavior topic receives user behavior logs, and the endpoint-security topic receives endpoint security events.

[0097] After initializing the data consumer, you can configure the parameters of the data consumer to ensure that the data consumer can run normally in the Apache Kafka system.

[0098] Specifically, by setting the connection address and port information of the Apache Kafka system, data consumers are connected to the Apache Kafka system to achieve effective monitoring of designated topics. Then, the above-mentioned security behavior data is formatted and uniformly converted into JSON format. Since the data transmitted in the Apache Kafka system exists in the form of a byte stream, after receiving the security behavior data, the data consumer needs to decode the byte stream into a string using the UTF-8 encoding method, and further parse it into a structured data object using the JSON format for subsequent processing and storage. In addition, by setting a data displacement reset strategy, data consumers can regularly synchronize and record the received data displacement information to the Apache Kafka system, thereby effectively avoiding repeated data reception. By configuring the automatic submission mechanism, the Apache Kafka system can accurately restore to the last reception progress after a failure or restart, ensuring the continuity and consistency of data processing.

[0099] After establishing a connection between the data consumer and the Apache Kafka system, the data consumer enters a data listening loop, continuously receiving and processing security behavior data from subscribed topics in real time. For each piece of security behavior data received, the Apache Kafka system extracts the topic name and data content. The topic name identifies the data type of the message source, such as network traffic monitoring data, user behavior logs, and terminal security events. Based on the topic type, the Apache Kafka system stores the security behavior data in a backend database (such as Elasticsearch).

[0100] Specifically, the subject name of the security behavior data is used as the index identifier in the backend database Elasticsearch, and the parsed data content is written to the index as document content, thus achieving unified storage of structured data. Through this approach, security behavior data from different sources can be clearly and orderly managed in a hierarchical manner, providing a high-quality data foundation for subsequent threat detection, threat intelligence generation, and visual analysis.

[0101] In one embodiment, after completing the above-mentioned data classification and storage, in order to further ensure the stability of the data processing process and the reliability of the analysis results, the Apache Kafka system also adopts a data verification and monitoring mechanism to ensure that the data written to the backend database is not lost or tampered with during the data transmission and storage process.

[0102] Specifically, after the Apache Kafka system stores data in a backend database, it implements a regular data validation mechanism to verify the integrity and accuracy of the data written to the backend database. This data validation can utilize data quality detection components integrated into data processing platforms (such as Apache NiFi) to automatically perform operations such as data field integrity checks, data format consistency checks, and duplicate data detection. This data validation mechanism enables the Apache Kafka system to promptly detect issues such as missing data or abnormal data structures caused by network delays, transmission anomalies, or system failures.

[0103] In one embodiment, after the configuration of the data producer and the data consumer is completed, data preprocessing is performed on the collected security behavior data.

[0104] like Figure 3 As shown, the step of preprocessing the collected safety behavior data includes: step 301 to step 305.

[0105] Step 301: Load the collected safety behavior data from the backend database.

[0106] Specifically, during the data loading phase, functional components for data processing and database interaction are used, such as the Python data analysis library Pandas or the Elasticsearch client library. The Pandas data analysis library is primarily used to convert database query results into a structured DataFrame, while the Elasticsearch client library is used to establish a connection with the backend database and perform data query operations.

[0107] Furthermore, an Elasticsearch client instance is created and connected to the locally deployed Elasticsearch client to initialize the Elasticsearch client. The listening port is 9200.

[0108] After completing the initialization operation of the above-mentioned Elasticsearch client, the threat intelligence automatic generation system extracts security behavior data from the above-mentioned indexes in turn and converts it into a structured data set.

[0109] Specifically, first, the network traffic monitoring data is queried from the index with the subject name "network-traffic", and the query results are converted into a tabular form using the DataFrame function of the data analysis library Pandas. Furthermore, the original nested JSON data structure is flattened using the jsonnormalize function, and the source field is extracted to form a structured network traffic data set. Secondly, the user behavior log is queried from the index with the subject name "user-behavior", and the data is processed in the above manner to obtain a structured user behavior data set. Finally, the terminal security events are extracted from the index with the subject name "endpoint-security", and the data is processed in the above manner to obtain a structured terminal security event data set. Through the above steps, the threat intelligence automation generation system realizes the unified loading and structured conversion of multiple types of security behavior data.

[0110] Step 302: De-duplication operations are performed on various types of security behavior data sets, wherein the security behavior data sets include: network traffic data sets (networkdata), user behavior data sets (userdata), and terminal security event data sets (endpointdata).

[0111] Specifically, for the network traffic data set, the timestamp (timestamp) field, source IP address (src_ip) field, and destination IP address (dst_ip) field are used as judgment criteria to identify and delete duplicate records in the network traffic data set that are identical to the above three fields. For the user behavior data set, the operation time (timestamp) field and user identification (userid) field are used as judgment criteria to identify and delete duplicate records generated by the same user at the same time point in the user behavior data set to ensure that each user operation record is unique. For the terminal security event data set, the event time (timestamp), device number (deviceid), and event type (eventtype) are used as judgment criteria to identify and delete duplicate security events of the terminal. The above deduplication operation reduces data redundancy and improves data consistency and accuracy.

[0112] Step 303: Standardize each type of safety behavior data set.

[0113] Specifically, first, the timestamp fields in various security behavior datasets are standardized. Using the todatetime function provided by the data analysis library Pandas, the timestamp fields representing time in the network traffic dataset, user behavior dataset, and terminal security event dataset are uniformly converted to a standardized time format. Secondly, the IP address fields in the network traffic dataset are standardized. Using the apply function and the lambda function, string standardization is performed on the source IP address field and the destination IP address field. That is, a strip operation is performed on each IP address field to remove blank characters at the beginning and end of the string. Then, a lower operation is performed to uniformly convert the IP address fields to lowercase. Finally, the action field in the user behavior dataset is standardized. All behavior labels are uniformly converted to lowercase to eliminate label redundancy and recognition errors caused by inconsistent uppercase and lowercase letters. Through the above standardization, the format of multiple types of security behavior data is unified, and the structural integrity of the data is improved.

[0114] Step 304: Perform type conversion processing on each type of security behavior data set.

[0115] Specifically, the bytes field in the network traffic dataset is converted to an integer type by calling the astype(int) function. The bytes field is uniformly converted from a string type or a floating-point type to an integer type. The user ID field in the user behavior dataset is converted to an integer type by using the astype('category') function. The user ID field is converted from a string type to a categorical variable type. The terminal security event type (eventtype) field in the terminal security event dataset is converted to an event type by using the astype('category') function. The above type conversion processing can effectively improve the subsequent model's ability to identify different event categories (such as malicious code execution, abnormal login, and illegal process startup).

[0116] Step 305: Perform data integrity check and missing value processing on each type of security behavior data set.

[0117] Specifically, we first perform an integrity check on each security behavior dataset. By calling the isnullsum function, we perform a field-by-field statistical analysis of missing values in each security behavior dataset. This generates missing value statistics (networkmissing) for network traffic monitoring data, usermissing for user behavior logs, and endpointmissing for endpoint security events. These missing value statistics are then output to help the security team understand the integrity distribution of each field in each security behavior dataset.

[0118] Furthermore, different missing value processing methods were used for each type of security behavior dataset. For the network traffic dataset, a forward-filling method was used to fill missing fields with the field values of the previous non-empty record. For the user behavior dataset, all missing fields were uniformly filled with "unknown," where "unknown" represents undefined or unrecognizable user behavior. For the terminal security event dataset, the dropna function was used to directly delete entire data records containing missing fields to prevent abnormal data from interfering with subsequent modeling.

[0119] In an embodiment of the present invention, by loading the collected multi-source security behavior data such as network traffic monitoring data, user behavior logs, and terminal security events from the back-end database, it is ensured that the security behavior data has a unified access interface. According to the preset key field combination, the duplicate records that may exist in various types of security behavior data sets are deduplicated, which effectively eliminates redundant information and avoids the interference of duplicate data on subsequent model training. On this basis, the key fields of the security behavior data are standardized to improve the format compatibility between multi-source security behavior data. The key fields are further type-converted to improve the adaptability of the data format and subsequent modeling algorithms. Through the integrity check and missing value processing mechanism, the automatic identification and statistical analysis of missing fields in the security behavior data are realized, and forward filling, default value filling or record elimination operations are selectively performed according to the field attributes, thereby ensuring the stability and integrity of the model input data. Through the above-mentioned data preprocessing operations, the data structure standardization and data quality reliability of the security behavior data are comprehensively improved.

[0120] Step 102: Extract features from the security behavior data to obtain target behavior features.

[0121] Specifically, from the preprocessed security behavior dataset, feature extraction is performed on network traffic monitoring data and user behavior logs to obtain target behavior features.

[0122] The target behavior features extracted from the network traffic data set include: timestamp (timestamp), that is, the time when the data packet is captured or the connection occurs, source IP address (src_ip), destination IP address (dst_ip), number of bytes (bytes), that is, the number of bytes transmitted in a single flow, protocol type (protocol), such as TCP protocol, UDP protocol and ICMP protocol, source port (src_port), destination port (dst_port), number of packets (packet_count) and session duration (flow_duration), etc. The present invention is not limited to this.

[0123] The target behavior features extracted from the user behavior data set include: user unique identifier (user_id); identifier of the user operation device (device_id); operation type (action), such as login, upload, execute, delete, etc.; user operation time; operation object (resource) and operation result (result), etc., but the present invention is not limited to this.

[0124] The target behavior features extracted from the terminal security event data set include: the device_id field represents the terminal device ID; the timestamp field represents the time when the security event occurred; the security event type (event_type), such as malware_detected and privilege_escalation; the suspicious process name (process_name) and the hash value (file_hash) involving the malicious file, etc., but the present invention is not limited to this.

[0125] like Figure 4 As shown, in one embodiment, the step of establishing a normal baseline model includes steps 401 and 402 .

[0126] Step 401: extract features from historical network traffic monitoring data and historical user behavior logs to obtain behavior feature data.

[0127] Specifically, historical network traffic monitoring data and historical user behavior logs are extracted from stored historical network security logs. Feature extraction is then performed on each of these data to obtain behavioral feature data. The behavioral feature data includes: timestamp, source IP address, destination IP address, number of bytes, protocol type, source port, destination port, number of packets, session duration, user unique identifier, identifier of the user's operating device, operation type, user operation time, operation object, and operation result, etc., although the present invention is not limited thereto.

[0128] Furthermore, the behavioral feature data is processed according to the data type. Behavior features with categorical or time data types are encoded (such as hash coding, integer mapping, or one-hot coding). Behavior features with numerical data types are standardized or normalized.

[0129] The behavioral feature data after data processing is converted into a set of historical feature vectors X, that is, X=[x1,x2,…,x n ]. Among them, x1,x2,…,x n There are n feature dimensions, each of which represents a behavioral feature.

[0130] For example, a piece of historical security behavior data can be:

[0131]

[0132] The extracted behavioral feature data includes: timestamp (timestamp), source IP address (src_ip), destination IP address (dst_ip), number of bytes (bytes), user unique identifier (user_id), operation type (action), identifier of the user operation device (device_id) and security event type (event_type).

[0133] The source and destination IP addresses are hashed or integer-mapped, mapping "192.168.0.15" to 215 and "10.0.0.8" to 97. The timestamp is feature-extracted to extract the hour, converting "2024-04-22T10:30:45Z" to the data value 10. The number of bytes is normalized, dividing 1345 by the maximum number of bytes to obtain 0.1345. The operation type and security event type are one-hot-encoded, encoding "login" to [1,0,0] and "malware_detected" to [1,0,0,0].

[0134] The processed behavioral feature data is converted into a set of historical feature vectors X, that is, X = [215, 97, 0.1345, 10, 1, 0, 0, 1, 0, 0, 0]. Among them, x1 represents the hash code value of the source IP address, x2 represents the hash code value of the target IP address, x3 represents the number of bytes after normalization, x4 represents the hour when the operation occurred, x5 to x7 represent the one-hot code value of the user behavior type, x8 to x9 represent the one-hot code value of the user behavior type, and x10 to x11 represent the one-hot code value of the user behavior type. 11 One-hot encoding value indicating the terminal event type.

[0135] Step 402: Determine a boundary threshold range of an abnormal value based on the behavior characteristic data.

[0136] Specifically, the quartile range method (IQR) is used to calculate each feature dimension x in the historical feature vector X. i The boundary threshold range of outliers.

[0137] like Figure 5 As shown, in one embodiment, step 402 includes: steps 501 to 503.

[0138] Step 501: Determine the first quartile and the third quartile according to the behavior characteristic data.

[0139] Specifically, each feature dimension x in the historical feature vector X is calculated according to formula (1) and formula (2). i The first quartile Q1 and the third quartile Q3.

[0140] Q1=P 25 (x i ) (1)

[0141] Among them, Q1 is the first quartile; x i is the feature dimension, i takes 1, 2, ..., n; P 25 Represents the 25th percentile value in a set of values.

[0142] Q3=P 75 (x i ) (2)

[0143] Among them, Q3 is the third quartile; x i is the feature dimension, i takes 1, 2, ..., n; P 25 Represents the 75th percentile value in a set of values.

[0144] Step 502: Determine the interquartile range based on the first quartile and the third quartile.

[0145] Specifically, the interquartile range (IQR) between the first quartile Q1 and the third quartile Q3 is calculated according to formula (3).

[0146] IQR=Q3-Q1 (3)

[0147] Among them, IQR is the interquartile range, Q1 is the first quartile, and Q3 is the third quartile.

[0148] Step 503: Determine the upper boundary threshold and the lower boundary threshold of the outlier according to the first quartile, the third quartile and the interquartile range.

[0149] Specifically, the upper bound threshold value and the lower bound threshold value of the outlier are calculated according to formula (4) and formula (5), respectively.

[0150] Upper Bound=Q3-1.5×IQR (4)

[0151] Among them, Upper Bound is the upper boundary threshold of the outlier, Q3 is the third quartile, and IQR is the interquartile range.

[0152] Lower Bound=Q1-1.5×IQR (5)

[0153] Among them, Lower Bound is the lower boundary threshold of the outlier, Q1 is the first quartile, and IQR is the interquartile range.

[0154] Each feature dimension x i The upper boundary threshold and the lower boundary threshold are stored in the back-end database to obtain the established normal baseline model.

[0155] For example, take the normalized number of bytes x3 as an example:

[0156] Assume that the normalized number of bytes x3 has the following values in the historical network traffic monitoring data:

[0157] x3={0.123,0.127,0.130,0.135,0.134,0.140,0.145,0.150}

[0158] According to the above formulas (1) and (2), Q1 = 0.127 and Q3 = 0.145. According to the above formula (3), IQR = 0.145-0.127 = 0.018. According to the above formulas (4) and (5), Lower Bound = 0.127-1.5×0.018 = 0.100 and Upper Bound = 0.145+1.5×0.018 = 0.172.

[0159] For each feature dimension x in the historical feature vector X i Establish a pair of boundary thresholds [Lower i ,Upper i ], the upper boundary threshold and the lower boundary threshold constitute the feature dimension x i The boundary threshold range is the normal behavior range.

[0160] All feature dimensions x i By integrating the boundary threshold range of , we can form a normal baseline model, which is expressed as follows:

[0161]

[0162] Among them, x i is the feature dimension, i takes 1, 2,…, n; Lower is the lower boundary threshold of the feature dimension, Upper is the upper boundary threshold of the feature dimension, and Baseline is the normal baseline model.

[0163] Step 103: Compare the target behavior feature with the boundary threshold range of the pre-trained normal baseline model.

[0164] In one embodiment, step 103 specifically includes: if the target behavior feature exceeds a boundary threshold range, determining the safe behavior data as abnormal behavior data.

[0165] Specifically, if the target behavior feature x new Less than the lower threshold Or greater than the upper threshold Right now or The target behavior feature x new The corresponding safe behavior data is determined to be abnormal behavior data.

[0166] In one embodiment, the method for automatically generating threat intelligence further includes: determining updated model parameters based on current model parameters, learning rate, real observation values corresponding to safe behavior data, and abnormal behavior data to update the normal baseline model.

[0167] Specifically, the model parameters of the normal baseline model are fine-tuned using an incremental learning algorithm (such as the stochastic gradient descent regression algorithm or the KNN algorithm). The parameter fine-tuning formula is as follows:

[0168]

[0169] Among them, θ new is the updated model parameter, θ old is the current model parameter, η is the learning rate, y is the real observation value in the safety behavior data, Abnormal behavior data predicted by the normal baseline model.

[0170] According to the above formula (7), the updated model parameter θ is calculated new , thereby automatically updating the normal baseline model.

[0171] In one embodiment, the boundary threshold range of the abnormal value can be periodically re-determined based on the safety behavior data to update the normal baseline model.

[0172] Specifically, the upper bound threshold and lower bound threshold of the outlier are recalculated periodically according to steps 401 and 402 to update the normal baseline model. The updated upper bound threshold, lower bound threshold, and normal baseline model are synchronously recorded in the backend database.

[0173] In an embodiment of the present invention, by performing feature extraction on network traffic monitoring data and user behavior logs, target behavior features that can reflect the user behavior status are extracted. A normal baseline model is constructed based on the IQR statistical method, and the boundary threshold range of the feature dimension can be calculated according to the first quartile and the third quartile of the behavior feature data, thereby avoiding the subjectivity and lag caused by manually setting thresholds. By comparing the target behavior features of the security behavior data with the boundary threshold range obtained by pre-training, it is possible to quickly identify whether the user behavior deviates from the normal baseline, thereby achieving efficient identification and early warning of potential abnormal behaviors. In addition, the present invention introduces a learning rate adjustment mechanism, which dynamically updates the model parameters of the normal baseline model based on the incremental learning strategy, so that the model has adaptive adjustment capabilities. The above-mentioned update mechanism does not require the full reconstruction of the model, reduces the consumption of computing resources, and effectively ensures the stability and real-time performance of the normal baseline model in a dynamic network environment.

[0174] In one embodiment, if Figure 6 As shown, the training steps of the anomaly detection model include: steps 601 to 604.

[0175] Step 601: extract historical behavior features from historical security behavior data and construct a historical feature matrix. The historical security behavior data includes: historical network traffic monitoring data, historical user behavior logs, and historical terminal security events.

[0176] Specifically, the historical behavior features extracted from the historical network traffic monitoring data include: timestamp (timestamp), source IP address (src_ip), destination IP address (dst_ip), number of bytes (bytes), protocol type (protocol), source port (src_port), destination port (dst_port), number of packets (packet_count) and session duration (flow_duration), etc., but the present invention is not limited to this.

[0177] The historical behavior features extracted from the historical user behavior log include: user unique identifier (user_id), identifier of the user operation device (device_id), operation type (action), timestamp field indicating user operation time, operation object (resource) and operation result (result), etc., but the present invention is not limited to this.

[0178] The historical behavior features extracted from historical terminal security events include: the device_id field indicates the terminal device ID; the timestamp field indicates the time when the security event occurred; the security event type (event_type), such as malware_detected and privilege_escalation; the path of the suspicious file and the hash value of the suspicious file, etc., but the present invention is not limited to this.

[0179] Furthermore, the proposed historical behavior features are integrated into a historical behavior feature vector y of a set length i , each piece of historical safety behavior data corresponds to a historical behavior feature vector y i . All historical behavior feature vectors y i Splice and construct a two-dimensional historical feature matrix Y. Among them, Y∈R n×m , n is the number of historical behavior feature vectors, and m is the number of feature dimensions of the historical behavior feature vectors.

[0180] Step 602: Construct multiple random isolation trees based on the historical feature matrix.

[0181] In one embodiment, Figure 7 As shown, step 602 includes steps 701 to 704.

[0182] Step 701: Randomly select a segmentation feature f in the historical feature matrix Y j .

[0183] Specifically, from all historical behavior feature vectors y in the historical feature matrix Y i In the example, a segmentation feature f is selected by random sampling with equal probability. j , as the basis variable for this round of segmentation. The segmentation feature f j It should have numerical or sortable properties to support subsequent interval partitioning operations.

[0184] Step 702: Based on the segmentation feature f j A split point c is randomly selected from the corresponding value range.

[0185] Specifically, first calculate the segmentation feature f j The minimum value Y in the current sample set (ie, historical feature matrix Y) min (f j ) and the maximum value Y max (f j ), and in the closed interval [Y min (f j ),Y max (f j )] randomly generates a split point c, namely:

[0186] c = random(Y min (f j ), Y max (f i )) (8)

[0187] Wherein, Y min (f j ) is the minimum value of the splitting feature f k ; Y max (f j ) is the maximum value of the splitting feature f j ; random is a random function; c is a splitting point, and this splitting point c is used to divide the current sample set into two subsets.

[0188] Step 703: Divide the historical feature matrix Y according to the splitting point c, and construct the left leaf node and the right leaf node of the random isolation tree.

[0189] Specifically, divide the current sample set according to the splitting condition Y j < c and Y j ≥ c. Wherein, Y j represents the value of each sample (i.e., the historical behavior feature vector y i ) on the splitting feature f j . Samples that meet the splitting condition Y j < c are divided into the left leaf node. Samples that meet the splitting condition Y j ≥ c are divided into the right leaf node. Through the above division operation, two branches of the current random isolation tree are formed, serving as the basis for subsequent recursive construction of the random isolation tree.

[0190] Step 704: Iteratively execute the above steps until the preset termination condition is met, and then stop constructing the random isolation tree.

[0191] Specifically, repeat the above steps 701 to 703 for the samples in the left leaf node and the right leaf node respectively until one of the following termination conditions is met, and then stop expanding the current sub-random isolation tree. Among them, the termination conditions include: the number of samples in the current leaf node is 1, the construction depth of the random isolation tree reaches the preset maximum depth, or the values of all samples in the current leaf node on any feature are exactly the same. The present invention is not limited thereto.

[0192] After stopping constructing the random isolation tree, record the depth information of this leaf node as the path length l t (y i ) of the sample in this random isolation tree.

[0193] Step 603: Determine the average path length of each historical behavior feature in the historical feature matrix Y in multiple random isolation trees.

[0194] Specifically, for each sample in the historical feature matrix Y, the average path length L(y) of each sample in multiple random isolation trees is calculated according to formula (9). i ), the average path length L(y i ) is inversely proportional to the degree of isolation of the sample, that is, the average path length L(y i ) is shorter, the higher the possibility that the sample is an abnormal sample.

[0195]

[0196] Where T is the number of random isolation trees, l t (y i ) is the path length of the sample in the tth tree.

[0197] Step 604: Convert the average path length into a historical anomaly score s(y i ).

[0198] Specifically, according to the average path length L(y i ) and formula (10) to calculate the historical anomaly score s(y i ), so that the historical anomaly score s(y i ) ranges from [0,1].

[0199]

[0200] Among them, L(y i ) is the average path length of the sample in the random isolation tree, and c(n) is the expected path length, which can be calculated based on the number of samples (i.e., the number of historical behavior feature vectors) n.

[0201] The expected path length c(n) is calculated according to formula (11).

[0202]

[0203] Where n is the number of samples, that is, the number of historical behavior feature vectors; γ is the Euler-Mascheroni constant, and its value is γ≈0.5772156649.

[0204] After obtaining the trained anomaly detection model, the abnormal sample threshold Threshold is calculated based on the historical anomaly score corresponding to the historical feature matrix Y. The formula is as follows:

[0205] Threshold=P 95 (s(Y)) (12)

[0206] Among them, Threshold is the abnormal sample threshold, P 95 represents the 25th percentile value in the value set, and s(Y) represents the value of each sample in the historical feature matrix Y (i.e., the historical behavior feature vector y i ) is the normalized anomaly score calculated. i ∈Y.

[0207] Step 104: Input the target behavior features into a pre-trained anomaly detection model to obtain a current anomaly score. The anomaly detection model is constructed by training a random isolation tree based on a historical feature matrix obtained based on historical behavior features.

[0208] Specifically, the extracted multiple target behavior features x new Continuously input into the pre-trained anomaly detection model. Calculate each target behavior feature x new The current anomaly score S(x new ),Right now Then each target behavior feature x new The current anomaly score S(x new ) is compared with the abnormal sample threshold Threshold.

[0209] Step 105: If the current anomaly score is greater than the preset abnormal sample threshold, the safety behavior data is determined to be abnormal sample data.

[0210] Specifically, if a target behavior feature x is input into the anomaly detection model new Target behavior characteristics x new is greater than the abnormal sample threshold Threshold, then the target behavior feature x new The corresponding security behavior data is determined to be abnormal sample data.

[0211] Step 106: Process the abnormal behavior data and abnormal sample data to obtain threat intelligence. The first abnormal data is the abnormal sample data, i.e., the abnormal data identified by the anomaly detection model; the second abnormal data is the abnormal behavior data, i.e., the abnormal data identified by the normal baseline model.

[0212] like Figure 8 As shown, in one embodiment, step 106 includes steps 801 to 803 .

[0213] Step 801: Perform field extraction on abnormal behavior data and abnormal sample data to obtain threat information fields.

[0214] Specifically, each abnormal event e in the abnormal behavior data and abnormal sample data iEach includes a threat information field in the following format: wherein the threat information field includes: timestamp, source IP address, target IP address, hash value of the suspicious file, path of the suspicious file, and anomaly score, etc., but the present invention is not limited thereto.

[0215] Abnormal events i :{

[0216] timestamp i ;

[0217] Source IP address src_ip i ;

[0218] Destination IP address dst_ip i ;

[0219] The hash value of the suspicious file as i ;

[0220] Path pat of the suspicious file i ;

[0221] Anomaly score i ;

[0222] }

[0223] Step 802: Determine the threat information field based on preset rules to generate a threat indicator.

[0224] Specifically, the rules for generating Indicators of Compromise (IoC) include:

[0225] Rule 1: If an abnormal event e i Contains the source IP address src_ip i and / or the destination IP address dst_ip i , set the source IP address src_ip i and / or the destination IP address dst_ip i As a threat indicator IoC, and add it to the threat indicator collection. Among them, the source IP address src_ip i and / or the destination IP address dst_ip i Used to monitor whether the same source IP address and / or destination IP address appears repeatedly in other abnormal events.

[0226] Rule 2: If an abnormal event e i Involving a suspicious file and the suspicious file has a hash value of the suspicious file as i (such as MD5 or SHA256, etc.), the hash value of the suspicious file as iAs a threat indicator IoC, and add it to the threat indicator collection. Among them, the hash value of the suspicious file is i Used to quickly identify whether the same malicious program samples exist in other abnormal events.

[0227] Rule 3: If an abnormal event e i The path of the suspicious file in pat i If an exception occurs (such as an unknown file path, a file path in an atypical directory, or a malicious program sample name in the file path), the path of the suspicious file is pat i As a threat indicator IoC, and add it to the threat indicator collection.

[0228] The threat indicator set IoCs generated according to the above rules is expressed as:

[0229] IoCs = {(src_ip i ,dst_ip i ,as i ,pat i )|e i}

[0230] For each abnormal event e in abnormal behavior data and abnormal sample data i , the threat intelligence automatic generation system generates rules according to the preset threat indicators, automatically i The threat information fields are extracted and a structured threat indicator set IoCs is formed.

[0231] Step 803: Perform intelligence integration operations on threat indicators and security behavior data to obtain threat intelligence.

[0232] Specifically, in the threat intelligence database (such as VirusTotal or AlienVault OTX), for each abnormal event e i Query each threat indicator IoC in the corresponding threat indicator set IoCs.

[0233] If a threat indicator (IoC) matches an attack event recorded in the threat intelligence database, a threat label is assigned to the IoC, such as the ransomware family name or the attacker organization name.

[0234] Each threat indicator (IoC) is integrated with its query results in the threat intelligence database to generate structured threat intelligence. This threat intelligence includes: threat indicator (IoC), hit threat label, threat intelligence database type and confidence level, historical attack time window, associated abnormal event ID, attacker behavior pattern, attack strategy and characteristics, and related malware sample information.

[0235] The threat intelligence structure is as follows:

[0236] {

[0237] Indicators of Compromise (IoCs);

[0238] Hit threat tag;

[0239] Threat intelligence database types and confidence levels;

[0240] Historical attack time window;

[0241] Associated abnormal event ID;

[0242] The attacker's behavioral patterns;

[0243] …

[0244] }

[0245] In one embodiment, after obtaining threat intelligence, the threat intelligence automatic generation system immediately triggers the alarm mechanism, automatically sends abnormal warning information to the security team, and uses structured report generation tools (such as Jupyter Notebooks) to visualize the threat intelligence, so that the security team can formulate corresponding security response strategies based on the abnormal warning information, thereby improving the real-time and accuracy of the security response.

[0246] Obtain the security team's response information for each abnormal warning information and establish a feedback mechanism. The response information includes: alert ID alert_id, generation time timestamp, threat indicator IoC i , the security team's response feedback (such as confirmation, false positives and missed negatives, etc.) and notes (such as processing suggestions), etc., the present invention is not limited to this.

[0247] Specifically, the response information is collected through an online feedback form, or through an existing security information and event management system (SIEM), although the present invention is not limited thereto. The acquired response information is structured to construct a response dataset F. Each record in response dataset F contains feedback content related to a particular abnormal warning information.

[0248] Based on the security team's actual response to abnormal events, the data in the response dataset F is labeled and classified. It is mainly divided into the following three types:

[0249] True Positive (TP) refers to the real abnormal data accurately identified by the model.

[0250] False Positive (FP) means that the model mistakenly classifies normal security behavior data as abnormal data.

[0251] False negatives (FN) indicate that the model fails to detect abnormal data that actually exists in the security behavior data.

[0252] Based on the response dataset F and its corresponding labels, determine the model's precision, recall, and F1 score.

[0253] Specifically, the accuracy of abnormal warning is calculated according to formula (13).

[0254]

[0255] Among them, TP is the number of true positive cases, FP is the number of false positive cases, and FN is the number of false negative cases.

[0256] Optimize the model based on its accuracy, recall, and F1 score. For example, re-extract features from security behavior data to add new features or remove unnecessary features, thereby avoiding false positives and false negatives caused by insufficient feature selection. Model tuning can also be performed by adjusting the algorithm based on the optimized feature set and model parameters, retraining the normal baseline model and anomaly detection model, and so on.

[0257] In an embodiment of the present invention, a multi-dimensional historical feature matrix is constructed using historical network traffic monitoring data, user behavior logs, and terminal security event information. This enables the anomaly detection model to comprehensively characterize the behavioral characteristics of users and devices, improving its modeling capabilities and adaptability for complex patterns. The anomaly detection model trained based on the random isolation forest algorithm, combined with a normal behavior baseline model constructed using the quartile method, can rapidly identify multiple types of abnormal behaviors, improving the accuracy and robustness of anomaly detection. Furthermore, field extraction and structured processing are performed on the anomaly data identified by the model, and threat indicators are automatically generated according to preset rules. By matching the generated threat indicators with a threat intelligence database, corresponding threat labels, attacker information, and historical attack records are automatically obtained, thereby generating structured threat intelligence, providing high-quality, traceable threat intelligence support for security operations teams. The present invention not only implements a closed-loop processing flow of anomaly detection, threat indicator generation, and threat intelligence fusion, but also enhances the model's ability to identify actual network security threats, its adaptability, and its interpretability.

[0258] The present invention also provides an automatic threat intelligence generation device, as described in the following embodiments. Since the principle of the device to solve the problem is similar to that of the automatic threat intelligence generation method, the implementation of the device can refer to the implementation of the automatic threat intelligence generation method, and the repeated parts will not be repeated.

[0259] like Figure 9 As shown, the threat intelligence automatic generation device 900 includes: a data acquisition module 901, a feature extraction module 902, a threshold comparison module 903, an anomaly detection module 904, a data determination module 905 and a threat intelligence generation module 906.

[0260] The data acquisition module 901 is used to acquire security behavior data in real time, including network traffic monitoring data, user behavior logs, and terminal security events.

[0261] The feature extraction module 902 is used to extract features from the safety behavior data to obtain target behavior features.

[0262] The threshold comparison module 903 is used to compare the target behavior feature with the boundary threshold range of a pre-trained normal baseline model, wherein the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction.

[0263] The anomaly detection module 904 is used to input the target behavior features into a pre-trained anomaly detection model to obtain a current anomaly score; wherein the anomaly detection model is constructed based on a historical feature matrix obtained from historical behavior features and is trained based on the random isolation tree.

[0264] The data determination module 905 is configured to determine the safety behavior data as first abnormal data if the current abnormal score is greater than a preset abnormal sample threshold.

[0265] The threat intelligence generation module 906 is used to process the first abnormal data to obtain threat intelligence.

[0266] In one embodiment, the threshold comparison module is specifically configured to determine the safe behavior data as second abnormal data if the target behavior feature exceeds the boundary threshold range.

[0267] In one embodiment, the threat intelligence generation module is specifically configured to perform data processing on the first abnormal data and the second abnormal data to obtain threat intelligence.

[0268] In one embodiment, the data acquisition module includes a theme creation unit and a parameter configuration unit.

[0269] The topic creation unit is used to create corresponding topics for the network traffic monitoring data, the user behavior log and the terminal security event in the distributed data collection system.

[0270] The parameter configuration unit is used to configure data producers and data consumers in a distributed data collection system so that the data consumers receive the security behavior data corresponding to the subject in real time.

[0271] In one embodiment, the normal baseline model building unit includes: a historical feature extraction subunit and a first threshold determination subunit.

[0272] The historical feature extraction subunit is used to extract features from the historical network traffic monitoring data and the historical user behavior logs to obtain behavior feature data.

[0273] The first threshold determination subunit is used to determine a boundary threshold range of an abnormal value according to the behavior characteristic data.

[0274] In one embodiment, the first threshold determination subunit includes: a quartile determination subunit, an interquartile range determination subunit, and a second threshold determination subunit.

[0275] The quartile determination subunit is used to determine the first quartile and the third quartile according to the behavior characteristic data.

[0276] The interquartile range determination subunit is configured to determine the interquartile range according to the first quartile and the third quartile.

[0277] The second threshold determination subunit is used to determine an upper boundary threshold and a lower boundary threshold of an outlier according to the first quartile, the third quartile and the interquartile range.

[0278] In one embodiment, the threat intelligence automatic generation device 900 also includes a model update module 907, which is specifically used to determine the updated model parameters based on the current model parameters, learning rate, the real observation values corresponding to the security behavior data and the first abnormal data to update the normal baseline model.

[0279] In one embodiment, the anomaly detection model training unit includes: a feature matrix construction subunit, a random isolation tree construction subunit, a path length determination subunit, and a historical anomaly score determination subunit.

[0280] The feature matrix construction subunit is used to extract historical behavior features from historical security behavior data and construct a historical feature matrix.

[0281] The random isolation tree construction subunit is used to construct multiple random isolation trees according to the historical feature matrix.

[0282] The path length determination subunit is used to determine the average path length of each historical behavior feature in the historical feature matrix in the plurality of random isolation trees.

[0283] The historical anomaly score determination subunit is configured to convert the average path length into a historical anomaly score.

[0284] In one embodiment, the random isolation tree construction subunit includes: a segmentation feature selection subunit, a segmentation point selection subunit, a child node division subunit and an iterative training subunit.

[0285] The segmentation feature selection subunit is used to randomly select a segmentation feature in the historical feature matrix.

[0286] The segmentation point selection subunit is used to randomly select a segmentation point from a value range corresponding to the segmentation feature.

[0287] The sub-node division sub-unit is used to divide the historical feature matrix according to the cutting point to construct the left leaf node and the right leaf node of the random isolation tree.

[0288] The iterative training subunit is used to iteratively execute the above steps until a preset termination condition is met, and then stop constructing the random isolation tree.

[0289] In one embodiment, the threat intelligence generation module includes: a field extraction unit, a threat indicator generation unit, and a threat intelligence generation unit.

[0290] The field extraction unit is configured to perform field extraction on the first abnormal data and the second abnormal data to obtain a threat information field.

[0291] The threat indicator generating unit is configured to determine the threat information field based on preset rules to generate a threat indicator.

[0292] The threat intelligence generating unit is used to perform an intelligence integration operation on the threat indicators and the security behavior data to obtain threat intelligence.

[0293] Figure 10 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 10 As shown, the electronic device 1000 includes a processor 1001 , a memory 1002 and a bus 1003 .

[0294] The processor 1001 and the memory 1002 communicate with each other via the bus 1003 .

[0295] The processor 1001 is used to call the program instructions in the memory 1002 to execute the methods provided by the above method embodiments.

[0296] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned method for automatically generating threat intelligence when executed by a processor.

[0297] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the above-mentioned method for automatically generating threat intelligence.

[0298] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0299] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0300] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0301] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1A step that specifies a function in one or more boxes.

[0302] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatically generating threat intelligence, characterized in that: include: Obtain security behavior data in real time; The security behavior data includes: network traffic monitoring data, user behavior logs and terminal security events; Extracting features from the security behavior data to obtain target behavior features; Comparing the target behavior feature with a boundary threshold range of a pre-trained normal baseline model; wherein the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction; Inputting the target behavior features into a pre-trained anomaly detection model to obtain a current anomaly score; wherein the anomaly detection model is constructed by constructing a random isolation tree based on a historical feature matrix obtained based on the historical behavior features, and is trained based on the random isolation tree; If the current anomaly score is greater than a preset abnormal sample threshold, the safety behavior data is determined to be first abnormal data; The first abnormal data is processed to obtain threat intelligence.

2. The method according to claim 1, characterized in that Comparing the target behavior feature with a boundary threshold range of a pre-trained normal baseline model includes: If the target behavior feature exceeds the boundary threshold range, the safe behavior data is determined to be second abnormal data.

3. The method according to claim 2, characterized in that The processing of the first abnormal data to obtain threat intelligence includes: Data processing is performed on the first abnormal data and the second abnormal data to obtain threat intelligence.

4. The method according to claim 1, wherein The real-time acquisition of safety behavior data includes: Creating corresponding topics for the network traffic monitoring data, the user behavior log, and the terminal security event in a distributed data collection system; Data producers and data consumers are configured in a distributed data collection system so that the data consumers receive the security behavior data corresponding to the topic in real time.

5. The method according to claim 1, wherein The steps of establishing the normal baseline model include: Perform feature extraction on historical network traffic monitoring data and historical user behavior logs to obtain behavioral feature data; Determine a boundary threshold range of abnormal values based on the behavior characteristic data.

6. The method according to claim 5, characterized in that Determining a boundary threshold range of an abnormal value based on the behavior characteristic data includes: determining a first quartile and a third quartile based on the behavioral characteristic data; determining an interquartile range based on the first quartile and the third quartile; An upper boundary threshold and a lower boundary threshold of an outlier are determined according to the first quartile, the third quartile, and the interquartile range.

7. The method according to claim 6, characterized in that Also includes: Determine updated model parameters based on current model parameters, a learning rate, true observation values corresponding to the safety behavior data, and the first abnormal data to update the normal baseline model.

8. The method according to claim 1, characterized in that The training steps of the anomaly detection model include: Extract historical behavior features from historical security behavior data and construct a historical feature matrix; constructing a plurality of random isolation trees according to the historical feature matrix; Determining an average path length of each historical behavior feature in the historical feature matrix in a plurality of the random isolation trees; The average path length is converted into a historical anomaly score.

9. The method according to claim 8, characterized in that The step of constructing a plurality of random isolation trees according to the historical feature matrix includes: Randomly select a segmentation feature in the historical feature matrix; Randomly select a segmentation point from the value range corresponding to the segmentation feature; Dividing the historical feature matrix according to the split point to construct a left leaf node and a right leaf node of a random isolation tree; The above steps are iteratively performed until a preset termination condition is met, and then the random isolation tree construction is stopped.

10. The method according to claim 3, characterized in that The processing of the first abnormal data and the second abnormal data to obtain threat intelligence includes: Performing field extraction on the first abnormal data and the second abnormal data to obtain a threat information field; Determine the threat information field based on preset rules to generate a threat indicator; The threat indicators are integrated with the security behavior data to obtain threat intelligence.

11. A device for automatically generating threat intelligence, characterized in that: include: Data acquisition module, used to obtain security behavior data in real time; The security behavior data includes: network traffic monitoring data, user behavior logs and terminal security events; A feature extraction module is used to extract features from the security behavior data to obtain target behavior features; A threshold comparison module, configured to compare the target behavior feature with a boundary threshold range of a pre-trained normal baseline model; wherein the normal baseline model is obtained based on the boundary threshold range of the behavior feature data obtained by feature extraction; An anomaly detection module, configured to input the target behavior features into a pre-trained anomaly detection model to obtain a current anomaly score; wherein the anomaly detection model is constructed by constructing a random isolation tree based on a historical feature matrix obtained based on the historical behavior features, and is trained based on the random isolation tree; a data determination module, configured to determine the safety behavior data as first abnormal data if the current abnormal score is greater than a preset abnormal sample threshold; The threat intelligence generation module is used to process the first abnormal data to obtain threat intelligence.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images