A method for obtaining training data sets for network threat identification models

Abstract

The present invention discloses a method for acquiring a training data set for a network threat identification model, and relates to the field of network security. The method comprises: when an intranet sends a message to an external network via an IPSec gateway, and when an external network sends a message to an intranet via an IPSec gateway, the IPSec gateway detects the traffic passing through it, and if an abnormality is found in the traffic, the sending of the traffic is interrupted; at the same time, after receiving the message, the IPSec gateway checks whether its own behavior baseline is abnormal, and whether the system state is abnormal; if any of the network traffic, behavior baseline and system state is abnormal, the plaintext and ciphertext data of the message under the abnormal condition, the behavior baseline corresponding to the message, and the system state are marked, and encrypted at the network layer and sent to a central platform to obtain a system state data set and a behavior baseline data set; the message mark obtained after information analysis is judged as abnormal, and a network message data set is obtained. The present invention can automatically, accurately, safely and comprehensively collect network threat data.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method for acquiring a training data set for a network threat identification model. Background Art

[0002] IP networks generally use IPSec (Internet Security Protocol) technology to provide network layer encryption protection for data, and use network traffic analysis technology that matches event feature information with feature libraries to detect security threats. With the rapid growth of network bandwidth and network applications, the increase in malicious network traffic, especially malicious encrypted traffic, has increased the difficulty of identifying security threats. Artificial intelligence technology has brought new ways to network traffic analysis and can further enhance the ability to identify network threats. However, the quality of artificial intelligence models depends largely on the data set used for training. In order to ensure the application effect of network threat identification models based on artificial intelligence technology in actual working network environments, it is urgent to solve the problem of effective training data sets.

[0003] Currently, there are many ways to obtain training data for network threat identification models, such as using existing public data sets, building simulated network environments to collect data, using synthetic data, and collecting data in real environments.

[0004] The main disadvantages of the solution of training intelligent recognition models based on existing public data sets are: first, the source of public data sets is mainly laboratory simulation environment, the real user traffic is limited, and it is impossible to fully simulate the actual network environment; second, the real-time performance of public data sets is not strong, and there is currently a lack of authoritative, public, high-quality, and timely new data sets.

[0005] The idea of ​​the data collection scheme based on building a simulated network environment by yourself is to obtain threat data by simulating or reproducing the threat behavior and network system environment when the threat occurs. The main disadvantages of this scheme are: first, the scale of the simulation environment is limited, and it is difficult to show the true appearance of the network information system; second, the amount of data is not sufficient and can only include specific categories of network threats.

[0006] The idea of ​​the synthetic data solution is to obtain network system threat data through construction and simulation based on a small amount of data collected in a real network environment. The main disadvantages of this solution are: first, the diversity of synthetic data is insufficient, and the data that has been synthesized multiple times will lose more and more real information, and the quality will drop sharply; second, the quality of data synthesized by different methods cannot be effectively quantified before the data is used for training.

[0007] The existing idea of ​​collecting data based on the real environment is to deploy probes at the network hub, filter the traffic obtained through mirroring or directly transmit it back to the central platform for processing. However, since the probes have very limited recognition accuracy for abnormal encrypted traffic, especially abnormal encrypted traffic at the network layer, the returned abnormal encrypted traffic data is small and of low quality. At the same time, in order to ensure the coverage of network traffic collection, a large number of new probes need to be deployed, which is costly.

[0008] In general, the data sets obtained by various existing schemes have various deficiencies, and are mainly based on plaintext traffic, lacking ciphertext traffic, especially encrypted traffic of network threats at the network layer. Summary of the invention

[0009] The data set for training the network threat intelligent identification model has the problems of incomplete network threat data, low timeliness, insufficient total data, low authenticity, lack of network layer ciphertext data, etc. The present invention provides a method for obtaining a training data set for a network threat identification model.

[0010] The present invention discloses a method for acquiring a training data set for a network threat identification model, which comprises:

[0011] When the intranet sends a message to the external network through the IPSec gateway, the IPSec gateway detects the traffic passing through it. If the traffic is found to be abnormal, it will interrupt the traffic from continuing to be sent to the external network. At the same time, after receiving the message, the IPSec gateway checks whether its own behavior baseline and system status are abnormal. If any of the network traffic, behavior baseline and system status are abnormal, the plaintext and ciphertext data of the message under abnormal conditions, the behavior baseline and system status corresponding to the message are marked, encrypted at the network layer and sent to the central platform.

[0012] When the external network sends a message to the internal network through the IPSec gateway, the IPSec gateway detects the traffic passing through it. If the traffic is found to be abnormal, the traffic will be interrupted and sent to the internal network. At the same time, after receiving the message, the IPSec gateway checks whether its own behavior baseline and system status are abnormal. If any of the network traffic, behavior baseline and system status is abnormal, the plaintext and ciphertext data of the message under abnormal conditions, the behavior baseline and system status corresponding to the message are marked, encrypted at the network layer and sent to the central platform.

[0013] The central platform decrypts and parses the received encrypted information to obtain a system status data set and a behavior baseline data set; it makes anomaly judgments on the message annotations obtained after information analysis to obtain a network message data set; the system status data set, behavior baseline data set, and network message data set constitute a training data set.

[0014] Furthermore, after receiving the message, the IPSec gateway checks whether its own behavior baseline is abnormal and checks whether the system status is abnormal, including:

[0015] After receiving a message, the IPSec gateway checks that if there is no abnormality in its own behavior baseline, it will continue to check whether there is any abnormality in its own behavior baseline after receiving subsequent messages;

[0016] After receiving the message, the IPSec gateway checks that if the system status is normal, it will continue to check whether the system status is abnormal after receiving subsequent messages.

[0017] Furthermore, when the IPSec gateway receives traffic from the intranet to the extranet, if the network traffic is normal, it sends the message to the extranet.

[0018] Furthermore, when the intranet sends a message to the external network via the IPSec gateway, if there is no abnormality in the network traffic, behavior baseline and system status, the message is encrypted according to the preset security policy and sent to the external network, or directly sent to the external network in plain text.

[0019] Furthermore, when the IPSec gateway receives traffic from the external network to the internal network, if the network traffic is normal, it sends the message to the internal network.

[0020] Furthermore, when the external network sends a message to the internal network via the IPSec gateway, if there is no abnormality in the network traffic, behavior baseline and system status, the message is decrypted according to the preset security policy and sent to the internal network, or directly sent to the internal network in plain text.

[0021] Furthermore, the marking of the plaintext and ciphertext data of the message under abnormal circumstances, the behavior baseline corresponding to the message, and the system status includes:

[0022] According to the specific circumstances of the abnormality, the plaintext of the message is marked as normal plaintext and abnormal plaintext, and the ciphertext of the message is marked as normal ciphertext and abnormal ciphertext; the behavior baseline is marked as normal behavior baseline and abnormal behavior baseline; the system status is marked as normal system status and abnormal system status.

[0023] Furthermore, when the central platform receives the data reported by the IPSec gateway, it classifies and saves the behavior baseline and system status according to the labels; and puts the plaintext and ciphertext marked as abnormal into the abnormal message data set in the network message data set according to the abnormal classification.

[0024] Furthermore, the behavior baseline and system status are classified and saved according to the annotations, including:

[0025] Put the behavior baseline marked as a normal behavior baseline into a normal behavior baseline data set, and put the behavior baseline marked as an abnormal behavior baseline into an abnormal behavior baseline data set; the normal behavior baseline data set and the abnormal behavior baseline data set constitute a behavior baseline data set;

[0026] The system state marked as a normal system state is put into a normal system state data set, and the system state marked as an abnormal system state is put into a system state data set; the normal system state data set and the abnormal system state data set constitute a system state data set.

[0027] Furthermore, when the central platform receives the data reported by the IPSec gateway, it performs message marking abnormality judgment. If the message is marked as abnormal, it is placed in the abnormal message data set; if the message is marked as normal plaintext and ciphertext, and its corresponding behavior baseline or system status is abnormal, the plaintext and ciphertext originally marked as normal are regarded as suspicious traffic, and a secondary network threat detection is performed on the suspicious traffic, and the detection result is judged as abnormal. If the judgment result is normal, the suspicious traffic is placed in the normal message data set in the network message data set, otherwise it is placed in the abnormal message data set; the normal message data set and the abnormal message data set constitute the network message data set.

[0028] Due to the adoption of the above technical scheme, the present invention has the following advantages: At present, in terms of artificial intelligence model training, it is generally carried out by adopting existing public data sets, building simulated network environments to collect data, and adopting synthetic data and other schemes to collect training data. The data obtained by the above methods have different degrees of low timeliness of network threat data, incomplete types, insufficient total amount, low authenticity, lack of encrypted traffic data and other problems. After the model trained by the above data is deployed in the actual network environment, it is necessary to perform secondary optimization on the model. The present invention makes full use of the IPSec gateway that has been widely deployed in the network to collect the clear and secret traffic of network threat messages, obtains the corresponding relationship between the clear and secret traffic, and further collects the system status and behavior baseline information of the IPSec gateway when it is affected by the network threat message, expands the data type of the training data set, and has no effect on the existing communication business. It has the advantages of high timeliness, high authenticity, rich ciphertext data, strong data reporting security, etc. Through continuous collection, it can improve the problem of insufficient total amount and incomplete type of network threat data. The present invention realizes the automatic real-time acquisition of artificial intelligence model training data, provides a high-quality data source for network threat intelligent model training, and is suitable for application in dedicated networks where IPSec gateways are widely deployed. The present invention can automatically, accurately, safely and comprehensively collect network threat data. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] Figure 1 A schematic diagram of an embodiment of the present invention in which an intranet and an extranet send messages to each other via an IPSec gateway;

[0031] Figure 2 The present invention is a flowchart of a method for obtaining a training data set for a network threat identification model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further described in conjunction with the accompanying drawings and embodiments, and the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present invention.

[0033] The present invention superimposes a traditional network traffic detection module, a behavior baseline monitoring module and a system status monitoring module on an IPSec gateway that is widely deployed at the boundary of the user's internal and external networks in a gatekeeper-like manner, uses the traffic detection module to analyze the network traffic in a real network environment, and makes a judgment based on the impact of abnormal network traffic on its own behavior baseline and system status, thereby locking abnormal plaintext / ciphertext network traffic. The plaintext traffic and ciphertext traffic of abnormal messages, as well as the behavior baseline and system status information, are securely sent to a central platform for aggregation and secondary analysis through an IPSec security channel under the condition of password protection at the network layer, thereby forming a training data set, thereby achieving the purpose of automatically, accurately, securely and comprehensively collecting highly timely network threat data.

[0034] See also Figure 1 and Figure 2 The present invention provides an embodiment of a method for obtaining a training data set for a network threat identification model, which includes:

[0035] When the intranet sends a message to the external network through the IPSec gateway, the IPSec gateway detects the traffic passing through it. If the traffic is found to be abnormal, it will interrupt the traffic from continuing to be sent to the external network. At the same time, after receiving the message, the IPSec gateway checks whether its own behavior baseline and system status are abnormal. If any of the network traffic, behavior baseline and system status are abnormal, the plaintext and ciphertext data of the message under abnormal conditions, the behavior baseline and system status corresponding to the message are marked, encrypted at the network layer and sent to the central platform.

[0036] When the external network sends a message to the internal network through the IPSec gateway, the IPSec gateway detects the traffic passing through it. If the traffic is found to be abnormal, the traffic will be interrupted and sent to the internal network. At the same time, after receiving the message, the IPSec gateway checks whether its own behavior baseline and system status are abnormal. If any of the network traffic, behavior baseline and system status is abnormal, the plaintext and ciphertext data of the message under abnormal conditions, the behavior baseline and system status corresponding to the message are marked, encrypted at the network layer and sent to the central platform.

[0037] The central platform decrypts and parses the received encrypted information to obtain a system status data set and a behavior baseline data set; it makes anomaly judgments on the message annotations obtained after information analysis to obtain a network message data set; the system status data set, behavior baseline data set, and network message data set constitute a training data set.

[0038] In one embodiment of the present application, after receiving the message, the IPSec gateway checks whether its own behavior baseline is abnormal and checks whether the system status is abnormal, including:

[0039] After receiving a message, the IPSec gateway checks that if there is no abnormality in its own behavior baseline, it will continue to check whether there is any abnormality in its own behavior baseline after receiving subsequent messages;

[0040] After receiving the message, the IPSec gateway checks that if the system status is normal, it will continue to check whether the system status is abnormal after receiving subsequent messages.

[0041] In one embodiment of the present application, when the IPSec gateway receives traffic from the intranet to the extranet, if the network traffic is normal, it sends the message to the extranet.

[0042] In one embodiment of the present application, when the intranet sends a message to the external network via the IPSec gateway, if there is no abnormality in the network traffic, behavior baseline and system status, the message is encrypted according to the preset security policy and sent to the external network, or directly sent to the external network in plain text.

[0043] In one embodiment of the present application, when the IPSec gateway receives traffic from the external network to the internal network, if the network traffic is normal, the message is sent to the internal network.

[0044] In one embodiment of the present application, when the external network sends a message to the internal network via the IPSec gateway, if there is no abnormality in the network traffic, behavior baseline and system status, the message is decrypted according to the preset security policy and sent to the internal network, or directly sent to the internal network as plain text.

[0045] In one embodiment of the present application, the marking of the plaintext and ciphertext data of the message under abnormal circumstances, the behavior baseline corresponding to the message, and the system status includes:

[0046] According to the specific circumstances of the abnormality, the plaintext of the message is marked as normal plaintext and abnormal plaintext, and the ciphertext of the message is marked as normal ciphertext and abnormal ciphertext; the behavior baseline is marked as normal behavior baseline and abnormal behavior baseline; the system status is marked as normal system status and abnormal system status.

[0047] In one embodiment of the present application, when the central platform receives data reported by the IPSec gateway, the behavior baseline and system status are classified and saved according to the labels; the plaintext and ciphertext marked as abnormal are put into the abnormal message data set in the network message data set according to the abnormal classification.

[0048] In one embodiment of the present application, the classification and storage of the behavior baseline and the system status according to the annotations includes:

[0049] Put the behavior baseline marked as a normal behavior baseline into a normal behavior baseline data set, and put the behavior baseline marked as an abnormal behavior baseline into an abnormal behavior baseline data set; the normal behavior baseline data set and the abnormal behavior baseline data set constitute a behavior baseline data set;

[0050] The system state marked as a normal system state is put into a normal system state data set, and the system state marked as an abnormal system state is put into a system state data set; the normal system state data set and the abnormal system state data set constitute a system state data set.

[0051] In one embodiment of the present application, after the central platform receives the data reported by the IPSec gateway, it performs a message marking abnormality judgment. If the message is marked as abnormal, it is placed in the abnormal message data set; if the message is marked as normal plaintext and ciphertext, and its corresponding behavior baseline or system status is abnormal, the plaintext and ciphertext originally marked as normal are regarded as suspicious traffic, and a secondary network threat detection is performed on the suspicious traffic, and the detection result is judged as abnormal. If the judgment result is normal, the suspicious traffic is placed in the normal message data set in the network message data set, otherwise it is placed in the abnormal message data set; the normal message data set and the abnormal message data set constitute the network message data set.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for obtaining a training data set for a network threat identification model, characterized in that: include: When the intranet sends a message to the extranet via the IPSec gateway, the IPSec gateway detects the traffic passing through it. If any abnormality is found in the traffic, the traffic will be interrupted from being sent to the extranet. At the same time, after receiving the message, the IPSec gateway checks whether its own behavior baseline is abnormal and whether the system status is abnormal. If there is any anomaly in the network traffic, behavior baseline, or system status, the plaintext and ciphertext data of the message under the anomaly, the behavior baseline corresponding to the message, and the system status are marked, encrypted at the network layer, and sent to the central platform; When the external network sends a message to the internal network through the IPSec gateway, the IPSec gateway detects the traffic passing through it. If any abnormality is found in the traffic, the traffic will be interrupted from continuing to be sent to the internal network. At the same time, after receiving the message, the IPSec gateway checks whether there is any abnormality in its own behavior baseline and whether there is any abnormality in the system status. If there is any anomaly in the network traffic, behavior baseline, or system status, the plaintext and ciphertext data of the message under the anomaly, the behavior baseline corresponding to the message, and the system status are marked, encrypted at the network layer, and sent to the central platform; The central platform decrypts and parses the received encrypted information to obtain the system status data set and the behavior baseline data set; it performs abnormal judgment on the message annotations obtained after information parsing to obtain the network message data set; The system status dataset, behavior baseline dataset and network message dataset constitute the training dataset.

2. The method according to claim 1, characterized in that After receiving the message, the IPSec gateway checks whether its own behavior baseline is abnormal and whether the system status is abnormal, including: After receiving a message, the IPSec gateway checks that if there is no abnormality in its own behavior baseline, it will continue to check whether there is any abnormality in its own behavior baseline after receiving subsequent messages; After receiving the message, the IPSec gateway checks that if the system status is normal, it will continue to check whether the system status is abnormal after receiving subsequent messages.

3. The method according to claim 1, characterized in that When the IPSec gateway receives traffic from the intranet to the extranet, if the network traffic is normal, it will send the message to the extranet.

4. The method according to claim 1, characterized in that When the intranet sends a message to the extranet via the IPSec gateway, if there is no abnormality in the network traffic, behavior baseline and system status, the message will be encrypted according to the preset security policy and sent to the extranet, or directly sent to the extranet in plain text.

5. The method according to claim 1, characterized in that When the IPSec gateway receives traffic from the external network to the internal network, if the network traffic is normal, it will send the message to the internal network.

6. The method according to claim 1, characterized in that When the external network sends a message to the internal network through the IPSec gateway, if there is no abnormality in the network traffic, behavior baseline and system status, the message will be decrypted according to the preset security policy and sent to the internal network, or directly sent to the internal network in plain text.

7. The method according to claim 1, characterized in that The marking of the plaintext and ciphertext data of the message under abnormal circumstances, the behavior baseline corresponding to the message, and the system status includes: According to the specific circumstances of the abnormality, the plaintext of the message is marked as normal plaintext and abnormal plaintext, and the ciphertext of the message is marked as normal ciphertext and abnormal ciphertext; the behavior baseline is marked as normal behavior baseline and abnormal behavior baseline; the system status is marked as normal system status and abnormal system status.

8. The method according to claim 1, characterized in that When the central platform receives the data reported by the IPSec gateway, it classifies and saves the behavior baseline and system status according to the labels; and puts the plaintext and ciphertext marked as abnormal into the abnormal message data set in the network message data set according to the abnormal classification.

9. The method according to claim 8, characterized in that The behavior baseline and system status are classified and saved according to the annotations, including: Put the behavior baseline marked as a normal behavior baseline into a normal behavior baseline data set, and put the behavior baseline marked as an abnormal behavior baseline into an abnormal behavior baseline data set; the normal behavior baseline data set and the abnormal behavior baseline data set constitute a behavior baseline data set; The system state marked as a normal system state is put into a normal system state data set, and the system state marked as an abnormal system state is put into a system state data set; the normal system state data set and the abnormal system state data set constitute a system state data set.

10. The method according to claim 1, characterized in that When the central platform receives the data reported by the IPSec gateway, it will judge the message marking anomaly. If the message is marked as abnormal, it will be placed in the abnormal message data set; if the message is marked as normal plaintext and ciphertext, and its corresponding behavior baseline or system status is abnormal, the plaintext and ciphertext originally marked as normal will be regarded as suspicious traffic, and the suspicious traffic will be subjected to secondary network threat detection, and the detection result will be judged as abnormal. If the judgment result is normal, the suspicious traffic will be placed in the normal message data set in the network message data set, otherwise it will be placed in the abnormal message data set; The normal message data set and the abnormal message data set constitute the network message data set.

Images