A method for constructing an internal / dedicated network security baseline based on normal traffic

Abstract

The present invention discloses a method for constructing an internal / private network security baseline based on normal traffic, which relates to the field of traffic anomaly detection. The present invention first constructs a normal traffic pattern tree based on the normal network traffic of a certain time span, and then models the normal behavior of the internal / private network in combination with the normal traffic timing characteristics of the corresponding ports and protocols within a preset time window, and finally forms a security baseline for the target internal / private network, thereby reducing the underreporting rate of unknown threats in the internal / private network and meeting the high security and control requirements of such networks.

Description

Technical Field

[0001] The present invention relates to the field of traffic anomaly detection, and in particular to a method for constructing an internal / dedicated network security baseline based on normal traffic. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] For internal / private networks with high control, strong isolation, and high security levels, anomaly detection is a necessary technical step and deployment step. However, the highly concealed and technically advanced nature of network threats makes network behavior modeling more difficult. Furthermore, most existing solutions based on anomaly signatures often have limitations when dealing with unknown threats due to the inability to obtain these signatures. Therefore, there is an urgent need to define and model normal network behavior based on the majority of normal data in traffic, addressing the underreporting problem of anomaly detection in internal / private networks. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing work. This invention mainly addresses the problem of high underreporting rate of unknown threats in internal / private networks and proposes a method for establishing an internal / private network security baseline based on normal traffic. Specifically, it solves the following problems:

[0005] (1) How to build a customized security baseline based on the particularity of internal / private networks to solve the problem of high missed detection rate of unknown threats in internal / private networks.

[0006] (2) How to design a generation mechanism for pattern tree child nodes to solve the problem of balancing the missed alarm rate and the false alarm rate.

[0007] (3) How to fully exploit the multi-dimensional characteristics of normal network traffic to complete the modeling of normal behavior of internal / private networks.

[0008] Specifically, in response to the above-mentioned problems and needs, the present invention provides a method for constructing an internal / private network security baseline based on normal traffic, aiming to meet the actual security needs of the internal / private network, make full use of the particularity of the internal / private network, fully mine the effective knowledge of normal traffic data that accounts for the vast majority of the traffic data, customize the normal traffic security baseline of the target network, and solve the problem of a high missed reporting rate of unknown threats faced by the internal / private network while ensuring a high level of recall rate.

[0009] The technical solutions of the present invention are as follows:

[0010] A method for building an internal / dedicated network security baseline based on normal traffic, including:

[0011] Step S1: constructing a traffic pattern tree based on the collected normal traffic data;

[0012] Step S2: Deploy the constructed traffic pattern tree to key nodes. For each network flow, match the leaf nodes of the traffic pattern tree. If a new leaf node needs to be created, it is abnormal, otherwise it is normal.

[0013] Furthermore, the step S1 includes:

[0014] Step S11: pre-processing the collected normal traffic data;

[0015] Step S12: classifying the pre-processed traffic data;

[0016] Step S13: Remove redundant features from the classified traffic data and cluster the remaining features to obtain feature clusters;

[0017] Step S14: Extract the time series features of the pre-processed traffic data and compare them with feature clusters to be spliced;

[0018] Step S15: Generate leaf nodes based on statistical features, and determine whether to generate new leaf nodes based on dynamic thresholds, so as to construct a complete traffic pattern tree.

[0019] Furthermore, the preprocessing includes:

[0020] Perform data cleaning on the collected raw data to remove default values and data with incorrect data types.

[0021] Furthermore, data cleaning includes:

[0022] Except for the destination port, protocol type, timestamp, and tag, other attributes are discretized using KBinsDiscretizer to identify and remove redundant constant features. The remaining attributes are then normalized to unify the numerical ranges of different features into a standard range.

[0023] Furthermore, the step S12 includes:

[0024] The pre-processed traffic data is classified according to the port number, and then classified according to the protocol type.

[0025] Furthermore, the redundant features include: destination port, protocol type, timestamp and label.

[0026] Furthermore, the clustering scheme adopted is: a feature grouping scheme based on a genetic algorithm, and the evaluation criterion for grouping is the sum of the mutual information of the features within the group.

[0027] Furthermore, an autoencoder is used to extract the temporal features of traffic data, and the long short-term memory (LSTM) is used as the network architecture of the encoder and decoder of the autoencoder.

[0028] Furthermore, the update strategy for leaf nodes includes:

[0029] Strategy 1: If the current stream The mean of group characteristics and The first feature cluster node under The difference between the mean values of leaf nodes is less than the threshold , and the number of leaf nodes in the current feature cluster is less than 5, then there is no need to create a new leaf node, and update the The mean and variance of the leaf nodes are sufficient;

[0030] Strategy 2: If the current stream The mean of group characteristics and The first feature cluster node under The difference between the mean values of leaf nodes is less than the threshold , and the number of leaf nodes in the current feature cluster is greater than 5, then continue to perform the t test. The mean and variance of leaf nodes are calculated with the current flow Value, if , indicating that there is a probability of more than 95% that no new leaf nodes need to be created, and the update The mean and variance of leaf nodes;

[0031] Strategy 3: If the mean of the current flow is not similar to all leaf nodes under the feature cluster node, a new leaf node is generated under the feature cluster.

[0032] Furthermore, the threshold is determined by the entropy weighted difference method. Dynamic updates.

[0033] Compared with the existing technology, the beneficial effects of the present invention are:

[0034] 1. Aiming at the particularity of internal / private networks, this paper designs a heuristic feature clustering algorithm based on the three dimensions of port, protocol type and feature, and proposes a traffic security baseline construction scheme to reduce the missed reporting rate of unknown threat detection while ensuring the recall rate.

[0035] 2. The present invention designs a generation strategy for pattern tree child nodes and a dynamic threshold setting algorithm to form a constrained pattern tree generation method to balance the missed alarm rate and false alarm rate of unknown threat detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Construct a flow chart for the traffic pattern tree;

[0037] Figure 2 Schematic diagram of traffic pattern tree structure. DETAILED DESCRIPTION

[0038] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0039] The features and performance of the present invention are further described in detail below with reference to the embodiments.

[0040] Example 1

[0041] Taking into account the fact that normal behaviors within internal / private networks are highly standardized and that applications and services within such networks are limited, this embodiment proposes a method for constructing a security baseline for internal / private networks based on normal traffic, starting from the three levels of ports, protocols, and features. This method aims to meet the actual security needs of internal / private networks, fully utilize the particularity of internal / private networks, fully tap into the effective knowledge of normal traffic data, which accounts for the vast majority of traffic data, and customize the normal traffic security baseline of the target network. This method first constructs a normal traffic pattern tree based on the normal traffic of the network over a certain time span, and then models the normal behavior of the internal / private network in combination with the normal traffic timing characteristics of the corresponding ports and protocols within a preset time window, ultimately forming a security baseline for the target internal / private network, reducing the underreporting rate of unknown threats in the internal / private network, and meeting the high security and control requirements of such networks.

[0042] Specifically, a method for establishing an internal / dedicated network security baseline based on normal traffic includes:

[0043] Step S1: constructing a traffic pattern tree based on the collected normal traffic data;

[0044] Step S2: Deploy the constructed traffic pattern tree to key nodes. For each network flow, match the leaf nodes of the traffic pattern tree. If a new leaf node needs to be created, it is abnormal, otherwise it is normal.

[0045] In this embodiment, please refer to Figure 1 , the step S1 comprises:

[0046] Step S11: pre-processing the collected normal traffic data;

[0047] Step S12: classifying the pre-processed traffic data;

[0048] Step S13: Remove redundant features from the classified traffic data and cluster the remaining features to obtain feature clusters;

[0049] Step S14: Extract the time series features of the pre-processed traffic data and compare them with The timing model is used to extract the timing features of the traffic classified by port and protocol type, and the timing features are spliced with the features of each feature cluster.

[0050] Step S15: Generate leaf nodes based on statistical features, and determine whether to generate new leaf nodes based on dynamic thresholds, so as to construct a complete traffic pattern tree.

[0051] In this embodiment, specifically, the preprocessing includes:

[0052] Clean the collected raw data to remove default values and data with incorrect data types; data cleaning includes:

[0053] Except for the destination port, protocol type, timestamp, and tag, other attributes are discretized using KBinsDiscretizer to identify and remove redundant constant features. The remaining attributes are then normalized to unify the numerical ranges of different features into a standard range.

[0054] It should be noted that since the pattern tree is an unsupervised model, the training process only learns the patterns of normal traffic. Therefore, it is necessary to collect a sufficient amount of normal traffic from the target network, and the data during the traffic collection period must cover all normal business behaviors and application service operations within the target network. First, the collected raw data is cleaned to remove default values and data with incorrect data types. Specifically, except for the destination port, protocol type, timestamp, and label, other attributes are discretized using KBinsDiscretizer to identify and remove redundant constant features. The remaining attributes are then normalized, and the numerical ranges of different features are unified into a standard range.

[0055] In this embodiment, specifically, step S12 includes:

[0056] The pre-processed traffic data is classified according to the port number, and then classified according to the protocol type, so as to obtain the first-level and second-level nodes of the traffic pattern tree.

[0057] In this embodiment, specifically, the redundant features include: destination port, protocol type, timestamp, and label.

[0058] In this embodiment, specifically, the clustering scheme adopted is: a feature grouping scheme based on a genetic algorithm, and the evaluation standard of grouping is the sum of mutual information of features within the group.

[0059] It's important to note that even after initial data cleaning, traffic information attribute features still contain a significant number of them. Generating leaf nodes based on these features will inevitably lead to an explosion in the number of leaf nodes. Therefore, it's necessary to group the cleaned attribute features, aggregating highly similar features for data dimensionality reduction. This embodiment uses a feature grouping scheme based on a genetic algorithm (GA), with the grouping evaluation criterion being the sum of the mutual information (MI) of the features within a group.

[0060] In this embodiment, it should be noted that the genetic algorithm is an existing algorithm. The following describes the principle of the genetic algorithm in combination with the application scenarios involved in this embodiment. Those skilled in the art can implement the specific operations in combination with the following description.

[0061] (1) Genetic algorithm feature grouping

[0062] The core idea of genetic algorithms is derived from Darwin's theory of evolution. It solves complex optimization problems by simulating the process of natural selection and inheritance. Each "individual" represents a feature grouping scheme. Individuals are encoded using integer encoding. Each gene corresponds to a feature, and its value represents the group to which the feature belongs. For example, for the problem of 10 features being divided into 3 groups, a possible individual code is , which means that the 1st, 3rd, 6th and 9th features belong to group 1, the 2nd, 5th and 8th features belong to group 2, and the rest belong to group 3. Specifically, it includes the following steps:

[0063] Step 1: Initialize the population randomly and introduce heuristic rules to improve the quality of the initial population. For example, using the mutual information matrix, features with high mutual information tend to be initialized into the same group. This "intelligent" initialization strategy helps the algorithm converge to a good solution faster. The design of the fitness function is the same as that of the ant colony algorithm.

[0064] Step 2: Tournament selection. Each time, randomly select from the population individuals, and then select the individual with the highest fitness The advantage of tournament selection is that it can maintain the diversity of the population while giving better individuals more opportunities to reproduce. to balance selection pressure and population diversity;

[0065] Step 3: Crossover. Taking the encoding method into consideration, this embodiment designs an improved uniform crossover operation for the genetic algorithm to generate new solutions. During the crossover process, not only are the genes of the parent individuals exchanged, but the validity of the offspring individuals after the crossover is also guaranteed (i.e., features are not repeatedly assigned to multiple groups). Specifically, a standard uniform crossover is first performed, followed by a repair step to ensure that each feature belongs to only one group. This maintains the advantages of uniform crossover while ensuring the feasibility of the solution.

[0066] Step 4: Mutation. This is used to maintain population diversity and prevent the algorithm from prematurely converging to a local optimum. This example implements two mutation methods: simple random mutation, which randomly selects a feature and changes its group; and intelligent mutation based on mutual information, which tends to move features to groups with features with higher mutual information. This intelligent mutation strategy increases the probability of generating better solutions.

[0067] Step 5: To improve algorithm performance, this example also introduces several advanced strategies. For example, an elite retention strategy is implemented to ensure that the best individuals in each generation are not lost during the evolution process. Furthermore, adaptive crossover and mutation probabilities are employed to dynamically adjust these parameters based on population diversity, balancing exploration and exploitation at different stages of the algorithm.

[0068] (2) Mutual information matrix calculation

[0069] In feature grouping algorithms, calculating the mutual information matrix is a core step, providing an important theoretical foundation for subsequent feature clustering. Mutual information, a key concept in information theory, effectively measures the degree of interdependence between two random variables. Compared to the traditional correlation coefficient, MI has the advantage of capturing nonlinear relationships between features, making it particularly important in feature selection and grouping tasks.

[0070] From a theoretical point of view, two random variables and The mutual information between can be expressed as: ,in and is the marginal entropy, is the joint entropy. However, since the data set is continuous, a histogram-based approach is used to estimate mutual information. First, the continuous variable is discretized into several groups using an equal-width partitioning strategy. Then, a two-dimensional histogram is used to estimate the joint probability distribution of the two variables. Finally, the marginal entropy and joint entropy are calculated based on the discretized data.

[0071] Due to the large number of features, the present invention adopts a parallel computing strategy to accelerate the calculation of the mutual information matrix during the implementation process. The calculation of the entire mutual information matrix is decomposed into multiple independent subtasks, and a multi-core processor is used to simultaneously calculate the mutual information between multiple feature pairs. Finally, the results of each subtask are combined into a complete mutual information matrix. In order to process large-scale feature sets, a row-based calculation method is further adopted. Only one row of the mutual information matrix is calculated at a time, that is, the mutual information between one feature and all other features. In terms of data preprocessing, all features are normalized to the same scale to ensure that the calculation of mutual information is not affected by the range of the original data.

[0072] In this embodiment, specifically, an autoencoder is used to extract the temporal features of traffic data, and a long short-term memory (LSTM) is used as the network architecture of the encoder and decoder of the autoencoder.

[0073] That is, in this embodiment, an autoencoder is used to extract time series features from traffic data. Long Short-Term Memory (LSTM) is used as the network architecture of the autoencoder's encoder and decoder. In this embodiment, a time window is set, and the traffic within the window is pre-processed and then sent to the encoder end of the autoencoder for model training. The specific steps include the following:

[0074] Step 1: The encoder represents the preprocessed input time series data as a fixed-length low-dimensional latent representation, which represents the time series features of the input data. The output of the LSTM can be the hidden state of the last time step (as the latent representation) or the hidden state of all time steps;

[0075] Step 2: The decoder reconstructs the original data from the latent representation. The RepeatVector layer repeats the latent representation multiple times to match the time steps of the original data. The LSTM unit is used to gradually reconstruct the original time series data. The fully connected layer maps the LSTM output to the original feature dimension.

[0076] Step 3: Use the mean square error loss function to calculate the difference between the reconstructed data and the original data. The loss function is expressed as: ,in, is the original data, To reconstruct the data, is the number of samples.

[0077] In this embodiment, it should be noted that, generally speaking, each port typically corresponds to a relatively fixed service. For example, web port 80 typically corresponds to a web service. However, in internal / private network environments, due to high security levels and strict control measures, the application services within the network are generally fixed. Therefore, constructing a tree based on the normal network flow of each port is equivalent to constructing a pattern tree for the service corresponding to each port.

[0078] like Figure 2 As shown, the first layer of the pattern tree is the port, the second layer is the protocol, the third layer is the feature cluster, each feature cluster contains similar features grouped by features, and the last layer is the statistical feature. In this embodiment, the mean and variance are used as examples. Each network flow will become a branch in the pattern tree, and the update strategy of the leaf node is as follows:

[0079] Strategy 1: If the current stream The mean of group characteristics and The first feature cluster node (third layer) under The difference between the mean values of leaf nodes (similarity_threshold) is less than the threshold , and the number of leaf nodes in the current feature cluster is less than 5 (min_samples), then there is no need to create a new leaf node, and update the The mean and variance of the leaf nodes are sufficient;

[0080] Strategy 2: If the current stream The mean of group characteristics and The first feature cluster node (third layer) under The difference between the mean values of leaf nodes is less than the threshold , and the number of leaf nodes in the current feature cluster is greater than 5, then continue to perform the t test. The mean and variance of leaf nodes are calculated with the current flow Value, if , indicating that there is a probability of more than 95% that no new leaf nodes need to be created, and the update The mean and variance of leaf nodes;

[0081] in, means Statistics ( -statistic), is test( -test) is a core calculation value used to measure the degree of difference between sample data and hypothesis, defined as:

[0082]

[0083] in, is the sample mean, is the population hypothesized mean (the reference value in the null hypothesis), is the sample standard deviation, is the sample size;

[0084] in, value( -value) is used in statistical hypothesis testing to measure the probability of the observed data (or more extreme data) occurring when the null hypothesis is true. , the results are usually considered statistically significant, , then the null hypothesis cannot be rejected.

[0085] Strategy 3: If the mean of the current flow is not similar to all leaf nodes under the feature cluster node (third layer), a new leaf node is generated under the feature cluster.

[0086] It is worth noting that this embodiment uses the entropy weighted value difference method to dynamically update the threshold, update the threshold of the feature cluster node, and use the distribution entropy of the value difference to calculate the threshold, which can capture the complexity and diversity of data changes. The dynamic threshold is defined as:

[0087]

[0088] in:

[0089] is the scaling factor;

[0090] is the total number of leaf nodes under the cluster node; ;

[0091] is the entropy of the weighted difference, which is used to measure the uncertainty of the weighted difference and is defined as:

[0092]

[0093] is the total number of leaf nodes.

[0094] in, It is the normalized weighted difference. Normalized weighted difference means normalizing the weighted difference and is defined as:

[0095]

[0096] in:

[0097] Is a leaf node The mean value stored for the first time;

[0098] Represents a leaf node Dynamic mean of (if similar flow feature values are merged into an existing leaf node, the mean of the leaf node will change dynamically);

[0099] Represents a leaf node The number of similar flows that have been merged;

[0100] The sum of weighted differences calculated for all leaf nodes under the cluster node.

[0101] The above-described embodiments merely represent specific implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of protection of the present application. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the technical concept of the present application, and all such variations and improvements fall within the scope of protection of the present application.

[0102] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of the description in this section that did not constitute prior art at the time of filing are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A method for constructing an internal / dedicated network security baseline based on normal traffic, characterized in that: include: Step S1: constructing a traffic pattern tree based on the collected normal traffic data; Step S2: Deploy the constructed traffic pattern tree to key nodes. For each network flow, match the leaf nodes of the traffic pattern tree. If a new leaf node needs to be created, it is abnormal; otherwise, it is normal. The step S1 includes: Step S11: pre-processing the collected normal traffic data; Step S12: classifying the pre-processed traffic data; Step S13: Remove redundant features from the classified traffic data and cluster the remaining features to obtain feature clusters; Step S14: Extract the time series features of the pre-processed traffic data and compare them with feature clusters to be spliced; Step S15: Generate leaf nodes based on statistical features, and determine whether to generate new leaf nodes based on dynamic thresholds, thereby constructing a complete traffic pattern tree; Leaf node update strategies include: Strategy 1: If the current stream The mean of group characteristics and The first feature cluster node under The difference between the mean values of leaf nodes is less than the threshold , and the number of leaf nodes in the current feature cluster is less than 5, then there is no need to create a new leaf node, and update the The mean and variance of the leaf nodes are sufficient; Strategy 2: If the current stream The mean of group characteristics and The first feature cluster node under The difference between the mean values of leaf nodes is less than the threshold , and the number of leaf nodes in the current feature cluster is greater than 5, then continue to perform the t test. The mean and variance of leaf nodes are calculated with the current flow Value, if , indicating that there is a probability of more than 95% that no new leaf nodes need to be created, and the update The mean and variance of leaf nodes; Strategy 3: If the mean of the current flow is not similar to all leaf nodes under the feature cluster node, a new leaf node is generated under the feature cluster; Thresholding by entropy weighted difference method Dynamic updates.

2. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 1, characterized in that: The preprocessing comprises: Perform data cleaning on the collected raw data to remove default values and data with incorrect data types.

3. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 2, characterized in that: Data cleaning, including: Except for the destination port, protocol type, timestamp, and tag, other attributes are discretized using KBinsDiscretizer to identify and remove redundant constant features. The remaining attributes are then normalized to unify the numerical ranges of different features into a standard range.

4. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 1, characterized in that: The step S12 includes: The pre-processed traffic data is classified according to the port number, and then classified according to the protocol type.

5. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 1, characterized in that: The redundant features include: destination port, protocol type, timestamp and label.

6. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 1, characterized in that: The clustering scheme used is: a feature grouping scheme based on genetic algorithm, and the evaluation criterion for grouping is the sum of mutual information of features within the group.

7. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 1, characterized in that: An autoencoder is used to extract the temporal features of traffic data, and long short-term memory (LSTM) is used as the encoder and decoder network architecture of the autoencoder.

8. The method for establishing an internal / dedicated network security baseline based on normal traffic according to claim 1, characterized in that: means Statistics, is The core calculated value of the test is used to measure the degree of difference between the sample data and the hypothesis, and is defined as: in, is the sample mean, is the hypothesized population mean, is the sample standard deviation, is the sample size.

Images