An end-to-end network application layer evasion detection method

Abstract

This invention discloses an end-to-end network application layer evasion detection method. It collects network flow data without application layer evasion techniques and network evasion data flows with various application layer evasion techniques applied, forming a sample dataset. Raw byte stream features are extracted from the network flow without evasion and the various evasion network flows. A neural network is used to classify the raw byte stream features extracted from each network flow, thereby achieving the detection and identification of various application layer evasion behaviors.

Description

Technical Field

[0001] This invention relates to the fields of network security and artificial intelligence, and specifically to an end-to-end network application layer evasion detection method. Background Technology

[0002] Network Intrusion Detection / Prevention System (NIDS / NIPS) protects other network devices by copying network data packets, reconstructing network data flows, detecting malicious behavior, and issuing alerts or interrupting malicious network data flows. Based on different detection principles, NIDS / NIPS are divided into anomaly detection-based NIDS / NIPS and misuse detection-based NIDS / NIPS. The former is rarely used in practice; the latter, often referred to as signature-based or rule-based NIDS / NIPS, compares network data flows, typically application-layer payloads, with pre-defined rules, and data flows matching the rules are considered malicious.

[0003] However, the effectiveness of NIDS / NIPS is greatly challenged by network evasion tactics. Network evasion (also known as network intrusion detection evasion) is an act of modifying and disguising malicious network data packets, altering or hiding their original characteristics to evade NIDS / NIPS detection and thus achieve malicious attacks. Discovered network evasion techniques can be categorized by the network protocol layer they manipulate: network layer / transport layer evasion, application layer evasion, and hybrid evasion combining both. Network layer / transport layer evasion exploits the differences in IP (network layer) and TCP (transport layer) protocol parsing between the target and NIDS / NIPS. It employs methods such as IP fragmentation and TCP segmentation fragmentation and overlap to make NIDS / NIPS misunderstand the attack payload received by the target, thereby evading detection. Application layer evasion exploits the differences in application layer protocol parsing between the target and NIDS / NIPS above the TCP layer. It manipulates the application layer protocol header and payload to make NIDS / NIPS misunderstand the attack payload received by the target, thereby evading detection. In addition, there are denial-of-service attacks targeting NIDS / NIPS, which can temporarily disable them and thus evade detection, and can also be considered a kind of evasion technique.

[0004] Currently, industry-wide methods for combating network evasion techniques can be broadly categorized into two types. The first type employs compliant methods to eliminate ambiguity in network flow processing between NIDS / NIPS and the attacked host. Examples include compliance technologies based on IP header fields and TCP states, but these are ineffective against TCP fragmentation evasion. Traffic preprocessing engines process traffic destined for the internal network and then re-forward it to eliminate ambiguity in IP fragmentation understanding between NIDS and the end host. Additionally, TCP flow compliance technologies calculate the hash value of each received TCP segment to determine its place in the TCP flow queue of the connection table, determine its position in the queue based on its sequence number, and decide whether to forward, reset, or temporarily store it, thus eliminating ambiguity in TCP fragmentation understanding between NIDS and the end host. Based on these technologies, most NIDS / NIPS systems have developed and incorporated processing for IP fragmentation and TCP fragmentation. However, after some testing, it has been found that using TCP segmentation or overlap can still evade NIDS / NIPS detection, indicating that compliant methods are still insufficient to defend against TCP / IP layer evasion techniques. In response to the differences in protocol parsing between terminal hosts and NIDS / NIPS, some experts have proposed methods for actively and passively probing the host operating system to determine the differences in protocol parsing between NIDS / NIPS and the host. However, this type of compliance method currently only performs compliance at the IP (network) layer and TCP (transport) layer and cannot deal with application layer evasion techniques.

[0005] The second type of adversarial network evasion techniques is only applicable to NIDS / NIPS based on anomaly detection, i.e., NIDS / NIPS employing machine learning techniques. The basic idea is to enhance the classifier used to determine whether a network flow is malicious, making it more resistant to network evasion behavior. For example, specific assumptions can be made about the evasion strategy, leading to targeted feature selection models; optimal parameters can be searched for in centroid-based clustering models, offering game theory-based solutions to combat network evasion; and robust classifiers can be constructed using only features that remain unchanged when evasion occurs. Some experts have proposed adding evasion behavior samples to the training set to enhance NIDS's resistance to link-layer evasion, but due to the inherent "model evasion" problem in the machine learning algorithms used, anomaly detection-based NIDS / NIPS is difficult to apply in practice, and the aforementioned methods to enhance its resistance to network evasion have not achieved good results.

[0006] The applicant previously proposed two network evasion detection methods: one based on deep recurrent neural networks (ZL201810082404.8) and the other based on Bayesian classifiers (ZL201810082613.2). Both methods extract features from network data streams and train classifiers to classify these streams, thereby detecting atomic evasion behaviors. However, these methods use manually designed features, which may not be the best at distinguishing evasion behaviors. Furthermore, since these features are extracted from the IP and TCP layers, they can only detect network layer transport layer evasion and cannot detect application layer evasion behaviors. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a detection method that can detect application-layer evasion behavior using artificial intelligence technology.

[0008] An end-to-end network application layer evasion detection method includes the following steps:

[0009] Step 1: Determine the application layer that needs to detect evasion behavior, obtain the network data streams that apply various evasion behaviors of the layer that need to be identified, and the non-evasion network data streams that do not apply evasion behaviors of the layer, and process the above two types of network data streams into different categories.

[0010] Step 2: Extract the original byte stream features from the non-escape network data stream and the various escape-type network data streams in the application layer, and construct a training sample set;

[0011] Step 3: Construct a neural network and perform classification training on the network data stream sample set obtained in Step 2;

[0012] Step 4: Using the neural network obtained in Step 3, identify and classify the network data stream to be detected, and determine whether the network data stream is an evasive network data stream or an evasive network data stream that uses application-layer evasion technology. If it is identified as an evasive network data stream, then identify the category of the application-layer evasion technology used.

[0013] The network data streams of various evasion behaviors described in step 1 are processed in the following way: Define the application layer payload set P, the application layer evasion technique set E, and the parameter set O corresponding to each evasion technique. Combine different payloads in P with evasion techniques E using different parameters O to obtain different network data streams and corresponding labels for each evasion technique. The combination can be represented by the following set:

[0014] {(p,e,o0×o1×…×o n |p∈P,e∈E,o0,o1…o n ∈Oe}

[0015] Among them, O e Let o0, o1…o represent the set of parameters for a single evasion technique e. n Let each of the following represent a set of values ​​for each parameter in the set: o0×o1×…×o n The Cartesian product of the row parameters of the evasion technique e; for data streams from evasion-free networks, they are classified into a separate class.

[0016] The construction method of the training samples in step 2 is as follows:

[0017] Take the i-th evasion-type network data stream, which contains len_i application layer data frames. Take the t-th application layer data frame and extract the original byte data from its header to form a row vector. Extract raw byte data from the parameter block and data block of the data frame to form a row vector. Will By superimposing along the row direction, we obtain a matrix.

[0018]

[0019] Will Concatenate the first and last elements in chronological order to form a row vector.

[0020]

[0021] Build and The tuples formed That is, the raw byte stream features extracted from the i-th evasion class network data stream, and their corresponding category label y. i This constitutes the sample set One of the samples:

[0022]

[0023] The neural network described in step 3 mainly consists of two parts: a temporal network and a convolutional network. The temporal network is used to capture inter-message features, while the convolutional network is used to capture intra-message features. The input data includes... and After being processed by temporal networks and convolutional networks respectively, they are concatenated into a one-dimensional vector in the merging layer, then fed forward by a fully connected layer, and finally converted into the probability of belonging to each escape category by the Softmax layer.

[0024] Step 4 specifically includes: separating the application layer data from the network data stream to be detected and identified; extracting the original byte stream features from the detected application layer data as input; and using the constructed neural network to classify the input original byte stream features and identify the specific category of the network stream corresponding to the sequence.

[0025] This invention addresses the inability of existing technologies to detect and identify application-layer evasion behaviors targeting NIDS / NIPS in networks. From a machine learning perspective, this invention utilizes a deep learning neural network-based method to achieve online identification of application-layer evasion behaviors. The principle of this invention is that when an attacker uses application-layer evasion techniques, the network data stream exhibits intra-frame and inter-frame characteristics in the application-layer protocol header and payload that differ from non-evading data streams. Different evasion techniques correspond to different characteristics. Therefore, this invention proposes to extract these characteristics by training a neural network based on recurrent and convolutional neurons, and then classifying these characteristics to achieve the detection and identification of application-layer evasion techniques.

[0026] The method of the present invention does not require manually designed extracted features and does not rely on manually preset detection methods or detection thresholds. Therefore, the technology of the present invention has the advantages of adaptability and self-learning. Attached Figure Description

[0027] Figure 1 This is a flowchart of an end-to-end network application layer evasion detection method;

[0028] Figure 2 This is a schematic diagram of a neural network structure;

[0029] Figure 3 This is a schematic diagram of a one-dimensional convolutional layer;

[0030] Figure 4 This is a schematic diagram of the unfolded layers;

[0031] Figure 5 This is a diagram showing the confusion matrix on the test set;

[0032] Figure 6 It is the ROC curve of the detection performance on the test set;

[0033] Figure 7 This shows the details of the ROC curve for the detection performance on the test set. Detailed Implementation

[0034] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. These embodiments are implemented based on the technical solution of the present invention, taking the detection of evasion techniques found in the SMB 1.0 application layer protocol proposed by Microsoft as an example, and provide detailed implementation methods and processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0035] Figure 1 This is a flowchart of an end-to-end network application layer evasion detection method, such as... Figure 1 As shown, the method includes the following steps:

[0036] Step 1: Obtain network data streams that have applied the various evasion behaviors of the SMB1.0 application layer that need to be identified, as well as non-evasion network data streams that have not applied the evasion behaviors of this layer;

[0037] McAfee provides Evader, a tool for testing the resilience of NIDS / IPS against various evasion attempts. This embodiment uses Evader to attack a Windows XP SP2 target machine with an unpatched CVE-2008-4250 vulnerability, applying various SMB 1.0 layer evasion techniques to obtain network data streams for identifying different SMB 1.0 application layer evasion attempts. By combining different payloads with various SMB 1.0 layer evasion techniques with different parameters, different network data streams for each evasion technique can be obtained.

[0038] Specifically, five types of loads were used, denoted by the set symbol P:

[0039] P={clean,shell,bshell,calc,msgbox},

[0040] The specific meanings of these five load types are shown in Table 1:

[0041] Table 1 Loads and their meanings

[0042]

[0043] Seven evasion techniques from the SMB 1.0 layer were used, denoted by the set symbol E:

[0044]

[0045] The parameters used when employing these 7 types of evasion are shown in Table 2 (E represents the evasion technique, and O represents the corresponding parameter). Each row in the table specifies the possible values ​​of a parameter. [m,n] specifies an integer range containing every integer from m to n, such as [64,1024]; [m,s,n] specifies an integer sequence containing every integer starting from m, increasing by s, and not exceeding n, such as [1024,8,2049]; {a,b,c} specifies a set, where each element can be a value for the parameter, such as {6,7,8} and {zero,random,random_alphanum}. Some evasions require only one parameter and correspond to only one row in the table, such as msrpc_groupsends; others require multiple parameters and correspond to multiple rows in the table, such as smb_chaff, which requires 3 parameters with values ​​of 100%, write_flag, and one element from {zero,rand,alphanum,msrpc}.

[0046] Table 2. Names and parameters of SMB layer evasion

[0047]

[0048] By combining different loads in P with evasion techniques E using various parameters O, different network data flows can be obtained for each evasion technique. The combinations can be represented by the following set:

[0049] {(p,e,o0×o1×…×o n |p∈P,e∈E,o0,o1…o n ∈O e}

[0050] Among them O e Let o0, o1…o represent the set of parameters for a single evasion technique e. n Let each of the following represent a set of values ​​for each parameter in the set: o0×o1×…×o n The Cartesian product of the row parameters of the evasion technique e.

[0051] The obtained evasion behavior network data stream is saved as a pcap file in pcap format, with each combination (p, e, o0×o1×…×o) n The generated network data stream is saved as a separate pcap file. These files are then organized into directories with the same name as the evasion technique e used. For example, the pcap file using the smb_seg evasion technique in the combination should be placed in a directory named smb_seg.

[0052] This embodiment also uses the evader tool to obtain evasion-free network data streams without applying any evasion techniques. By obtaining pcap files of evasion-free network data streams without applying any evasion techniques when attacking the target machine, these files are placed in a directory named "normal".

[0053] In this way, eight types of network data streams were obtained, including seven evasion techniques of the SMB1.0 application layer and those that did not use SMB1.0 application layer evasion techniques (no evasion network data streams). These were saved in pcap file format in eight directories with the same name as the class. The number of each type of network data stream collected in this embodiment is shown in Table 3:

[0054] Table 3 Quantity of various network data streams

[0055]

[0056] The seven evasion technique categories and the class of non-evasion data flow (normal) were encoded, and the resulting category labels are shown in Table 4.

[0057] Table 4 Data Stream Category Labels

[0058]

[0059] Step 2: Extract raw byte stream features from the non-evasive network data stream and the various evasion-type network data streams at the application layer to construct a training sample set. This embodiment extracts raw byte stream features from the 8 types of network data streams obtained in Step 1. From the pcap file of each network data stream, extract the raw bytes of the SMB1.0 protocol header in each frame containing SMB1.0 application layer protocol data, as well as the raw bytes of the SMB parameter block and data block that may appear after the SMB protocol header, to constitute the sample corresponding to that data stream. Specifically, the total number of samples is N, expressed as D. i This represents the i-th raw byte data stream from the network, and its category label is y. i D indicates. i It contains len_i frames containing SMB1.0 messages, where the raw byte data of the t-th frame is in the format d. t express:

[0060] D i ={d1,…,d t ,…d len_i}, 1≤t≤len_i

[0061] From the t-th frame d t Extract 32 bytes of the original SMB 1.0 application layer protocol header data from each segment (in words). Row vector representation), and parameter blocks and data blocks of variable byte length (in words). (Row vector representation), where each byte of data is represented by an integer:

[0062]

[0063] From D i Captured from each frame By superimposing along the row direction, we obtain a matrix.

[0064]

[0065] From D i Captured from each frame By concatenating the first and last frames in chronological order, a row vector is formed.

[0066]

[0067] and The tuples formed That is, the raw byte stream features extracted from the i-th network data stream, and the label y i This constitutes the sample set One of the samples:

[0068]

[0069] 1≤i≤N

[0070] This embodiment extracts data from an SMB 1.0 application layer evasion network data stream (this data stream contains 22 frames of SMB 1.0 protocol messages). As shown in Table 5:

[0071] Table 5 Sample SMB Protocol Header Raw Byte Data Example

[0072]

[0073] This is a 22×32 matrix, with each row representing an SMB protocol message header. The extracted... It is a row vector of length 1284.

[0074] Step 3: Construct a neural network and perform classification training on the network data stream training sample set obtained in Step 2. To accurately capture the different characteristics of various evasion behaviors of the SMB1.0 application layer within and between SMB messages, the neural network constructed in this embodiment consists of two parts: a temporal network for capturing inter-message features and a convolutional network for capturing intra-message features. Figure 2 This is a schematic diagram of the constructed neural network structure. The "data stream sample" is the tuple extracted from each network data stream in step 2. 1 ≤ i ≤ N is the network input. The output "Data Flow Category" is the category label shown in Table 4. The input... and After being processed by the temporal network part (upper branch) and the convolutional network part (lower branch), they are concatenated into a one-dimensional vector in the "merging layer", then fed forward by two fully connected layers, and finally converted into the probability of belonging to each class by the Softmax layer.

[0075] The temporal network portion is composed of a long short-term memory (LSTM) network, based on the sample... As input, consider it as a 32-dimensional time series, one row of a matrix Given an input at time t, calculate the hidden node state h at time t using the following formula. t and unit state c t :

[0076]

[0077]

[0078]

[0079]

[0080] c t =f t ⊙c t-1 +i t ⊙g t

[0081] h t =o t ⊙tanh(c t )

[0082] Where h t-1 Let c be the hidden node state at time t-1. t-1 This represents the cell state at time t-1. t For cell gates, f represents candidate update values ​​for cell states; tThe forget gate determines the cell state c from time t-1. t-1 What information is retained? t The input gate determines the candidate update value g of the cell state. t In the updated cell state c t Weights in; o t For output gate, and c t Together determine the updated hidden node state h t σ is the sigmoid function, tanh is the hyperbolic tangent function, and ⊙ represents the element-wise multiplication of matrices.

[0083] The number of SMB1.0 messages contained in each sample is different, corresponding to... The number of rows differs. In this embodiment, the temporal part of the neural network consists of a single-layer LSTM network with 32 units. When a sample is input... After each line, output the hidden node state h at the last time step. len_i , is a single-channel one-dimensional vector with a length of 32.

[0084] The convolutional network portion consists of stacked one-dimensional convolutional layers (Conv1d), with the sample... Given the input, the first L elements are truncated. The working principle of a one-dimensional convolutional layer is as follows: Figure 3 As shown.

[0085] Using c convolutional kernels of width k, the raw byte stream of SMB parameter blocks and data blocks extracted from the network data stream is processed. Convolution operations are performed on the first L elements of this one-dimensional signal. When the stride is 1, c different convolution results are obtained, also called channels, and the length of each channel is L-k+1. This embodiment uses two one-dimensional convolutional layers. The first layer has c=48 channels and a kernel width k=3. In this embodiment, the truncation length L=512 is set, so the output of the first convolutional layer is 48 channels with a length of 510, as shown below. Figure 2 As shown in Figure 48@510. The second layer has c=64 channels and k=3 kernel width, so the output of the second convolutional layer is 64 channels with a length of 508, as shown in Figure 48@510. Figure 2 As shown in 64@508.

[0086] The expansion layer expands multiple output channels of the previous layer into a single channel, that is, it connects all input channels end-to-end into one channel, such as... Figure 4 As shown:

[0087] Therefore, this layer expands the 64 channels of length 508 into a single-channel one-dimensional vector of length 64 × 508 = 32512, as follows: Figure 2 As shown in 1@32512.

[0088] The merging layer will output a 32-bit h from the temporal network portion. len_i A one-dimensional vector is concatenated with a one-dimensional vector of length 32512 from the output of the unfolded layer of the convolutional network to form a new one-dimensional vector, as shown below. Figure 2 As shown in 1@32544.

[0089] Each neuron in the fully connected layer is fully connected to all neurons in the layer preceding it. The first fully connected layer after the merged layer is fully connected to all 32,544 outputs of the merged layer, resulting in a single-channel output of length 8, or 1@8. The second fully connected layer is connected to the first fully connected layer, also outputting a single-channel output of length 8, or 1@8. The Softmax layer is the standard output layer for a multi-class classification problem, implementing a Softmax operation on the outputs of each neuron in the second fully connected layer. The output is a one-dimensional vector consisting of the probabilities of the sample belonging to one of the eight classes defined in this embodiment, as shown below. Figure 2 1@9. For the output x of the j-th neuron in the second fully connected layer. j Transform it into 0≤c≤7,0≤g(x j )≤1 represents the probability that the input network data stream sample belongs to the j-th class. The index j corresponding to the unit with the largest transformed result value is the predicted type label of the network data stream sample, as shown in Table 4.

[0090] Using the sample set obtained in step 2 Train the network. The samples of each category are randomly sorted, and the sorted sample set is divided into three parts: training set, validation set, and test set. Let the features of the i-th sample in the training set be... After the "data stream sample" is presented at the network input, the value of the "data stream category" at the output is equal to its label y. j During training, an early termination strategy to prevent overfitting was adopted, meaning that training was stopped if the accuracy on the validation set began to decline.

[0091] The performance on the test set is shown in Table 6, which contains a confusion matrix. Figure 5 (It is its more intuitive graphical representation), Table 7 Detection Evasion Performance and Figure 6 The ROC curve is shown. Since most ROC curves are concentrated in... Figure 6 The top left corner is for displaying details. Figure 7 Details of the ROC curve within the false positive rate range [0, 0.01] are shown.

[0092] Table 6. Confusion Matrix on the Test Set

[0093]

[0094] Table 7 Detection performance on the test set

[0095]

[0096] Step 4: Separate the application layer data from the network data stream to be detected and identified. Using the method in Step 2, extract the original byte stream features from the detected application layer data and use them as the input to the neural network constructed in Step 3. Based on the network output, retrieve Table 4 to identify whether the network stream corresponding to the sequence is a non-evasive network stream or a network stream with some kind of application layer evasion, and identify the category of application layer evasion.

Claims

1. An end-to-end network application layer evasion detection method, characterized in that, Includes the following steps: Step 1: Determine the application layer that needs to detect evasion behavior, obtain the network data streams that apply various evasion behaviors of the layer that need to be identified, and the non-evasion network data streams that do not apply evasion behaviors of the layer, and process the above two types of network data streams into different categories. The network data streams of various evasion behaviors described in step 1 are processed in the following way: Define the application layer payload set P, the application layer evasion technique set E, and the parameter set O corresponding to each evasion technique. Combine different payloads in P with evasion techniques E using different parameters O to obtain different network data streams and corresponding labels for each evasion technique. The combination can be represented by the following set: {(p,e,o0×o1×…×o n )|p∈P,e∈E,o0,o1…o n ∈O e } Among them, O e Let o0, o1…o represent the set of parameters for a single evasion technique e. n Let each of the following represent a set of values ​​for each parameter in the set: o0×o1×…×o n The Cartesian product of the row parameters of the evasion technique e; for data streams from evasion-free networks, they are classified into a separate class. Step 2: Extract the original byte stream features from the non-escape network data stream and the various escape-type network data streams in the application layer, and construct a training sample set; The construction method of the training samples in step 2 is as follows: Take the i-th evasion-type network data stream, which contains len_i data frames. Take the t-th data frame and extract the original byte data from its header to form a row vector. Extract raw byte data from the parameter block and data block of the data frame to form a row vector. Will By superimposing along the row direction, we obtain a matrix. Will Concatenate the first and last elements in chronological order to form a row vector. Build and The tuples formed That is, the raw byte stream features extracted from the i-th evasion class network data stream, and their corresponding category label y. i This constitutes the sample set One of the samples: Step 3: Construct a neural network and perform classification training on the network data stream sample set obtained in Step 2; The neural network described in step 3 consists of two parts: a temporal network and a convolutional network. The temporal network is used to capture inter-message features, while the convolutional network is used to capture intra-message features. The input data includes... and After being processed by temporal networks and convolutional networks respectively, they are concatenated into a one-dimensional vector in the merging layer, then fed forward by a fully connected layer, and finally converted into the probability of belonging to each escape category by the Softmax layer. Step 4: Using the neural network obtained in Step 3, identify and classify the network data stream to be detected, and determine whether the network data stream is an evasive network data stream or an evasive network data stream that uses application-layer evasion technology. If it is identified as an evasive network data stream, then identify the category of the application-layer evasion technology used.

Images