Abnormal traffic detection method based on time-frequency domain information fusion

Abstract

The application discloses an abnormal flow detection method based on time-frequency domain information fusion, collects a plurality of normal data flows and abnormal data flows from a network as data flow samples according to actual needs, respectively extracts a packet-level time domain information matrix and a flow-level time domain information matrix for each data flow sample, constructs a frequency domain information matrix of each data flow based on the packet-level time domain information matrix, generates a data flow feature image according to the three information matrices, constructs an abnormal detection model according to actual needs and trains the abnormal detection model by using the data flow samples, captures data flows from the network and generates data flow feature images during abnormal flow detection, inputs the trained abnormal detection model, and obtains an abnormal flow detection result. The application can effectively improve the accuracy, efficiency and robustness of abnormal flow detection.

Description

Technical Field

[0001] This invention belongs to the field of traffic analysis technology, and more specifically, relates to a method for abnormal traffic detection based on time-frequency domain information fusion. Background Technology

[0002] Anomaly detection is a crucial task in traffic analysis, involving the detection of abnormal data from a given traffic dataset. The results of anomaly detection can then be used to identify network intrusion attempts.

[0003] Currently, there are four main types of anomaly detection methods: classification-based, statistical-based, information theory-based, and clustering-based. However, each of these methods has its own limitations when used individually: classification techniques, which rely on extensive expert knowledge of network attack characteristics, struggle to detect new types of anomalies; statistical-based methods typically require pre-setting static thresholds for anomaly detection, making them unsuitable for dynamic changes in network traffic; and classification methods based on information theory or clustering tend to have high false positive rates in anomaly detection. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide an abnormal traffic detection method based on time-frequency domain information fusion. By extracting the packet-level time-domain information matrix, the flow-level time-domain information matrix, and the frequency-domain information matrix of the data flow, a data flow feature image is constructed for abnormal traffic detection, thereby improving the accuracy, efficiency, and robustness of abnormal traffic detection.

[0005] To achieve the above-mentioned objectives, the abnormal traffic detection method based on time-frequency domain information fusion of the present invention includes the following steps:

[0006] S1: Collect several normal and abnormal data streams from the network as data stream samples according to actual needs. Label each data stream sample. If it belongs to normal traffic, set the label flag=1; otherwise, set the label flag=0.

[0007] S2: Extract the packet-level and stream-level time-domain information matrices of the data stream. Specifically, set M data packet features according to actual needs, with the value of M determined based on those needs. For each data stream, extract the value p of each data packet feature from the first N data packets. m,n Given n = 0, 1, ..., N-1 and m = 0, 1, ..., M-1, we can construct the packet-level time-domain information matrix P:

[0008]

[0009] M data stream features are set according to actual needs. For each data stream, the value f of each data stream feature is extracted from the data stream consisting of the 0th to the nth data packets. n,m Thus, the stream-level time-domain information matrix F is constructed:

[0010]

[0011] S3: Construct the frequency domain information matrix (LTM) of each data stream based on the packet-level time domain information matrix P. The specific method is as follows:

[0012] S3.1: Perform max-min normalization on the packet-level time-domain information matrix P column-wise to obtain the normalized packet-level time-domain information matrix.

[0013] S3.2: Packet-level temporal information matrix after column-pair normalization Perform a discrete Fourier transform to obtain the complex matrix FTP;

[0014] S3.3: Calculate the modulus of each complex number in the complex matrix FTP to obtain the modulus matrix FTPM;

[0015] S3.4: For each element value ftpm of the modular matrix FTPM m,n The logarithmic value ltm is obtained by performing a logarithmic transformation. m,n The resulting logarithmic matrix LTM is used as the packet-level frequency domain information matrix;

[0016] S4: For each data stream, the packet-level time-domain information matrix P, the stream-level time-domain information matrix F, and the packet-level frequency-domain information matrix LTM are assigned to the red, green, and blue channels respectively, and a three-channel color image is constructed as the data stream feature image;

[0017] S5: Construct an anomaly detection model according to actual needs. Its input is the data stream feature image, and its output is the judgment result of whether the data stream is normal. Take the data stream feature image of each data stream sample as input and the corresponding label as the expected output to train the anomaly detection model and obtain the trained anomaly detection model.

[0018] S6: Capture a data stream containing at least N data packets from the network. Extract the packet-level time-domain information matrix and the stream-level time-domain information matrix using the same method as in step S2. Obtain the frequency-domain information matrix based on the packet-level time-domain information matrix using the same method as in step S3. Generate a data stream feature image using the same method as in step S4. Then input the anomaly detection model trained in step S5 to obtain the judgment result of whether the data stream is normal, thus completing the data stream detection.

[0019] This invention relates to an abnormal traffic detection method based on time-frequency domain information fusion. It collects several normal and abnormal data streams from the network as data stream samples according to actual needs. For each data stream sample, it extracts packet-level and flow-level time-domain information matrices. Based on the packet-level time-domain information matrix, it constructs a frequency-domain information matrix for each data stream. A data stream feature image is generated based on the three information matrices. An anomaly detection model is constructed according to actual needs and trained using data stream samples. During abnormal traffic detection, data streams are captured from the network, and a data stream feature image is generated and input into the trained anomaly detection model to obtain the abnormal traffic detection result.

[0020] This invention generates data stream feature images based on packet-level time-domain information matrices, flow-level time-domain information matrices, and packet-level frequency-domain information matrices. It can effectively fuse time-frequency domain information and improve the accuracy, efficiency, and robustness of abnormal traffic detection. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a specific implementation of the abnormal traffic detection method based on time-frequency domain information fusion according to the present invention.

[0022] Figure 2 This is a flowchart illustrating the construction of a frequency domain information matrix based on a packet-level time domain information matrix in this invention.

[0023] Figure 3 This is a structural diagram of the first implementation of the anomaly detection model in this embodiment;

[0024] Figure 4 This is a structural diagram of the second implementation of the anomaly detection model in this embodiment. Detailed Implementation

[0025] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0026] Example

[0027] Figure 1 This is a flowchart illustrating a specific implementation of the abnormal traffic detection method based on time-frequency domain information fusion according to the present invention. Figure 1 As shown, the specific steps of the abnormal traffic detection method based on time-frequency domain information fusion of the present invention include:

[0028] S101: Obtain a data stream sample:

[0029] Based on actual needs, collect several normal and abnormal data streams from the network as data stream samples. Label each data stream sample. If it belongs to normal traffic, set the label flag=1; otherwise, set the label flag=0.

[0030] In this embodiment, a network traffic dataset containing normal and abnormal traffic is first crawled. The original network traffic dataset is then segmented into streams according to the five-tuple, and the stray streams are cleaned out. The remaining data streams are used as data stream samples.

[0031] S102: Constructing the time-domain information matrix:

[0032] There are some key differences between abnormal network traffic and normal network traffic, which can help identify potential cybersecurity incidents or anomalous behavior, such as:

[0033] Traffic volume and frequency: Abnormal network traffic is typically characterized by very high traffic volumes or unusual traffic frequencies. This could be due to malware, distributed denial-of-service attacks, botnets, or other network attacks.

[0034] Packet structure: Abnormal network traffic may have a different packet structure than normal traffic. This includes very large or very small packets, invalid packet headers, abnormal packet flags, etc.

[0035] Data content: Abnormal network traffic may contain unusual data content. This could include specific files transmitted by malware, unusual data formats, known malicious code, etc.

[0036] User behavior: Abnormal network traffic may involve unusual patterns of user behavior. For example, abnormal login attempts, abnormal data access patterns, and abnormal data download volumes.

[0037] To represent the temporal information of the data stream, this invention extracts the packet-level temporal information matrix and the stream-level temporal information matrix of the data stream. The extraction methods are as follows:

[0038] 1) Packet-level time-domain information matrix

[0039] This invention extracts the first N data packets of each data stream to construct a packet-level temporal information matrix. The specific method is as follows:

[0040] M data packet features are set according to actual needs, and the value of M is determined based on actual needs. For each data stream, the value p of each data packet feature is extracted from the first N data packets. m,n Given n = 0, 1, ..., N-1 and m = 0, 1, ..., M-1, we can construct the packet-level time-domain information matrix P:

[0041]

[0042] In this embodiment, there are a total of 6 data packet characteristics, including the chi-square statistic and corresponding P-value (the frequency of its transport layer payload bytes compared to the frequency of normal stream transport layer payload bytes), information entropy, packet length, protocol type, and uplink / downlink attributes.

[0043] 2) Stream-level time-domain information matrix

[0044] Similarly, this invention extracts the first N data packets of each data stream to construct a stream-level time-domain information matrix. Specifically, M data stream features are set according to actual needs. For each data stream, the value f of each data stream feature is extracted from the data stream composed of the 0th to the nth data packets. n,m Thus, the stream-level time-domain information matrix F is constructed:

[0045]

[0046] In this invention, the selection of data stream features adopts a feature selection technique based on the Gini coefficient, which is verified by the random forest algorithm to obtain six data stream features, including total uplink packet length, variance of uplink and downlink packet length, total downlink packet length, total uplink and downlink packet length, variance of uplink packet length, and average uplink and downlink packet length.

[0047] S103: Constructing the frequency domain information matrix:

[0048] Next, the packet-level time-domain information matrix P of each data stream is used to construct the frequency-domain information matrix LTM for that data stream. Figure 2 This is a flowchart illustrating the construction of a frequency domain information matrix based on a packet-level time domain information matrix in this invention. For example... Figure 2 As shown, the specific steps for constructing a frequency domain information matrix based on a packet-level time domain information matrix in this invention include:

[0049] S201: Normalization process:

[0050] The packet-level time-domain information matrix P is subjected to min-max normalization column-wise to obtain the normalized packet-level time-domain information matrix. The specific method is as follows: First, find the maximum value (max) of each column in the packet-level time-domain information matrix. n and minimum value min n Then, the following formula is used to map each value to the range of 0 to 1, resulting in the normalized feature values.

[0051]

[0052] Using this formula, each value in the packet-level temporal information matrix is ​​remapped to the range of 0 to 1. The minimum value corresponds to 0, the maximum value to 1, and other values ​​are mapped according to their position relative to the minimum and maximum values. This results in a minimax-normalized packet-level temporal information matrix.

[0053] S202: Discrete Fourier Transform:

[0054] The normalized packet-level temporal information matrix by column Perform a Discrete Fourier Transform to obtain the complex matrix FTP. Specifically, this involves transforming the normalized packet-level time-domain information matrix... Each column in the matrix undergoes a Discrete Fourier Transform (DFT) to convert the time-domain signal into a complex frequency-domain signal, containing both amplitude and phase information. This results in a new complex matrix, FTP, where each column corresponds to a column in the original packet-level time-domain information matrix but now contains a complex value represented in the frequency domain. Each complex value in FTP... m,n This represents the amplitude and phase of the corresponding frequency component:

[0055]

[0056] S203: Calculate the modulus matrix:

[0057] Calculate the modulus of each complex number in the complex matrix FTP to obtain the modulus matrix FTPM. The specific method is as follows:

[0058] ftp m,n =a m,n +jb m,n

[0059]

[0060] Therefore:

[0061]

[0062] Thus, the modulus matrix FTPM is obtained:

[0063]

[0064] S204: Logarithmic Transformation

[0065] To ensure the numerical stability of frequency domain features and prevent floating-point overflow during machine learning model training, the value of each element ftpm in the modulus matrix FTPM is... m,n The logarithmic value ltm is obtained by performing a logarithmic transformation. m,n The resulting logarithmic matrix LTM is used as the packet-level frequency domain information matrix. The formula for the logarithmic transform is as follows:

[0066]

[0067] Where C represents a preset constant used to adjust the range of frequency domain characteristics.

[0068] Thus, the logarithmic matrix LTM is obtained:

[0069]

[0070] S104: Generate data stream feature image:

[0071] For each data stream, the packet-level time-domain information matrix P, the stream-level time-domain information matrix F, and the packet-level frequency-domain information matrix LTM are assigned to the red, green, and blue channels respectively, constructing a three-channel color image as the data stream feature image. It can be seen that each pixel in the data stream feature image contains color information from all three channels. Such a color image can more vividly and richly express the features and attributes contained in each original information matrix.

[0072] S105: Construct and train the anomaly detection model:

[0073] An anomaly detection model is constructed based on actual needs. Its input is the data stream feature image, and its output is the judgment result of whether the data stream is normal. The data stream feature image of each data stream sample is used as input, and the corresponding label is used as the expected output. The anomaly detection model is trained to obtain the trained anomaly detection model.

[0074] In practical applications, anomaly detection models can be set up as needed. Figure 3 This is a structural diagram of the first implementation of the anomaly detection model in this embodiment. For example... Figure 3 As shown, the anomaly detection model in this embodiment includes a residual neural network (ResNet) and a fully connected layer, wherein:

[0075] Residual neural networks are used to extract feature vectors from feature images in a data stream and output them to fully connected layers.

[0076] The fully connected layer is used to process the feature vectors and output the probability that the data stream corresponding to the feature image of the data stream is a normal flow.

[0077] Residual neural networks (RNNs) are a commonly used type of neural network. They utilize "shortcut connections" to address degradation, significantly reducing the training difficulties of excessively deep neural networks and improving network performance. This embodiment employs a Residual Neural Network with an attention mechanism (ResNet With CBAM). Attention is introduced into each ResBlock along both the channel and spatial dimensions. The module infers the attention map sequentially along these two independent dimensions, then multiplies the weights by the input feature map for adaptive feature refinement. For the channel attention module, the feature map is first compressed in the spatial dimension using average pooling and max pooling to obtain a one-dimensional vector before further processing. Average pooling provides feedback to every pixel in the feature map, while max pooling, during gradient backpropagation, only provides gradient feedback to the areas with the largest response in the feature map, serving as a supplement to average pooling. For the spatial attention module, average pooling and max pooling are used to compress the input feature map, but this compression is now at the channel level, performing averaging and maximizing operations on the input features along the channel dimension. Finally, two two-dimensional features were obtained, which were concatenated together according to the channel dimension to obtain a feature map with 2 channels. Then, a hidden layer containing a single convolutional kernel was used to perform a convolution operation on it.

[0078] for Figure 3 The anomaly detection model shown has a loss function LOSS1 during training, which is the sum of the loss from the anomaly detection task and the regularization term. The calculation formula is as follows:

[0079] LOSS1=L+αL reg

[0080] Where L represents the cross-entropy loss of the anomaly detection task, L reg The L2 norm regularization term is used to prevent overfitting in the neural network, and α represents a preset hyperparameter used to balance the weight of the regularization term in the overall objective function. Cross-entropy loss L and regularization term Li reg The calculation formulas are as follows:

[0081]

[0082] Where N represents the number of training samples in a patch, y i y i ' and ' represent the true label and predicted label of the i-th training sample in a patch, respectively.

[0083]

[0084] Where M represents the number of layers in the neural network, W(k) b (k) These represent the weights and bias parameters of each layer in the neural network of the anomaly detection model.

[0085] Figure 4 This is a structural diagram of the second implementation of the anomaly detection model in this embodiment. For example... Figure 4 As shown, the anomaly detection model in this embodiment includes a residual neural network (ResNet), a feature clustering module, and an anomaly determination module, wherein:

[0086] The residual neural network (RNN) is used to extract feature vectors from the data stream feature image. It sends the feature vectors corresponding to data stream samples to the feature clustering module and sends the feature vectors corresponding to the data stream to be detected to the anomaly detection module. The RNN uses contrastive learning to determine its parameters. Specifically, the RNN acts as an encoder network, and the fully connected neural network acts as a projection network. The RNN extracts feature vectors from the data stream feature image and sends them to the projection network. The RNN compares and contrasts the feature images of normal traffic and abnormal traffic as two classes of data to determine the parameters of the RNN. The loss function used in the contrastive learning is the commonly used InfoNCE loss.

[0087] The feature clustering module is used to cluster the feature vectors of data stream samples to obtain the feature vector sets of normal data streams and abnormal data streams. It then calculates the cluster centers of the normal data streams, calculates the mean distance D between each normal data stream and its corresponding cluster center, and finally sends the cluster centers and mean distance D of the normal data streams to the anomaly detection module.

[0088] The anomaly detection module is used to determine abnormal traffic based on the cluster center and mean distance D between the feature vector of the detected data stream and the normal data stream. Specifically, the distance d between the feature vector of the detected data stream and the cluster center of the normal data stream is calculated. If d > D, that is, the feature vector of the detected data stream deviates from the cluster center of the normal data stream, the detected data stream is determined to be abnormal data; otherwise, it is normal data.

[0089] A comparison of the implementation methods of the two anomaly detection models in this embodiment shows that the first implementation method is more suitable for anomaly detection tasks in closed-world scenarios, while the second implementation method is more suitable for anomaly detection tasks in open-world scenarios. In practical applications, the appropriate method can be selected as needed.

[0090] S106: Data Stream Inspection

[0091] The data stream containing at least N data packets is captured from the network. The packet-level time-domain information matrix and the stream-level time-domain information matrix are extracted using the same method as in step S102. The frequency-domain information matrix is ​​obtained based on the packet-level time-domain information matrix using the same method as in step S103. The data stream feature image is generated using the same method as in step S104. Then, it is input into the anomaly detection model trained in step S105 to obtain the judgment result of whether the data stream is normal, thus completing the data stream detection.

[0092] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A method for detecting abnormal traffic based on time-frequency domain information fusion, characterized in that, The method comprises the following steps: S1: Collecting a plurality of normal data streams and abnormal data streams from the network according to actual needs as data stream samples, and labeling each data stream sample, if it belongs to normal traffic, then the label flag = 1, otherwise the label flag = 0; S2: Extract the packet-level time domain information matrix and the stream-level time domain information matrix of the data stream, specifically: set M packet features according to actual needs, the value of M is determined according to actual needs, for each data stream, extract the value p of each packet feature in the first N packets m,n , n = 0, 1,..., N-1, m = 0, 1,..., M-1, thereby constructing the packet-level time domain information matrix P: According to actual needs, M data stream features are set, and for each data stream, values f of each data stream feature in data streams composed of 0th to nth data packets are extracted respectively n,m Thus, a flow-level time-domain information matrix F is constructed S3: Constructing the frequency domain information matrix LTM of each data stream based on the packet-level time domain information matrix P of the data stream, the specific method being: S3.1: Performing maximum and minimum normalization processing on the packet-level time domain information matrix P by column to obtain the normalized packet-level time domain information matrix P; S3.2: Performing discrete Fourier transform on the normalized packet-level time domain information matrix P by column to obtain a complex matrix FTP; S3.3: Calculating the modulus of each complex number in the complex matrix FTP to obtain the modulus matrix FTPM; S3.4: the value of each element of the modulo matrix FTPM is ftpm m,n The logarithmic value ltm is obtained by performing logarithmic transformation m,n The obtained logarithmic matrix LTM is taken as the packet-level frequency domain information matrix; S4: For each data stream, respectively assigning the packet-level time domain information matrix P, the flow-level time domain information matrix F and the packet-level frequency domain information matrix LTM to the red, green and blue three channels to construct a three-channel color image as a data stream feature image; S5: Constructing an anomaly detection model according to actual needs, the input of which is the data stream feature image, and the output of which is the determination result of whether the data stream is normal, taking the data stream feature image of each data stream sample as the input and the corresponding label as the expected output, training the anomaly detection model to obtain the trained anomaly detection model; S6: Grabbing a data stream containing at least N data packets from the network, extracting the packet-level time domain information matrix and the flow-level time domain information matrix by the same method in step S2, obtaining the frequency domain information matrix based on the packet-level time domain information matrix by the same method in step S3, generating the data stream feature image by the same method in step S4, and then inputting the trained anomaly detection model in step S5 to obtain the determination result of whether the data stream is normal, thereby completing the data stream detection.

2. The abnormal traffic detection method according to claim 1, characterized by, The data packet features in step S2 include a total of 6 kinds, including the chi-square statistics and the corresponding P value, the information entropy, the packet length, the protocol type and the uplink and downlink attributes.

3. The abnormal traffic detection method according to claim 2, characterized by, The data stream features in step S2 include a total of 6 kinds, including the total length of uplink packets, the uplink and downlink packet length variance, the total length of downlink packets, the total length of uplink and downlink packets, the uplink packet length variance and the uplink and downlink packet length average.

4. The abnormal traffic detection method according to claim 1, characterized by, The anomaly detection model in step S5 includes a residual neural network and a fully connected layer, wherein: The residual neural network is used to extract a feature vector from the data stream feature image and output it to the fully connected layer; The fully connected layer is used to process the feature vector and output the probability that the data stream corresponding to the data stream feature image is normal traffic.

5. The abnormal traffic detection method according to claim 1, characterized by, The anomaly detection model in step S5 includes a residual neural network, a fully connected network, a feature clustering module and an anomaly determination module, wherein: The residual neural network is used to extract a feature vector from a data stream feature image, and the feature vector corresponding to the data stream sample is sent to the feature clustering module, and the feature vector corresponding to the to-be-detected data stream is sent to the anomaly judgment module; the residual neural network adopts contrast learning to determine parameters, and the specific method is as follows: the residual neural network is used as an encoding network, and a fully connected neural network is used as a projection network; the residual neural network is used to extract a feature vector from a data stream feature image and send it to the projection network; the data stream feature image of normal traffic and the data stream feature image of abnormal traffic are used as two types of data for contrast learning to determine the parameters of the residual neural network; The feature clustering module is used to cluster the feature vectors of the data stream samples to obtain a feature vector set of normal data streams and a feature vector set of abnormal data streams, calculate the clustering center of the normal data streams, and then calculate the distance mean D of each normal data stream in the data stream sample and the clustering center of the normal data streams, and then send the clustering center of the normal data streams and the distance mean D to the anomaly judgment module; The anomaly judgment module is used to judge abnormal traffic according to the feature vector corresponding to the to-be-detected data stream and the clustering center and distance mean D of the normal data streams, and the specific method is as follows: the distance d between the feature vector corresponding to the to-be-detected data stream and the clustering center of the normal data streams is calculated, if d>D, that is, the feature vector of the to-be-detected data stream deviates from the clustering center of the normal data streams, then the to-be-detected data stream is determined as abnormal data, otherwise, it is normal data.

Images