A method and device for detecting malicious traffic based on window features

Abstract

The present invention discloses a malicious traffic detection method and device based on window features, the method comprising: obtaining network traffic data to be detected; calculating the characteristic value of the network traffic data to be detected according to a window-based feature extraction method to obtain a characteristic vector of first characteristic data, inputting second characteristic data whose characteristic vector meets preset conditions into a pre-trained traffic detection model to perform data type determination to obtain a data type determination result; and determining malicious traffic data in the network traffic data to be detected according to the data type determination result. The scheme implemented by the present invention can reduce the delay of machine learning in detecting malicious traffic, and can also reduce the overhead caused to network equipment during machine learning training and detection. It can also improve detection accuracy while reducing detection delay and overhead, and help improve the ease of deployment of machine learning models in network equipment with limited resources.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a malicious traffic detection method and device based on window features. Background Art

[0002] Internet technology has greatly promoted global development, but it also faces serious security issues, such as various network attacks. Distributed denial of service (DDoS) attacks are one of the most serious attacks today. Therefore, detecting malicious traffic, such as DDoS, is a crucial research area. DDoS malicious traffic depletes resources and disrupts Internet services through a large number of data packets. To avoid detection, some DDoS attackers attempt to mimic normal traffic, making detection even more difficult. Traditional DDoS detection methods, such as rule-based detection methods, are no longer able to cope with increasingly complex attacks. Rule-based detection methods generally only target specific types of attacks, such as TCP SYN floods and DNS amplification attacks. Although they can detect single attacks with high accuracy, they struggle to capture more complex attack patterns.

[0003] As a detection method, machine learning has been applied to detect malicious traffic such as DDoS. Machine learning algorithms can automatically identify attacks using knowledge learned from malicious traffic data, significantly improving detection performance. Consequently, the industry is also beginning to adopt machine learning-based DDoS detection technologies. However, due to the high requirements of deep learning models for equipment, such as high-performance computing and massive data support, it is difficult to deploy and use deep learning models on current network equipment. Compared with deep learning models, machine learning models are generally less complex and require fewer computing resources, making their deployment on network equipment more feasible for DDoS detection. Therefore, most intelligent methods deployed in the real world tend to use lightweight and easy-to-deploy machine learning models, such as decision trees, random forests, and naive Bayes.

[0004] Feature engineering is a crucial step in machine learning, as it determines whether the model can be trained on important and effective information. A classic feature extraction approach for machine learning DDoS detection is to extract traffic features based on flows (five-tuples). To extract flow-level features, a typical process is to first aggregate each received packet into different flows based on the five-tuple information in the packet header, and then calculate the feature values corresponding to each flow. While flow-based feature extraction preserves detailed flow-by-flow information, it also incurs certain detection latency and computational overhead on the detection device due to the complex flow tracking process. Furthermore, general-purpose network devices (such as gateways and home routers) generally have limited memory and computing resources. When deploying machine learning-based attack detection models on such devices, the computational overhead of the machine learning model must be kept as low as possible, enabling the device to handle high-bandwidth traffic while still running the model. From the perspective of detection overhead and latency, machine learning detection methods based on per-flow features have certain shortcomings and require further optimization. Summary of the Invention

[0005] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0006] To this end, the present invention aims to address the high detection latency and computational overhead of machine learning in malicious traffic detection. It proposes a window-based machine learning detection method that reduces detection latency and computational overhead, thereby making machine learning-based malicious traffic detection models easier to deploy in network devices. This technical solution is not only more efficient than traditional per-flow feature extraction methods, but also maintains the accuracy of machine learning algorithms for malicious traffic detection, offering the advantages of low overhead and low latency.

[0007] To achieve the above objectives, the present invention proposes a window-based machine learning detection method, comprising:

[0008] Get the network traffic data to be detected;

[0009] Calculating the network traffic data to be detected according to a window-based feature extraction method to obtain a feature vector of the first feature data, inputting the second feature data whose feature vector meets the preset conditions into the pre-trained traffic detection model to perform data type determination, and obtaining a data type determination result;

[0010] Malicious traffic data in the network traffic data to be detected this time is determined according to the data type determination result.

[0011] The window-based machine learning detection method according to an embodiment of the present invention may also have the following additional technical features:

[0012] Furthermore, in one embodiment of the present invention, the characteristic values of the network traffic data to be detected include: packet rate, byte rate, IPID average value, IPID standard deviation, TTL average value, TTL standard deviation, packet size average value, packet size standard deviation, source port number average value, source port number standard deviation, destination port number average value, destination port number standard deviation, TCP packet sequence number average value, TCP packet sequence number standard deviation, TCP packet response number average value, TCP packet response number standard deviation, TCP packet window average value and TCP packet window standard deviation data.

[0013] Furthermore, in one embodiment of the present invention, before the second feature data of the feature vector that meets the preset conditions is input into the pre-trained traffic detection model for data type determination, it also includes: obtaining the current sample data, wherein the current sample data includes normal traffic data and malicious traffic data with annotations; using a window-based feature extraction method to extract features from the feature values in the current sample data to obtain a training feature data set of the feature vector of the current sample data; from the training feature data set of the feature vector of the current sample data, the third feature data corresponding to the feature vector that meets the preset conditions is input into the current traffic detection model for model training and verification to obtain the pre-trained traffic detection model.

[0014] Furthermore, in one embodiment of the present invention, the feature extraction of the feature values in the current sample data using a window-based feature extraction method to obtain a training feature data set of the feature vector of the current sample data includes: sampling the current sample data in the time window using a preset sampling frequency to obtain multiple data packets, calculating the feature values in the current sample data from the multiple data packets to obtain fourth feature data; generating a sample feature vector for each sample feature data in the fourth feature data, and obtaining the current training feature data set based on all the sample feature vectors.

[0015] Furthermore, in one embodiment of the present invention, the method also includes: when performing malicious traffic data detection in the next network traffic data to be detected, updating the traffic detection model based on the updated sample data, and performing malicious traffic data detection according to the updated traffic detection model.

[0016] To achieve the above-mentioned object, the present invention further proposes a malicious traffic detection device based on window features, comprising:

[0017] The data acquisition module is used to obtain the network traffic data to be detected;

[0018] a data type determination module, configured to calculate the feature value of the network traffic data to be detected according to a window-based feature extraction method to obtain a feature vector of the first feature data, input the second feature data whose feature vector meets a preset condition into the pre-trained traffic detection model to perform data type determination, and obtain a data type determination result;

[0019] The traffic detection module is used to determine malicious traffic data in the network traffic data to be detected this time according to the data type determination result.

[0020] The window-based malicious traffic detection method and device of the present invention address the latency and overhead issues associated with machine learning in detecting DDoS malicious traffic. Compared to machine learning methods based on per-flow features, this method reduces detection latency and computational overhead while maintaining comparable accuracy. Furthermore, this method improves the overall efficiency of machine learning methods through window-based feature extraction. It is also compatible with a variety of machine learning algorithms, demonstrating excellent versatility and scalability.

[0021] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram of an application scenario according to an embodiment of the present invention;

[0024] Figure 2 Flowchart of a malicious traffic detection method based on window features according to an embodiment of the present invention;

[0025] Figure 3 Flowchart of flow detection model training according to an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of a window-based feature extraction method according to an embodiment of the present invention;

[0027] Figure 5 This is an architecture diagram of a malicious traffic detection method based on window features according to an embodiment of the present invention;

[0028] Figure 6 2 is a schematic structural diagram of a malicious traffic detection device based on window features according to an embodiment of the present invention;

[0029] Figure 7A schematic diagram of the structure of a model training module according to an embodiment of the present invention;

[0030] Figure 8 Schematic diagram of the structure of a malicious traffic detection device including a model updating module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0032] The following describes a malicious traffic detection method and apparatus based on window features according to an embodiment of the present invention with reference to the accompanying drawings.

[0033] Figure 1 This paper demonstrates the application scenarios of the present invention, including potential deployment locations and the distribution of attackers and victims within the network. In this application scenario, a DDoS attack can be initiated from anywhere on the internet. The attacker controls a zombie host to launch malicious DDoS traffic, impacting and disrupting the victim server's operation. The present method can be deployed at the victim server's entry point (e.g., a router or other intermediate device close to the server), achieving real-time detection through a window-based malicious traffic detection method.

[0034] Figure 2 This is a flow chart of a malicious traffic detection method based on window features according to an embodiment of the present invention.

[0035] like Figure 2 As shown, the method includes the following steps:

[0036] Step S1, obtaining the network traffic data to be detected this time.

[0037] Specifically, first determine N characteristic attributes related to DDoS malicious traffic, including but not limited to the following characteristic attributes: packet rate, byte rate, IPID average value, IPID standard deviation, TTL average value, TTL standard deviation, packet size average value, packet size standard deviation, source port number average value, source port number standard deviation, destination port number average value, destination port number standard deviation, TCP packet sequence number average value, TCP packet sequence number standard deviation, TCP packet reply number average value, TCP packet reply number standard deviation, TCP packet window average value, TCP packet window standard deviation.

[0038] The N characteristic attributes of the network traffic data to be detected are used for subsequent characteristic value calculation.

[0039] Step S2, calculate the characteristic value of the network traffic data to be detected according to the window-based feature extraction method to obtain the characteristic vector of the first characteristic data, input the second characteristic data whose characteristic vector meets the preset conditions into the pre-trained traffic detection model for data type determination, and obtain the data type determination result.

[0040] It is understandable that before the traffic detection model determines the data type, the model needs to be trained using sample data.

[0041] Figure 3 The flow chart of the flow detection model training provided by the embodiment of the present invention is as follows: Figure 3 As shown, in one embodiment, the training step includes the following sub-steps:

[0042] S21, obtaining current sample data, wherein the current sample data includes annotated normal traffic data and malicious traffic data;

[0043] S22, extracting features from the feature values in the current sample data using a window-based feature extraction method to obtain a training feature data set of feature vectors of the current sample data;

[0044] S23, from the training feature data set of the feature vector of this sample data, the third feature data corresponding to the feature vector that meets the preset conditions is input into the current flow detection model for model training and verification to obtain the pre-trained flow detection model.

[0045] Specifically, the training data includes labeled normal traffic data and DDoS malicious traffic data. A window-based feature extraction method is used, that is, W packets are regularly sampled from the traffic within a fixed time window (e.g., within T seconds), and the feature values listed in step S1 are calculated for the packets sampled within the window.

[0046] As an example, Figure 4 This is the details of the window-based feature extraction method. The feature extraction window is set to 1 second. W packets are sampled from the N packets within the window. The required feature values are calculated from these W packets to obtain the feature vector FV of one sample.

[0047] Furthermore, the features extracted from the sampled data packets in each window generate a sample, and each sample corresponds to a feature vector v = (FV1, ..., FV N ). The training feature data set Ω={v1,…,v n}, n is the number of feature vectors contained in the training data set.

[0048] Furthermore, a Chi-square test algorithm is used to screen out important features from the feature data extracted in the above steps, which are the third feature data corresponding to the feature vector in the training feature data set.

[0049] Furthermore, the traffic detection model is trained using the third feature data corresponding to the feature vectors in the training feature dataset. There are no specific restrictions on the algorithm used in training; classic machine learning algorithms such as decision trees, random forests, multi-layer perceptrons, and naive Bayes can all be used in conjunction with window-based features. Furthermore, a ten-fold cross-validation method is used for verification, and the resulting model is used for subsequent online testing.

[0050] When performing actual traffic detection, the second feature data corresponding to the above feature vector is input into the pre-trained traffic detection model to determine the data type and obtain the data type determination result. The determination result may include two types: "malicious" and "normal".

[0051] Step S3: Determine malicious traffic data in the network traffic data to be detected according to the data type determination result.

[0052] Specifically, when the determination result is that the data is of two types, "malicious" and "normal", malicious traffic in the network traffic data to be detected can be detected.

[0053] Furthermore, when performing malicious traffic data detection in the next network traffic data to be detected, the traffic detection model is updated based on the updated sample data, and malicious traffic data detection is performed according to the updated traffic detection model.

[0054] It is understood that the model of the embodiment of the present invention is updated as needed. When a new DDoS attack type needs to be detected, the above-mentioned model training steps can be re-executed to obtain a new training model, which can then be used in subsequent online detection, thereby achieving dynamic model updates.

[0055] Figure 5 The following is an architectural diagram of the malicious traffic detection method based on window features, illustrating the training and detection workflow described above. During the training phase, the window-based feature extraction step extracts all the features listed in step S1, followed by feature selection. During the detection phase, the feature extraction step only extracts the features selected during the feature selection step. Furthermore, the model update is completed by simply replacing the old model used during the detection phase with the trained new model.

[0056] In summary, the basic concept of the present invention is that the characteristics of DDoS malicious traffic can be directly observed at the overall traffic level without distinguishing each specific flow according to the five-tuple. Therefore, treating the traffic as a whole and directly extracting features can also realize the detection of DDoS malicious traffic, while achieving less overhead and delay than the five-tuple-based flow feature extraction, thereby reducing the overall overhead and delay of the machine learning detection method. Specifically, the window-based feature extraction method treats all the data packets in the traffic as a whole, samples a fixed number of data packets in each time window to calculate the feature values, and combines the feature values calculated for each window into a training sample. During online detection, the same window-based method is used to extract features, and the trained model is used to detect new samples. Since the window-based feature extraction module is used in both the training and detection stages, this method can effectively reduce the overhead and delay of machine learning training and detection. Because DDoS attacks significantly alter the statistical characteristics of traffic overall (e.g., a dramatic increase in packet transmission rates and a rapid increase in traffic), these changes can be effectively captured and identified using window-based feature extraction methods. Therefore, this method reduces latency and overhead while also maintaining a high level of accuracy in machine learning detection of malicious DDoS traffic. It is understood that the method proposed in this invention can be used in conjunction with a variety of machine learning algorithms. By combining these algorithms with the present invention, the overall effectiveness of machine learning models in detecting malicious DDoS traffic can be enhanced.

[0057] The window-based malicious traffic detection method according to an embodiment of the present invention addresses the latency and overhead issues associated with machine learning in detecting DDoS malicious traffic. It reduces detection latency and computational overhead compared to per-flow feature-based machine learning methods, while maintaining comparable accuracy. Furthermore, this method improves the overall efficiency of machine learning methods through window-based feature extraction. It is also compatible with a variety of machine learning algorithms, demonstrating excellent versatility and scalability.

[0058] Next, a malicious traffic detection device based on window features according to an embodiment of the present invention will be described with reference to the accompanying drawings.

[0059] Figure 6 It is a structural diagram of a malicious traffic detection device based on window features according to an embodiment of the present invention.

[0060] like Figure 6 As shown, the device 10 includes: a data acquisition module 100, a data type determination module 200 and a flow detection module 300.

[0061] The data acquisition module 100 is used to obtain the network traffic data to be detected;

[0062] The data type determination module 200 is configured to calculate the feature value of the network traffic data to be detected according to a window-based feature extraction method to obtain a feature vector of the first feature data, input the second feature data whose feature vector meets the preset conditions into the pre-trained traffic detection model to perform data type determination, and obtain a data type determination result;

[0063] The traffic detection module 300 is used to determine malicious traffic data in the network traffic data to be detected according to the data type determination result.

[0064] Furthermore, the characteristic values in the data acquisition module 100 include the following data:

[0065] Packet rate, byte rate, average IPID, standard deviation of IPID, average TTL, standard deviation of TTL, average packet size, standard deviation of packet size, average source port number, standard deviation of source port number, average destination port number, standard deviation of destination port number, average TCP packet sequence number, standard deviation of TCP packet sequence number, average TCP packet acknowledgment number, standard deviation of TCP packet acknowledgment number, average TCP packet window, and standard deviation of TCP packet window.

[0066] Furthermore, if Figure 7 As shown, a model training module 101 is further included before the data type determination module 200, including:

[0067] The training data subunit 1011 is used to obtain the current sample data, wherein the current sample data includes normal traffic data and malicious traffic data with annotations;

[0068] The feature extraction subunit 1012 is used to extract the feature values in the current sample data using a window-based feature extraction method to obtain a training feature data set of the feature vectors of the current sample data;

[0069] The training and verification subunit 1013 is used to input the third feature data corresponding to the feature vector that meets the preset conditions from the training feature data set of the feature vector of this sample data into the current flow detection model for model training and verification to obtain the pre-trained flow detection model.

[0070] Furthermore, the feature extraction subunit 1012 is further configured to:

[0071] Sampling the current sample data in the time window using a preset sampling frequency to obtain a plurality of data packets, and calculating feature values in the current sample data from the plurality of data packets to obtain fourth feature data;

[0072] A sample feature vector is generated for each sample feature data in the fourth feature data, and the feature data set for this training is obtained based on all the sample feature vectors.

[0073] Further, if Figure 8 As shown, the above-mentioned device 10 also includes a model updating module 400, which is used to update the traffic detection model based on the updated sample data when performing malicious traffic data detection in the next network traffic data to be detected, and perform malicious traffic data detection according to the updated traffic detection model.

[0074] The window-based malicious traffic detection device according to an embodiment of the present invention addresses the latency and overhead issues associated with machine learning in detecting malicious DDoS traffic. It reduces detection latency and computational overhead compared to per-flow feature-based machine learning methods, while maintaining comparable accuracy. Furthermore, this method, through its window-based feature extraction approach, improves the overall efficiency of machine learning methods and is compatible with a variety of machine learning algorithms, demonstrating excellent versatility and scalability.

[0075] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0076] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0077] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A malicious traffic detection method based on window features, characterized in that: The following steps are involved: Obtaining network traffic data to be detected with N characteristic attributes related to malicious DDoS traffic; calculating characteristic values of the network traffic data to be detected according to a window-based feature extraction method to obtain a characteristic vector of first characteristic data; inputting second characteristic data whose characteristic vector meets preset conditions into the pre-trained traffic detection model to perform data type determination, and obtaining a data type determination result; Determine the DDoS malicious traffic data in the network traffic data to be detected according to the data type determination result; Calculating the characteristic value of the network traffic data to be detected according to the window-based feature extraction method to obtain the characteristic vector of the first characteristic data; including: Set the feature extraction window to 1 second, sample W data packets from the N data packets within the feature extraction window, and calculate the required feature values from these W data packets to obtain a feature vector FV of a sample; wherein, the features extracted from the sampled data packets in each window generate a sample.

2. The method according to claim 1, characterized in that The characteristic values of the network traffic data to be detected include: Packet rate, byte rate, IPID average, IPID standard deviation, TTL average, TTL standard deviation, packet size average, packet size standard deviation, source port number average, source port number standard deviation, destination port number average, destination port number standard deviation, TCP packet sequence number average, TCP packet sequence number standard deviation, TCP packet acknowledgment number average, TCP packet acknowledgment number standard deviation, TCP packet window average and TCP packet window standard deviation data.

3. The method according to claim 2, characterized in that Before inputting the second feature data whose feature vector meets the preset conditions into the pre-trained traffic detection model for data type determination, the method further includes: Obtaining current sample data, wherein the current sample data includes annotated normal traffic data and malicious traffic data; Extracting features from the eigenvalues of the sample data using a window-based feature extraction method to obtain a training feature data set of the eigenvectors of the sample data; From the training feature data set of the feature vector of the sample data, the third feature data corresponding to the feature vector that meets the preset conditions is input into the flow detection model for model training and verification to obtain the pre-trained flow detection model.

4. The method according to claim 3, characterized in that The feature extraction method based on the window is used to extract the feature values in the current sample data to obtain the training feature data set of the feature vector of the current sample data, including: Sampling the current sample data in the time window using a preset sampling frequency to obtain a plurality of data packets, and calculating feature values in the current sample data from the plurality of data packets to obtain fourth feature data; A sample feature vector is generated for each sample feature data in the fourth feature data, and the current training feature data set is obtained based on all the sample feature vectors.

5. The method according to claim 3, characterized in that The method further comprises: When performing malicious traffic data detection in the network traffic data to be detected next time, the traffic detection model is updated based on the updated sample data, and malicious traffic data detection is performed according to the updated traffic detection model.

6. A malicious traffic detection device based on window features, characterized in that: The following steps are involved: A data acquisition module is used to obtain the network traffic data to be detected with N characteristic attributes related to the DDoS malicious traffic; A data type determination module is configured to calculate the characteristic values of the network traffic data to be detected according to a window-based feature extraction method to obtain a characteristic vector of the first characteristic data, and input the second characteristic data whose characteristic vector satisfies a preset condition into the pre-trained traffic detection model to perform data type determination to obtain a data type determination result; wherein, the characteristic values of the network traffic data to be detected are calculated according to the window-based feature extraction method to obtain the characteristic vector of the first characteristic data; and the module comprises: setting a feature extraction window to 1 second, sampling W data packets from N data packets within the feature extraction window, and calculating the required characteristic values from the W data packets to obtain a characteristic vector FV of a sample; wherein, the features extracted from the sampled data packets within each window generate a sample; The traffic detection module is used to determine the DDoS malicious traffic data in the network traffic data to be detected this time according to the data type determination result.

7. The device according to claim 6, characterized in that The characteristic values in the data acquisition module include: Packet rate, byte rate, IPID average, IPID standard deviation, TTL average, TTL standard deviation, packet size average, packet size standard deviation, source port number average, source port number standard deviation, destination port number average, destination port number standard deviation, TCP packet sequence number average, TCP packet sequence number standard deviation, TCP packet acknowledgment number average, TCP packet acknowledgment number standard deviation, TCP packet window average and TCP packet window standard deviation data.

8. The device according to claim 7, characterized in that Before the data type determination module, a model training module is also included, including: The training data subunit is used to obtain the current sample data, wherein the current sample data includes annotated normal traffic data and malicious traffic data; A feature extraction subunit is used to extract the feature values in the current sample data using a window-based feature extraction method to obtain a training feature data set of the feature vectors of the current sample data; The training and verification subunit is used to input the third feature data corresponding to the feature vector that meets the preset conditions from the training feature data set of the feature vector of the sample data into the current flow detection model for model training and verification to obtain the pre-trained flow detection model.

9. The device according to claim 8, characterized in that The feature extraction subunit is further configured to: Sampling the current sample data in the time window using a preset sampling frequency to obtain a plurality of data packets, and calculating feature values in the current sample data from the plurality of data packets to obtain fourth feature data; A sample feature vector is generated for each sample feature data in the fourth feature data, and the current training feature data set is obtained based on all the sample feature vectors.

10. The device according to claim 8, characterized in that The device also includes a model updating module, The model updating module is used to update the traffic detection model based on the updated sample data when performing malicious traffic data detection in the next network traffic data to be detected, and perform malicious traffic data detection according to the updated traffic detection model.

Images