Data packet filtering method and device, equipment and medium

Abstract

The invention relates to the technical field of network security, in particular to a data packet filtering method and device, equipment and a medium. In the embodiment of the invention, the arrival time pattern of the data packet is analyzed to determine the frequency and persistence corresponding to the target identifier of the data packet, so that complex flow aggregation or feature extraction is not needed; the data packets belonging to the DDoS attack traffic are screened out according to the activation function obtained by training the historical frequency count and the historical persistence count of the historical data packets and the historical classification information of the data packets on the basis of whether the historical data packets are the DDoS attack data packets, and a preset threshold value, so that the DDoS attack intensity is greatly reduced; and normal service flow is protected from being affected by attacks.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a data packet filtering method, device, equipment and medium. Background Art

[0002] With the rapid development of the Internet, Distributed Denial of Service (DDoS) attacks have become a common network security threat. They cause service interruptions by flooding the target website or service with massive requests, making it unable to process requests from legitimate users.

[0003] Therefore, it is necessary to detect DDoS attacks to improve the accuracy and real-time performance of DDoS attack detection, thereby improving the ability to defend against DDoS attacks, helping network administrators to promptly discover abnormal traffic patterns, quickly take defensive measures, and minimize the negative impact of attacks. Summary of the invention

[0004] The present application provides a data packet filtering method, apparatus, device and medium to solve the problem that existing DDoS attacks cause target websites or services to be overwhelmed by massive requests and unable to process requests from legitimate users, thereby causing service interruptions.

[0005] In a first aspect, an embodiment of the present application provides a method for filtering a data packet, the method comprising:

[0006] Determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet;

[0007] Obtaining a frequency count of the data packets of the target identifier appearing before the current time of monitoring statistics, and a persistence count of the data packets of the target identifier appearing within a set time window;

[0008] According to the frequency count, the persistence count, the activation function and a preset threshold, determine whether the target data packet is the first classification information of a data packet of a DDoS attack; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

[0009] In a second aspect, an embodiment of the present application further provides a data packet filtering device, the device comprising:

[0010] A processing module is used to determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; obtain the frequency count of the data packet with the target identifier before the current time of the monitoring statistics, and the persistence count of the data packet with the target identifier appearing within a set time window;

[0011] A classification module is used to determine the first classification information of whether the target data packet is a data packet of a DDoS attack based on the frequency count, the persistence count, the activation function and a preset threshold; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

[0012] In a third aspect, an embodiment of the present application further provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned data packet filtering methods when executing a computer program stored in the memory.

[0013] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the steps of any of the above-mentioned data packet filtering methods when executed by a processor.

[0014] In an embodiment of the present application, the electronic device determines the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; obtains the frequency count of the data packet with the target identifier before the current time of the monitoring statistics, and the persistence count of the data packet with the target identifier appearing in the set time window; determines whether the target data packet is the first classification information of the data packet of the DDoS attack according to the frequency count, persistence count, activation function and preset threshold; wherein the activation function is trained based on the historical frequency count of the historical data packet, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of the DDoS attack. In an embodiment of the present application, by analyzing the time pattern of the arrival of the data packet, the frequency and persistence corresponding to the target identifier of the data packet are determined, without complex flow aggregation or feature extraction, and according to the activation function trained based on the historical frequency count of the historical data packet, the historical persistence count and the historical classification information of whether the historical data packet is a data packet of the DDoS attack and the preset threshold, the data packet belonging to the DDoS attack traffic is screened out, which greatly reduces the intensity of the DDoS attack and protects the normal business traffic from the attack. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A schematic diagram of a data packet filtering process provided in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of a DDoS attack detection process combining online detection and offline detection provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of the filtering rule generation and classification model training process provided in the embodiment of the present application;

[0019] Figure 4 A schematic diagram of the structure of a data packet filtering device provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0022] The existing detection method for DDoS attacks is to use machine learning and data mining technology to collect and analyze the multi-dimensional attributes in network traffic, so as to ensure efficient and accurate real-time detection of large-volume DDoS attacks on the Internet. For example, machine learning models such as decision trees, random forests, and XGBoost are used, and performance is optimized by adjusting parameters, and finally the detection accuracy is improved through model fusion. However, with the continuous changes in DDoS attack methods, traditional detection methods based on static feature rules (such as traffic size, packet rate, protocol type, etc.) can no longer meet the requirements. Modern DDoS attack detection requires smarter and more efficient technologies to cope with the ever-changing attack patterns. The development of machine learning and data mining technology provides new possibilities for real-time traffic detection. By analyzing the behavioral characteristics of network traffic, abnormal patterns that are different from normal traffic can be automatically identified, thereby realizing real-time detection of DDoS attacks. The latest machine learning-based attack detection algorithms can already achieve detection with a precision of more than 99.99% at the flow level, and their classification performance is quite outstanding. However, most of them can only be detected after the attack flow is completed, and most online models have single rules. Although the time performance is sufficient to detect and deal with the attack flow in progress, the classification performance is limited.

[0023] In order to accurately identify DDoS attacks and improve network security, the embodiments of the present application provide a data packet filtering method, apparatus, device and medium.

[0024] In an embodiment of the present application, the electronic device determines the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; obtains the frequency count of the data packet with the target identifier before the current time of the monitoring statistics, and the persistence count of the data packet with the target identifier within the set time window; determines whether the target data packet is the first classification information of the data packet of the DDoS attack based on the frequency count, the persistence count, the activation function and the preset threshold; wherein the activation function is trained based on the historical frequency count of the historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is the data packet of the DDoS attack.

[0025] Embodiment 1:

[0026] Figure 1 A schematic diagram of a packet filtering process provided in an embodiment of the present application, the process includes:

[0027] S101: Determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; obtain the frequency count of the data packet with the target identifier before the current time of the monitoring statistics, and the persistence count of the data packet with the target identifier within the set time window.

[0028] A data packet filtering method provided in an embodiment of the present application is applied to an electronic device, which may be a PC or a server.

[0029] After investigation, it is found that the data flow statistical summary method based on Sketch has been quite popular in recent years. The Sketch method uses limited memory and can accelerate the measurement of data flow. By using Sketch to evaluate the statistical indicators in the data flow, a rough classification can be performed. As a kind of Sketch method, LTC (Long-TailClock) only analyzes the time pattern of data packet arrival in network traffic, without the need for time-consuming traffic aggregation and statistical feature extraction, and can complete the evaluation of the frequency and persistence of element arrival. Such processing is reasonable for the detection of DDoS attacks, because DDoS attack traffic is not only very frequent in a short period of time (high frequency), but also this high-frequency traffic continues to appear in multiple time periods (high persistence).

[0030] Based on this, compared to the existing methods that mostly use traditional counter methods such as Bloom Sketch and CountingBloomFilter, which only consider frequency, the embodiment of the present application can simultaneously measure the frequency and persistence of elements, and can more flexibly define and filter more complex DDoS attack patterns. In the embodiment of the present application, the electronic device can design an improved algorithm based on the LTC capability, and based on the distribution pattern of the arrival time of the data packet, it can achieve coarse-grained online screening of DDoS attack traffic at the data packet level, greatly reducing the proportion of DDoS attack traffic.

[0031] Specifically, for each data packet in the traffic, extract the header quintuple (source address, destination address, source port, destination port and protocol information) as the identifier of the data packet, denoted as f ID , where the protocol information is the L4 protocol.

[0032] For each identifier, the electronic device uses the LTC algorithm to monitor and count the frequency count of the data packets of the target identifier, as well as the persistence count of the data packets of the target identifier within the set time window. The frequency count is the frequency count (frequency) performed by the LTC algorithm each time the data packets of the target identifier arrive by combining the long-tail recovery technology and the improved CLOCK algorithm, and the persistence count is the persistence count (persistence) according to a certain observation period.

[0033] When an electronic device identifies a target data packet to be processed, the electronic device determines the source address, destination address, source port, destination port and protocol information carried in the target data packet as the target identifier of the target data packet, and based on the search algorithm of the LCT algorithm, searches for the frequency count of the occurrence of data packets with the target identifier before the current time, and the persistence count of the occurrence of data packets with the target identifier within the set time window.

[0034] Among them, the LTC algorithm is an advanced pattern mining algorithm for streaming data, specially designed to quickly and accurately identify significant items in data streams under memory constraints. For network traffic, the only feature considered and used by the LTC algorithm is the arrival time interval of the data packets, without the need for complex statistical work such as flow aggregation or feature extraction. This is of great significance for early detection in high-speed traffic. The high throughput and memory efficiency make it an ideal choice for preliminary screening of large amounts of network traffic.

[0035] S102: Determine the first classification information of whether the target data packet is a data packet of a DDoS attack based on the frequency count, the persistence count, the activation function and a preset threshold; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

[0036] The original LTC algorithm aims to extract the top N most important elements from the flow, where importance is the weighted sum of frequency count and persistence count. According to the pattern of DDoS attacks, DDoS attack traffic can be screened out by setting an importance threshold after giving equal weights to frequency and persistence. However, when dealing with complex DDoS attack patterns, the simple weighted sum method is not enough to accurately screen out attack traffic. For example, some attack traffic may show abnormally high levels of persistence even if its frequency is not outstanding. Similarly, some attacks may suddenly increase their frequency within a specific time period, including but not limited to the burstiness, intermittency of traffic, and the ability to disguise as normal traffic.

[0037] Based on this, in order to improve the accuracy and robustness of detection, it is necessary to design more complex filtering rules FP-Rule based on frequency and persistence, filter out most of the DDoS attack traffic that meets the characteristic rules, and only retain benign traffic and a small part of DDoS attack traffic, so that the original traffic is filtered into traffic data with lower imbalance. Among them, the electronic device can use a fitting algorithm based on logistic regression to derive the filtering rule FP-Rule. FP-Rule is implemented as a function f(f,p) (whose return value is 0 or 1) with f and p as input to determine whether the target data packet is a DDoS attack. If the return value is 1, it is determined that the target data packet is a DDoS attack. If the return value is 0, it is determined that the target data packet is not a DDoS attack.

[0038] Specifically, in an embodiment of the present application, the electronic device obtains an activation function through training based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packets are data packets of DDoS attacks. The electronic device uses the trained activation function and a preset threshold to construct a filtering rule, and filters the target data packet based on the filtering rule.

[0039] Among them, in the embodiment of the present application, the filtering rules derived by the electronic device include two parts. One is to determine the target probability of the target data packet being a data packet of a DDoS attack based on the frequency count, persistence count and trained activation function of the target data packet; the other part is to determine the first classification information of whether the target data packet is a data packet of a DDoS attack based on the target probability and a preset threshold.

[0040] Among them, the activation function can map a real number to the interval (0,1), which can be used for binary classification. It works better when the feature differences are complex or not very large.

[0041] Specifically, after the electronic device obtains the frequency count corresponding to the target data packet, which is used to represent the frequency, and the persistence count which is used to represent the persistence, the electronic device can determine the target probability that the target data packet is a data packet of a DDoS attack based on the frequency count and the persistence count, as well as the trained activation function.

[0042] The electronic device is also pre-configured with a threshold value, which is adjusted according to the detected Precision value to ensure a low false negative rate. The electronic device can determine the first classification information of whether the target data packet is a data packet of a DDoS attack based on the threshold and the target probability. If the target probability exceeds the threshold, the first classification information determined is the first classification information that the target data packet is a data packet of a DDoS attack; if the target probability does not exceed the threshold, the first classification information determined is the first classification information that the target data packet is not a data packet of a DDoS attack. This method is relatively "loose" during the fitting process and can filter out most of the DDoS attack traffic without erroneously intercepting benign traffic.

[0043] Specifically, the electronic device can determine the target probability of whether the target data packet is a data packet of a DDoS attack by using the following formula:

[0044] P(label=1|x)=σ(w T x+b),x=[f,p]

[0045] Among them, σ(z) represents the activation function, x is the feature vector composed of the frequency count f and the persistence count p, w and b are the parameters obtained through training, among which w is the weight vector and b is the bias term.

[0046] The electronic device can determine whether the target data packet is the first classification information of the data packet of the DDoS attack by the following formula:

[0047]

[0048] Among them, σ(w T x+b) is the target probability, and θ is the preset threshold.

[0049] Based on the above embodiment, the first classification information of the electronic device to determine whether the target data packet is a data packet of a DDoS attack is 1 or 0. If the electronic device determines that the first classification information is 1, the electronic device determines that the target data packet is a DDoS attack; if the electronic device determines that the first classification information is 0, the electronic device determines that the target data packet is not a DDoS attack.

[0050] In addition, in the embodiment of the present application, the method for generating FP-Rule in the embodiment of the present application includes but is not limited to logistic regression, linear fitting, density estimation and other methods of decision rules that can dynamically adjust the classification plane. The activation function can be a sigmoid function or other functions, which are not limited here.

[0051] In an embodiment of the present application, by analyzing the time pattern of the arrival of the data packet, the frequency count and persistence count corresponding to the target identifier of the data packet are determined without the need for complex flow aggregation or feature extraction, and the data packets belonging to the DDoS attack traffic are screened out according to the activation function trained based on the historical frequency counts of historical data packets, the historical persistence counts and the historical classification information of whether the historical data packets are DDoS attack data packets and the preset threshold, thereby greatly reducing the intensity of the DDoS attack and protecting normal business traffic from the impact of the attack.

[0052] Embodiment 2:

[0053] In order to accurately identify DDoS attacks and improve network security, based on the above embodiments, in the embodiment of the present application, before obtaining the frequency count of the occurrence of the data packet of the target identifier before the current time of the monitoring statistics, the method further includes:

[0054] Obtaining a configured data packet whitelist, wherein the data packet whitelist stores identifiers of data packets that have been determined not to be DDoS attacks;

[0055] If the target identifier is not included in the data packet whitelist, a subsequent process of obtaining a frequency count of occurrences of data packets with the target identifier before the current time of monitoring statistics is performed.

[0056] In order to improve the efficiency of DDoS attack detection, in an embodiment of the present application, a data packet whitelist is configured in the electronic device, and the data packet whitelist stores the identification of data packets that have been determined to be not DDoS attacks. The data packet whitelist is configured based on historical DDoS attack detection results.

[0057] The data protection whitelist can quickly identify and allow data packets with history and prior knowledge to pass. Based on this, before the electronic device filters the target data packet based on the filtering rules, the electronic device obtains the configured data packet whitelist. If the electronic device determines that the data packet whitelist does not contain the target identifier, it continues to filter the target data packet; if the electronic device determines that the data packet whitelist contains the target identifier, it processes the target data packet.

[0058] Among them, in the embodiment of the present application, the implementation method of the data packet whitelist includes but is not limited to various types of hash tables, sequence tables, red-black trees, Bloom Filters and other data structures and algorithms that support fast member search.

[0059] Embodiment 3:

[0060] In order to accurately identify DDoS attacks and improve network security, based on the above embodiments, in the embodiment of the present application, if it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes:

[0061] Acquire characteristic data of a target flow corresponding to the target data packet, wherein the characteristic data includes flow duration, a length standard deviation of each data packet contained in the target flow, and a total number of data packets transmitted per second by the target flow;

[0062] Inputting the feature data into a pre-trained classifier, and obtaining second classification information output by the classifier indicating whether the target flow is a DDoS attack;

[0063] If the second classification information indicates that the target flow is a DDoS attack, the target data packet is determined to be a data packet of the DDoS attack.

[0064] In order to further improve the accuracy of DDoS attack detection, in an embodiment of the present application, the electronic device can also use a machine learning classifier to perform flow-level detection on the traffic after reducing the imbalance rate between DDoS attacks and normal business traffic, to complete further accurate judgment.

[0065] Specifically, the electronic device obtains characteristic data of a target flow corresponding to a target data packet, wherein the characteristic data includes flow duration, a standard deviation of the length of each data packet contained in the target flow, and a total number of data packets transmitted per second by the target flow. Based on the characteristic data, the electronic device constructs a feature vector corresponding to the target flow, and inputs the feature vector into a pre-trained classifier to obtain second classification information output by the classifier as to whether the target flow is a DDoS attack.

[0066] If the electronic device determines that the second classification information output by the classifier is that the target flow is a DDoS attack, the electronic device determines that the target data packet is a data packet of the DDoS attack.

[0067] It should be noted that in the embodiment of the present application, the second classification information output by the classifier is 1 or 0. If the second classification information is 1, it is determined that the target flow is a DDoS attack. If the second classification information is 0, it is determined that the target flow is not a DDoS attack.

[0068] On the basis of the above-mentioned embodiments, the packet filtering method provided in the embodiment of the present application is divided into two parts: online monitoring and offline detection. Among them, online detection is realized by packet whitelist matching and filtering rule screening, which belongs to the packet level; offline detection is realized by a classifier, which belongs to the data flow level. The embodiment of the present application adopts a hierarchical method combining online detection and offline detection. On the one hand, based on the LTC capability, an improved algorithm is designed to achieve coarse-grained online screening of DDoS attack traffic at the packet level based on the distribution pattern of packet arrival time, which greatly reduces the proportion of DDoS attack traffic; on the other hand, a machine learning classifier is used in the second layer to perform flow-level detection on the traffic after reducing the imbalance rate of DDoS attacks and normal business traffic, so as to complete further accurate discrimination. Through the embodiment of the present application, it is possible to achieve the weakening of the intensity of DDoS attacks on devices within the protection detection range after a DDoS attack occurs, and protect normal business traffic from being affected by the attack.

[0069] In order to accurately identify DDoS attacks and improve network security, based on the above embodiments, in the embodiment of the present application, the classifier is a CatBoost classifier.

[0070] In the embodiment of the present application, the classifier used by the electronic device may be a CatBoost classifier.

[0071] It should be noted that the classifiers used in the embodiments of the present application include but are not limited to CatBoost, nearest neighbor, random forest and other machine learning-based methods. The use of CatBoost in the embodiments of the present application is based on the principle of optimal experimental effect.

[0072] Figure 2 A schematic diagram of a DDoS attack detection process combining online detection and offline detection provided in an embodiment of the present application, such as the Figure 2 As shown, the process includes:

[0073] 1. Model loading and preparation. Run the program, load and generate the filtering rule FP-Rule and CatBoost classifier, and start listening to the original network traffic from the specified port / queue / api.

[0074] 2. Data packet whitelist (whitelist) matching. The whitelist can quickly identify and allow data packets with history and prior knowledge to pass. If the target identifier of the data packet is f ID If it is on the whitelist, it will return benign directly.

[0075] It should be emphasized that the implementation of the whitelist includes but is not limited to various hash tables, sequence tables, red-black trees, BloomFilter and other data structures and algorithms that support fast member search.

[0076] 3. FP-Rule filtering. If the f II If it is not found on the whitelist, LTC is used to calculate f ID The corresponding frequency count (Frequency) and persistence count (Persistence) are filtered using FP-Rule. If FP-Rule returns a DDoS attack, the packet is filtered out, otherwise the packet is added to the buffer to be classified.

[0077] 4. CatBoost classification. When the data packets in the buffer to be classified meet the conditions (the conditions are freely defined according to the needs, including but not limited to the number of packets, the refresh time, the overall size, etc.), the buffer data packets are aggregated and the traffic features are extracted as items. The CatBoost classifier is used for judgment. If it is not a DDoS attack, it is recorded in the whitelist, otherwise it is output as a DDoS attack.

[0078] Embodiment 4:

[0079] In order to accurately identify DDoS attacks and improve network security, based on the above embodiments, in the embodiment of the present application, the training process of the CatBoost classifier includes:

[0080] Obtaining a pre-configured sample flow, sample feature data corresponding to the sample data packet, and sample second classification information of whether the sample flow is a DDoS attack;

[0081] Inputting the sample feature data into the CatBoost classifier to be trained, and obtaining the predicted second classification information output by the CatBoost classifier;

[0082] A loss value is determined according to the sample second classification information, the predicted second classification information and the F1-score algorithm, and the parameters of the CatBoost classifier are adjusted according to the loss value.

[0083] In an embodiment of the present application, the electronic device can train the CatBoost classifier based on historical traffic data to improve the classification accuracy of the CatBoost classifier.

[0084] Specifically, the electronic device obtains a pre-configured sample stream, sample feature data corresponding to a sample data packet, and sample second classification information of whether the sample stream is a DDoS attack; the electronic device inputs the sample feature data into a CatBoost classifier to be trained, and obtains predicted second classification information output by the CatBoost classifier; the electronic device determines a loss value based on the sample second classification information, the predicted second classification information, and an F1-score algorithm, and adjusts the parameters of the CatBoost classifier based on the loss value.

[0085] Among them, the electronic device can determine the loss value based on the following formula:

[0086]

[0087] Among them, TP represents the number of sample streams whose corresponding predicted second classification information and sample second classification information are both 1; FP represents the number of sample streams whose corresponding predicted second classification information is 1 and whose corresponding sample second classification information is 0; FN represents the number of sample streams whose corresponding predicted second classification information is 0 and whose corresponding sample second classification information is 1.

[0088] Embodiment 5:

[0089] In order to accurately identify DDoS attacks and improve network security, based on the above embodiments, in the embodiment of the present application, if it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes:

[0090] The target identifier is added to the data packet whitelist.

[0091] In an embodiment of the present application, if the electronic device determines that the target data packet is not a data packet of a DDoS attack, the electronic device adds the target identifier corresponding to the target data packet to the data packet whitelist, so that when other data packets with the target identifier are subsequently received, the other data packets are directly passed.

[0092] Embodiment 6:

[0093] In order to accurately identify DDoS attacks and improve network security, based on the above embodiments, in the embodiment of the present application, the training process of the activation function includes:

[0094] Obtaining a pre-configured sample data packet, a sample frequency count corresponding to the sample data packet, a sample persistence count, and sample first classification information of whether the sample data packet is a data packet of a DDoS attack;

[0095] Inputting the sample frequency count and the sample persistence count into the activation function to be trained, obtaining the predicted probability based on the output of the activation function and the preset threshold, and obtaining the predicted first classification information of whether the sample data packet is a data packet of a DDoS attack determined based on the predicted probability output of the activation function and the preset threshold;

[0096] A loss value is determined according to the sample first classification information, the predicted first classification information, and the maximized log-likelihood function, and the parameters of the activation function are adjusted according to the loss value.

[0097] In an embodiment of the present application, when an electronic device obtains an activation function through training based on a historical frequency count of a historical data packet, a historical persistence count, and historical first classification information of whether the historical data packet is a data packet of a DDoS attack, the electronic device may first obtain a pre-configured sample data packet, a sample frequency count, a sample persistence count, a threshold, and sample first classification information of whether the sample data packet is a data packet of a DDoS attack corresponding to the sample data packet; the electronic device inputs the sample frequency count and the sample persistence count into the activation function to be trained, obtains a predicted probability of whether the sample data packet is a data packet of a DDoS attack determined based on the activation function, and determines the predicted first classification information based on the predicted probability and the threshold; the electronic device determines a loss value based on the sample first classification information, the predicted first classification information, and the maximized log-likelihood function, and adjusts the parameters of the activation function based on the loss value.

[0098] Among them, when a DDoS attack occurs, the traffic is mainly attack traffic, and only a small part is benign traffic. According to this feature, the electronic device uses a simulation platform or a public network data set to obtain a network traffic pcap file containing DDoS attacks and normal behaviors and a label file labels.file. The electronic device obtains each sample data packet from the network traffic pcap file and determines the sample first classification information of whether each data packet corresponds to a data packet of a DDoS attack according to the label file labels.file.

[0099] The electronic device replays the sample data packets according to the arrival time of the sample data packets, and uses LTC to measure, to obtain a frequency count indicating frequency and a persistence count indicating persistence corresponding to the target identifier of each sample data packet.

[0100] Due to the principle of LTC, in the initial stage, all counters of LTC are 0, which has no discrimination. The output results have no practical meaning and cannot be used to generate filtering rules. In order to solve this "cold start" problem, the embodiment of the present application is designed to start recording when a certain number (the present invention is set to 100,000 based on experience) of data packets pass through LTC. At this time, the counters will not be all zero, and these are regarded as valid fitting samples. After the cold start wait, when the marked traffic passes through LTC, the frequency count f representing the frequency and the persistence count p representing the persistence corresponding to each data packet are determined, and the label of this data packet in labels.file is merged and added to the fitting data set (FP-set), the format is<f,p|label> .

[0101] Use of electronic devices<f,p|label> , and derive FP-Rule based on the activation function of the fitting algorithm of logistic regression. The FP-Rule is implemented as a function f(f,p) (whose return value is 0 or 1) with f and p as input to determine whether the traffic is a DDoS attack. The loss value is determined based on the maximum log-likelihood function to adjust the parameters of the activation function.

[0102] Among them, the electronic device can determine the loss value according to the following formula:

[0103]

[0104] Among them, y i represents the first classification information of sample data packet i, σ(w T x i +b) represents the predicted first classification information of sample data packet i, w and b are parameters to be adjusted, x i is the sample frequency count f of sample packet i i and the sample persistence count p of sample packet i i The sample feature vector composed of i =[f i ,p i ].

[0105] Figure 3 A schematic diagram of the filtering rule generation and classification model training process provided in the embodiment of the present application, such as the Figure 3 As shown in the figure, the process of generating filtering rules includes:

[0106] 1-1. Marked traffic replay. Use marked DDoS attack traffic, replay it according to the arrival time of the data packets in the data set, and use LTC to measure to obtain each target identifier f ID The corresponding frequency count and persistence count.

[0107] 1-2. Cold start waiting. In the initial stage, all counters of LTC are 0, which has no discrimination, and the output results have no practical meaning and cannot be used to generate FP-Rule. In order to solve this "cold start" problem, the present invention is designed to start recording when a certain number (the present invention sets it to 100,000 based on experience) of data packets pass through LTC. At this time, the counters will not be all zero, and these are regarded as valid fitting samples.

[0108] 1-3. Construction of fitting data set. After the cold start wait, when the marked traffic passes through LTC, use step 2 to calculate the frequency f and persistence p of each data packet, and merge the label of this data packet in labels.file and add it to the fitting data set (FP-set), in the format of <persistence count f, p persistence count | persistence count label>.

[0109] 1-4. Fitting of FP-Rule. Use the activation function based on the fitting algorithm of logistic regression to derive FP-Rule. FP-Rule is implemented as a function f(f,p) (whose return value is 0 or 1) with f and p as input to determine whether the traffic is a DDoS attack.

[0110] 1-5. Adjust the threshold θ. Adjust θ according to the precision value of the detection to ensure a low false negative rate.

[0111] The training process of CatBoost classifier includes:

[0112] 2-1. Obtaining filtered traffic. After obtaining the FP-Rule rule, similarly, replay the original traffic and calculate the persistence count and frequency count corresponding to each data packet.

[0113] 2-2. According to the trained FP-Rule, each data packet is judged, the data packets judged as DDoS attacks by the rule are filtered, and the data packets judged as non-DDoS attacks by the rule are added to train_pkt.

[0114] 2-3. After all packets are played back, extract the flow-level features of the packets in train_pkt, including but not limited to flow duration, standard deviation of packet length, total number of packets transmitted backward (received) per second, etc., to obtain the feature vector The labels in labels.file are inserted into CatBoost-trainset, where the data items are encapsulated in the format of

[0115] 2-4. Train the CatBoost classifier, use F1-score as the loss indicator, and start iterating 10-fold cross training.

[0116] 2-5. Save the classifier as model(item) and export the model file for fast loading.

[0117] The embodiment of the present application provides a hierarchical method combining online detection and offline detection, which innovatively achieves the following contributions:

[0118] 1. The LTC algorithm is used to analyze the time pattern of data packet arrival without complex flow aggregation or feature extraction. On this basis, the present invention innovatively designs a fast filtering rule based on a logistic regression model to filter out DDoS attack traffic, greatly reducing the intensity of DDoS attacks and protecting normal business traffic from being affected by attacks.

[0119] 2. A hierarchical detection method is adopted, combining online and offline screening mechanisms: Online coarse screening uses the LTC algorithm to quickly identify and screen out DDoS traffic based on the packet arrival time pattern. Offline fine screening uses machine learning classifiers (such as CatBoost) to perform flow-level detection and classification based on the data with a lower imbalance rate, which can effectively improve both classification performance and time performance.

[0120] 3. Compared with the current measurement method that only focuses on the frequency of elements, the present invention quickly evaluates the frequency and persistence of elements in network flows. Through dual measurement, the embodiment of the present application can more accurately identify complex DDoS attack patterns. Based on this, the embodiment of the present application can achieve high-speed identification and coarse screening of DDoS traffic with extremely low resource consumption, and is suitable for real-time analysis of network traffic in real environments.

[0121] Embodiment 7:

[0122] Based on the above embodiments, Figure 4 A schematic diagram of the structure of a data packet filtering device provided in an embodiment of the present application, the device comprising:

[0123] Processing module 401 is used to determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; obtain the frequency count of the data packet with the target identifier before the current time of monitoring statistics, and the persistence count of the data packet with the target identifier appearing within a set time window;

[0124] The classification module 402 is used to determine the first classification information of whether the target data packet is a data packet of a DDoS attack based on the frequency count, the persistence count, the activation function and a preset threshold; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

[0125] In one possible implementation, the processing module 401 is also used to obtain a configured data packet whitelist, wherein the data packet whitelist stores identifiers of data packets that have been determined not to be DDoS attacks; if the data packet whitelist does not contain the target identifier, a subsequent process of obtaining a frequency count of occurrences of data packets with the target identifier before the current time of monitoring statistics is executed.

[0126] In a possible implementation manner, the processing module 401 is further configured to obtain characteristic data of a target flow corresponding to the target data flow if it is determined that the target data flow is not a data flow of a DDoS attack, wherein the characteristic data includes flow duration, a standard deviation of the length of each data flow included in the target flow, and a total number of data flows transmitted per second by the target flow;

[0127] The classification module 402 is also used to input the feature data into a pre-trained classifier to obtain second classification information output by the classifier as to whether the target flow is a DDoS attack; if the second classification information is that the target flow is a DDoS attack, then determine that the target data packet is a DDoS attack data packet.

[0128] In a possible implementation, the classifier is a CatBoost classifier.

[0129] In a possible implementation, the device further includes:

[0130] The training module 403 is used to obtain a pre-configured sample stream, sample feature data corresponding to the sample data packet, and sample second classification information of whether the sample stream is a DDoS attack; input the sample feature data into the CatBoost classifier to be trained, and obtain the predicted second classification information output by the CatBoost classifier; determine the loss value based on the sample second classification information, the predicted second classification information and the F1-score algorithm, and adjust the parameters of the CatBoost classifier based on the loss value.

[0131] In a possible implementation manner, the processing module 401 is further configured to add the target identifier to a data packet whitelist if it is determined that the target data packet is not a data packet of a DDoS attack.

[0132] In a possible implementation, the training module 403 is further used to obtain a pre-configured sample data packet, a sample frequency count corresponding to the sample data packet, a sample persistence count, and sample first classification information of whether the sample data packet is a data packet of a DDoS attack; input the sample frequency count and the sample persistence count into the activation function to be trained, and obtain the predicted first classification information of whether the sample data packet is a data packet of a DDoS attack determined based on the predicted probability output by the activation function and the preset threshold; determine a loss value based on the sample first classification information, the predicted first classification information, and the maximized log-likelihood function, and adjust the parameters of the activation function based on the loss value.

[0133] Embodiment 8:

[0134] Based on the above embodiments, the present application also provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, it includes: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0135] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps:

[0136] Determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet;

[0137] Obtaining a frequency count of the data packets of the target identifier appearing before the current time of monitoring statistics, and a persistence count of the data packets of the target identifier appearing within a set time window;

[0138] According to the frequency count, the persistence count, the activation function and a preset threshold, determine whether the target data packet is the first classification information of a data packet of a DDoS attack; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

[0139] In a possible implementation manner, before obtaining the frequency count of occurrence of the data packet of the target identifier before the current time of the monitoring statistics, the method further includes:

[0140] Obtaining a configured data packet whitelist, wherein the data packet whitelist stores identifiers of data packets that have been determined not to be DDoS attacks;

[0141] If the target identifier is not included in the data packet whitelist, a subsequent process of obtaining a frequency count of occurrences of data packets with the target identifier before the current time of monitoring statistics is performed.

[0142] In a possible implementation manner, if it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes:

[0143] Acquire characteristic data of a target flow corresponding to the target data packet, wherein the characteristic data includes flow duration, a length standard deviation of each data packet contained in the target flow, and a total number of data packets transmitted per second by the target flow;

[0144] Inputting the feature data into a pre-trained classifier, and obtaining second classification information output by the classifier indicating whether the target flow is a DDoS attack;

[0145] If the second classification information indicates that the target flow is a DDoS attack, the target data packet is determined to be a data packet of the DDoS attack.

[0146] In a possible implementation, the classifier is a CatBoost classifier.

[0147] In a possible implementation, the training process of the CatBoost classifier includes:

[0148] Obtaining a pre-configured sample flow, sample feature data corresponding to the sample data packet, and sample second classification information of whether the sample flow is a DDoS attack;

[0149] Inputting the sample feature data into the CatBoost classifier to be trained, and obtaining the predicted second classification information output by the CatBoost classifier;

[0150] A loss value is determined according to the sample second classification information, the predicted second classification information and the F1-score algorithm, and the parameters of the CatBoost classifier are adjusted according to the loss value.

[0151] In a possible implementation manner, if it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes:

[0152] The target identifier is added to the data packet whitelist.

[0153] In a possible implementation, the training process of the activation function includes:

[0154] Obtaining a pre-configured sample data packet, a sample frequency count corresponding to the sample data packet, a sample persistence count, and sample first classification information of whether the sample data packet is a data packet of a DDoS attack;

[0155] Inputting the sample frequency count and the sample persistence count into the activation function to be trained, and obtaining the predicted first classification information of whether the sample data packet is a data packet of a DDoS attack based on the predicted probability output by the activation function and the preset threshold;

[0156] A loss value is determined according to the sample first classification information, the predicted first classification information, and the maximized log-likelihood function, and the parameters of the activation function are adjusted according to the loss value.

[0157] Since the principle of solving the problem by the above electronic device is similar to the data packet filtering method, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0158] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 502 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0159] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0160] Embodiment 9:

[0161] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0162] Determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet;

[0163] Obtaining a frequency count of the data packets of the target identifier appearing before the current time of monitoring statistics, and a persistence count of the data packets of the target identifier appearing within a set time window;

[0164] According to the frequency count, the persistence count, the activation function and a preset threshold, determine whether the target data packet is the first classification information of a data packet of a DDoS attack; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

[0165] In a possible implementation manner, before obtaining the frequency count of occurrence of the data packet of the target identifier before the current time of the monitoring statistics, the method further includes:

[0166] Obtaining a configured data packet whitelist, wherein the data packet whitelist stores identifiers of data packets that have been determined not to be DDoS attacks;

[0167] If the target identifier is not included in the data packet whitelist, a subsequent process of obtaining a frequency count of occurrences of data packets with the target identifier before the current time of monitoring statistics is performed.

[0168] In a possible implementation manner, if it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes:

[0169] Acquire characteristic data of a target flow corresponding to the target data packet, wherein the characteristic data includes flow duration, a length standard deviation of each data packet contained in the target flow, and a total number of data packets transmitted per second by the target flow;

[0170] Inputting the feature data into a pre-trained classifier, and obtaining second classification information output by the classifier indicating whether the target flow is a DDoS attack;

[0171] If the second classification information indicates that the target flow is a DDoS attack, the target data packet is determined to be a data packet of the DDoS attack.

[0172] In a possible implementation, the classifier is a CatBoost classifier.

[0173] In a possible implementation, the training process of the CatBoost classifier includes:

[0174] Obtaining a pre-configured sample flow, sample feature data corresponding to the sample data packet, and sample second classification information of whether the sample flow is a DDoS attack;

[0175] Inputting the sample feature data into the CatBoost classifier to be trained, and obtaining the predicted second classification information output by the CatBoost classifier;

[0176] A loss value is determined according to the sample second classification information, the predicted second classification information and the F1-score algorithm, and the parameters of the CatBoost classifier are adjusted according to the loss value.

[0177] In a possible implementation manner, if it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes:

[0178] The target identifier is added to the data packet whitelist.

[0179] In a possible implementation, the training process of the activation function includes:

[0180] Obtaining a pre-configured sample data packet, a sample frequency count corresponding to the sample data packet, a sample persistence count, and sample first classification information of whether the sample data packet is a data packet of a DDoS attack;

[0181] Inputting the sample frequency count and the sample persistence count into the activation function to be trained, and obtaining the predicted first classification information of whether the sample data packet is a data packet of a DDoS attack based on the predicted probability output by the activation function and the preset threshold;

[0182] A loss value is determined according to the sample first classification information, the predicted first classification information, and the maximized log-likelihood function, and the parameters of the activation function are adjusted according to the loss value.

[0183] Since the principle of solving the problem by the above computer program product is similar to that of the data packet filtering method, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0184] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A data packet filtering method, characterized in that: The method comprises: Determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; Obtaining a frequency count of the data packets of the target identifier appearing before the current time of monitoring statistics, and a persistence count of the data packets of the target identifier appearing within a set time window; According to the frequency count, the persistence count, the activation function and a preset threshold, determine whether the target data packet is the first classification information of a data packet of a DDoS attack; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

2. The method according to claim 1, characterized in that: Before obtaining the frequency count of the occurrence of the data packet of the target identifier before the current time of the monitoring statistics, the method further includes: Obtaining a configured data packet whitelist, wherein the data packet whitelist stores identifiers of data packets that have been determined not to be DDoS attacks; If the target identifier is not included in the data packet whitelist, a subsequent process of obtaining a frequency count of occurrences of data packets with the target identifier before the current time of monitoring statistics is performed.

3. The method according to claim 1 or 2, characterized in that: If it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes: Acquire characteristic data of a target flow corresponding to the target data packet, wherein the characteristic data includes flow duration, a length standard deviation of each data packet contained in the target flow, and a total number of data packets transmitted per second by the target flow; Inputting the feature data into a pre-trained classifier, and obtaining second classification information output by the classifier indicating whether the target flow is a DDoS attack; If the second classification information indicates that the target flow is a DDoS attack, the target data packet is determined to be a data packet of the DDoS attack.

4. The method according to claim 3, characterized in that The classifier is a CatBoost classifier.

5. The method according to claim 4, characterized in that The training process of the CatBoost classifier includes: Obtaining a pre-configured sample flow, sample feature data corresponding to the sample data packet, and sample second classification information of whether the sample flow is a DDoS attack; Inputting the sample feature data into the CatBoost classifier to be trained, and obtaining the predicted second classification information output by the CatBoost classifier; A loss value is determined according to the sample second classification information, the predicted second classification information and the F1-score algorithm, and the parameters of the CatBoost classifier are adjusted according to the loss value.

6. The method according to claim 3, characterized in that If it is determined that the target data packet is not a data packet of a DDoS attack, the method further includes: The target identifier is added to the data packet whitelist.

7. The method according to claim 1, characterized in that The training process of the activation function includes: Obtaining a pre-configured sample data packet, a sample frequency count corresponding to the sample data packet, a sample persistence count, and sample first classification information of whether the sample data packet is a data packet of a DDoS attack; Inputting the sample frequency count and the sample persistence count into the activation function to be trained, and obtaining the predicted first classification information of whether the sample data packet is a data packet of a DDoS attack based on the predicted probability output by the activation function and the preset threshold; A loss value is determined according to the sample first classification information, the predicted first classification information, and the maximized log-likelihood function, and the parameters of the activation function are adjusted according to the loss value.

8. A data packet filtering device, characterized in that: The device comprises: A processing module is used to determine the source address, destination address, source port, destination port and protocol information carried in the target data packet to be processed as the target identifier of the target data packet; obtain the frequency count of the data packet with the target identifier before the current time of the monitoring statistics, and the persistence count of the data packet with the target identifier appearing within a set time window; According to the frequency count, the persistence count, the activation function and a preset threshold, determine whether the target data packet is the first classification information of a data packet of a DDoS attack; wherein the activation function is trained based on the historical frequency count of historical data packets, the historical persistence count and the historical first classification information of whether the historical data packet is a data packet of a DDoS attack.

9. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the data packet filtering method as claimed in any one of claims 1 to 7 when executing a computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the data packet filtering method as claimed in any one of claims 1 to 7.

Images