A fastflood detection method and device based on a Flink stream processing engine

Abstract

The application discloses a FastFlood detection method and device based on a Flink stream processing engine, and the method comprises the following steps: collecting traffic data from various data sources in real time by the Flink stream processing engine, and distributing the traffic data to corresponding sliding time windows; the sliding time window allows the window to slide on a time axis, and the change of traffic is captured by setting the window size and the sliding step; in each sliding-out window, statistical analysis is performed on the traffic data; the statistical analysis result is compared with a preset threshold value, and it is judged whether the typical characteristics of the pulse type DDoS attack are presented; if the pulse type DDoS attack is identified, the corresponding protection mechanism is triggered. The method and device can not only cross the limit of the operator-level network scale, realize seamless coverage and efficient data collection, but also accurately extract attack characteristics in a complex and changeable network environment, and build a high-sensitivity real-time detection model.

Description

Technical Field

[0001] This invention relates to the field of DDoS attack detection, and in particular to a FastFlood detection method and apparatus based on the Flink stream processing engine. Background Technology

[0002] Operator A's anti-DDoS detection system collaborates with university laboratories and requires a rapid detection solution for its anti-DDoS capabilities—one that can generate attack alerts more quickly and identify pulse attacks. The objective is to generate alerts within 30 seconds, maximizing the timeliness of attack response. Unlike the previous one-minute time limit, the technical requirement is to use a real-time stream processing engine to analyze network traffic in real time and promptly report alert events.

[0003] Currently, the A operator group has a DDoS attack detection system, but it does not support FastFlood, which is a type of high-frequency pulse attack. High-frequency pulse attacks are attacks with very high traffic in a short period of time, usually lasting only 1 second or even less, but the attack traffic is very large. FastFlood is this type of attack. Therefore, based on the existing implementation, a model for identifying pulse attacks needs to be designed. Summary of the Invention

[0004] Addressing the severe challenges of rapid DDoS detection in current large-scale network environments—namely, the bottleneck in the granularity of wide-area traffic monitoring and the lack of real-time identification of pulse-type attacks—this invention provides a FastFlood detection method and device based on the Flink stream processing engine. By integrating big data processing, advanced feature engineering, and optimized training of the stream processing engine, it enables intelligent and rapid detection of pulse-type DDoS attacks with massive traffic volumes. This not only overcomes the limitations of carrier-grade network scale, achieving seamless coverage and efficient data collection, but also accurately extracts attack features in complex and ever-changing network environments, constructing a highly sensitive real-time detection model.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In one embodiment of the present invention, a FastFlood detection method based on the Flink stream processing engine is proposed, the method comprising:

[0007] The Flink stream processing engine collects traffic data from various data sources in real time and allocates it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis, capturing changes in traffic by setting the window size and sliding step.

[0008] Within each slide-out window, statistical analysis of the traffic data is performed;

[0009] The statistical analysis results are compared with preset thresholds to determine whether they exhibit typical characteristics of a pulse DDoS attack. If a pulse DDoS attack is identified, the corresponding protection mechanism is triggered.

[0010] Furthermore, for the sliding window traffic statistics below a preset threshold, a label is added, and the average of the statistical sliding window traffic data is used as the baseline traffic.

[0011] Furthermore, for the traffic statistics of the slideout window that exceed the preset threshold, if the traffic statistics are more than three times the threshold, the normal detection process is initiated. The average traffic within the slideout window is compared with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event. If the traffic statistics are more than three times the threshold, the Fastflood detection process is initiated. If the traffic of the slideout window is more than three times the baseline traffic of the previous two slideout windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

[0012] In one embodiment of the present invention, a FastFlood detection device based on the Flink stream processing engine is also proposed, the device comprising:

[0013] The traffic acquisition and allocation module is used by the Flink stream processing engine to collect traffic data from various data sources in real time and allocate it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis, and by setting the window size and sliding step, changes in traffic can be captured.

[0014] The traffic statistics and analysis module is used to perform statistical analysis on traffic data within each slide-out window;

[0015] The attack identification module compares the statistical analysis results with preset thresholds and determines whether they exhibit typical characteristics of a pulse DDoS attack. If a pulse DDoS attack is identified, the corresponding protection mechanism is triggered.

[0016] Furthermore, for the sliding window traffic statistics below a preset threshold, a label is added, and the average of the statistical sliding window traffic data is used as the baseline traffic.

[0017] Furthermore, for the traffic statistics of the slideout window that exceed the preset threshold, if the traffic statistics are more than three times the threshold, the normal detection process is initiated. The average traffic within the slideout window is compared with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event. If the traffic statistics are more than three times the threshold, the Fastflood detection process is initiated. If the traffic of the slideout window is more than three times the baseline traffic of the previous two slideout windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

[0018] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned FastFlood detection method based on the Flink stream processing engine is designed.

[0019] In one embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that executes a FastFlood detection method based on the Flink stream processing engine.

[0020] Beneficial effects:

[0021] 1. In this invention, Flink's real-time processing capability enables FastFlood detection to analyze network traffic data in real time, promptly detecting and responding to traffic anomalies. This is crucial for protecting networks from threats such as flood attacks.

[0022] 2. In this invention, Flink provides rich windowing operation functions, such as scrolling windows, sliding windows, and session windows. These functions allow users to flexibly group, aggregate, and calculate network traffic data in FastFlood detection according to actual needs, thereby more accurately identifying abnormal behavior.

[0023] 3. In this invention, Flink supports distributed operation and can be easily scaled to thousands of nodes. This enables the FastFlood detection device to handle large-scale network traffic data, meeting the needs of high concurrency and large data volumes. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the FastFlood detection method based on the Flink stream processing engine according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the FastFlood detection device based on the Flink stream processing engine of the present invention.

[0026] Figure 3This is a schematic diagram of the computer device structure of the present invention. Detailed Implementation

[0027] The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of the disclosure to those skilled in the art.

[0028] Those skilled in the art will recognize that embodiments of the present invention can be designed as an apparatus, device, device, method, or computer program product. Therefore, this disclosure can be specifically designed as: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0029] According to an embodiment of the present invention, a FastFlood detection method based on the Flink stream processing engine is proposed. Employing the Flink stream processing engine, and through a carefully designed time window mechanism and real-time traffic analysis technology, it effectively identifies and protects against pulse-type DDoS attacks. In the Flink stream processing engine, the time window is a key mechanism for processing streaming data, used to divide an infinite data stream into finite data blocks for processing. Considering the characteristics of pulse-type DDoS attacks, the following time window design can be adopted: a sliding time window allows the window to slide along the time axis. By setting the window size and sliding step, changes in the data stream can be flexibly captured. When identifying pulse-type DDoS attacks, a relatively short window size (60 seconds) and a relatively small sliding step (5 seconds) are set. This allows the window to update data frequently, reflecting traffic changes in a timely manner. In addition to the time factor, a traffic threshold is also needed as a trigger condition for the window. When the traffic within the window exceeds the preset threshold, the attack identification process is triggered.

[0030] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments.

[0031] The FastFlood detection method based on the Flink stream processing engine proposed in this invention includes:

[0032] 1. The Flink stream processing engine collects traffic data in real time from various data sources.

[0033] 2. Use the window allocator of the Flink stream processing engine (such as Flink's SlidingTimeWindows) to distribute the collected traffic data into the corresponding sliding time windows. The sliding time window allows the window to slide on the time axis, capturing changes in traffic by setting the window size and sliding step.

[0034] 3. Perform statistical analysis on the traffic data within each slide-out window, such as calculating total traffic, average traffic, and peak traffic.

[0035] 4. Compare the statistical analysis results with preset thresholds and determine whether they exhibit typical characteristics of a pulse-type DDoS attack (such as a sudden increase in traffic followed by a rapid decrease). If the traffic statistics within a certain slip-out window exceed the preset threshold and exhibit typical characteristics of a pulse-type DDoS attack, it is identified as a pulse-type DDoS attack. Once a pulse-type DDoS attack is identified, corresponding protection mechanisms can be triggered, such as automatically adjusting network configurations and activating traffic scrubbing equipment, to mitigate or eliminate the impact of the attack.

[0036] It should be noted that although the operation of the method of the present invention has been described in a specific order in the above embodiments and figures, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0037] To provide a clearer explanation of the FastFlood detection method based on the Flink stream processing engine, a specific embodiment will be used for illustration below. However, it is worth noting that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation of the present invention.

[0038] Example:

[0039] The following example uses the Flink stream processing engine and NetFlow data as the data source for network traffic to simulate and implement fast flood detection capabilities. Figure 1 As shown.

[0040] 1. Flink receives NetFlow data from Kafka in real time and parses the received NetFlow data into the following format:

[0041] XFlowRecord:{

[0042] ipVer (IP version);

[0043] String srcAddr(source IP address, e.g., 192.168.1.1);

[0044] String dstAddr(destination IP address, e.g., 192.168.1.1);

[0045] long packets (number of packets);

[0046] long octets(number of bytes);

[0047] int srcPort(source port);

[0048] int dstPort(destination port);

[0049] int protocol;

[0050] int attackType; (attack characteristics)

[0051] }

[0052] 2. Set an initial threshold, based on experience. This will not be explained here, assuming all preparations for detecting pulse attacks have been completed.

[0053] 3. Build a Flink runtime environment, set a 60-second sliding window with a 5-second sliding step, group by destination IP, and calculate the average traffic within each sliding window according to IP + attack type identifier. Record the traffic in bps for each sliding window, identify windows with traffic below a threshold, and calculate the average window traffic in real time as the baseline traffic for that IP and attack type. Record the flow rate for each 5-second sliding window in the StoreDstIpDataSink output method. For entries below the threshold, add a lowerThreshold label. The sliding window is a fixed-length time window; for example, a 60-second sliding window always has a window length of 60 seconds and will slide forward over time. The concept of step size is related to the slide-out window. For example, a 5-second step size means the slide-out window slides once every five seconds, i.e., 0-60, 5-65, 10-70, 15-75, and so on. The slide-out window is the data within this window that slides once every five seconds. For example, from 5 to 65 seconds, the data in the slide-out window is the data from 0 to 60 seconds, also known as expired window data. The StoreDstIpDataSink output method: Tag the traffic data of each slide-out window with "lowerThreshold" and record it in the database.

[0054] StreamExecutionEnvironment env=StreamExecutionEnvironment.getExecutionEnvironment();

[0055] String topic = "flow_test";

[0056] env.addSource(createKafkaConsumer(env,topic)).map(item->JSON.parseObject((String)item,XFlowRecord.class))

[0057] .keyBy((KeySelector<XFlowRecord,String> XFlowRecord::getDstAddress) / / Calculates data in groups using the destination IP as the key

[0058] .window(SlidingProcessingTimeWindows.of(Time.se conds(60),Time.seconds(5))) / / Defines a sliding window with a length of 60 seconds, which slides once every 5 seconds.

[0059] .apply(new

[0060] DstIpAttackTypeThresholdHoldCompareFunction() / / Simulates the Flink window computation module to calculate the bps size of the target IP attack type.

[0061] .addSink(newStoreDstIpDataSink()) / / Store window data below and above the threshold separately.

[0062] .setParallelism(10);

[0063] env.execute("fastFlood-test");

[0064] 4. Record window traffic exceeding the set threshold, categorized into the following cases: First, if the traffic exceeds the set threshold but is less than three times the threshold, this is considered a normal attack event. The average flow rate within the sliding window needs to be calculated and compared to the threshold. If it exceeds the threshold, an alarm is triggered, but the FastFlood flag is not recorded. Second, if the traffic exceeds the set threshold by more than three times, the FastFlood detection process begins. If the traffic in this sliding window is more than three times the baseline traffic of the previous two windows, it is determined to be a pulse attack, generating an attack event. The processing logic is as follows: Figure 1 As shown.

[0065] Figure 1 The processing logic mainly demonstrates the Flink stream processing engine's method logic for processing window data. The first method is `DstIpAttackTypeThresholdHoldCompareFunction`: This method is mainly for processing data within a sliding window with a step size of 5 seconds. It compares the bit speed (bps) of the data within the window with a set threshold, and determines whether to proceed with the normal detection process or the fast detection process (FastFlood) based on the comparison result.

[0066] StoreDstIpDataSink: This method is mainly used to store the data results within a sliding window with a step size of 5 seconds. The stored record should be marked with lowerThreshold to indicate whether the flow of the window is higher than the threshold, so that the flow window result below the threshold can be retrieved in the DstIpAttackTypeThresholdHoldCompareFunction method as the baseline flow for comparison with the window flow in the FastFlood detection process.

[0067] 5. Business operations following the attack are not described here.

[0068] Based on the same inventive concept, this invention also proposes a FastFlood detection device based on the Flink stream processing engine. The implementation of this device can refer to the implementation of the method described above, and repeated details will not be repeated. The term "module" used below can refer to a combination of software and / or hardware designed with a predetermined function. Although the device described in the following embodiments is preferably designed in software, hardware designs, or combinations of software and hardware, are also possible and contemplated.

[0069] Figure 2 This is a schematic diagram of the FastFlood detection device based on the Flink stream processing engine of the present invention. Figure 2 As shown, the device includes:

[0070] The traffic acquisition and allocation module 101 is used by the Flink stream processing engine to acquire traffic data from various data sources in real time and allocate it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis, and by setting the window size and sliding step, changes in traffic can be captured.

[0071] The traffic statistics and analysis module 102 is used to perform statistical analysis on traffic data within each slide-out window.

[0072] The attack identification module 103 is used to compare the statistical analysis results with a preset threshold and determine whether it exhibits typical characteristics of a pulse DDoS attack; if a pulse DDoS attack is identified, the corresponding protection mechanism is triggered.

[0073] For sliding window traffic statistics below a preset threshold, a label is added, and the average of the statistical sliding window traffic data is used as the baseline traffic.

[0074] For traffic statistics within a slideout window that exceed a preset threshold, if the traffic statistics are more than three times the threshold, the normal detection process is initiated. The average traffic within the slideout window is compared with the preset threshold. If it exceeds the preset threshold, an alarm is triggered and the event is recorded as a normal attack. If the traffic statistics are more than three times the threshold, the Fastflood detection process is initiated. If the traffic within the slideout window is more than three times the baseline traffic of the previous two slideout windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

[0075] It should be noted that although several modules of the FastFlood detection device based on the Flink stream processing engine have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in a single module. Conversely, the features and functions of a single module described above can be further divided and embodied by multiple modules.

[0076] Based on the aforementioned inventive concept, such as Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, it designs the aforementioned FastFlood detection method based on the Flink stream processing engine.

[0077] Based on the aforementioned inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program that executes the aforementioned FastFlood detection method based on the Flink stream processing engine.

[0078] The FastFlood detection method and apparatus based on the Flink stream processing engine proposed in this invention have the following advantages:

[0079] 1. Real-time detection and response:

[0080] Flink's real-time processing capabilities enable FastFlood detection to analyze network traffic data in real time, promptly detecting and responding to traffic anomalies. This is crucial for protecting networks from threats such as flood attacks.

[0081] 2. Flexible windowed operation:

[0082] Flink offers a rich set of windowing functionalities, such as scrolling windows, sliding windows, and session windows. These features allow users to flexibly group, aggregate, and compute network traffic data in FastFlood detection according to their actual needs, thereby more accurately identifying anomalous behavior.

[0083] 3. High scalability:

[0084] Flink supports distributed operation and can be easily scaled to thousands of nodes. This enables FastFlood detection devices to handle large-scale network traffic data, meeting the needs of high concurrency and large data volumes.

[0085] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

[0086] Regarding the limitation of the scope of protection of this invention, those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solution of this invention are still within the scope of protection of this invention.

Claims

1. A FastFlood detection method based on the Flink stream processing engine, characterized in that, The method includes: The Flink stream processing engine collects traffic data from various data sources in real time and allocates it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis, capturing changes in traffic by setting the window size and sliding step. Within each slide-out window, statistical analysis of the traffic data is performed; The statistical analysis results are compared with preset thresholds to determine whether they exhibit typical characteristics of a pulse-type DDoS attack. For outflow window traffic statistics below a preset threshold, a label is added, and the average of the outflow window traffic statistics is used as the baseline traffic. For the traffic statistics of the slideout window that exceed the preset threshold, if the traffic statistics are more than three times the threshold, the normal detection process is entered. The average traffic in the slideout window is compared with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event. If the traffic statistics are more than three times the threshold, the Fastflood detection process is entered. If the traffic in the slideout window is more than three times the baseline traffic of the previous two slideout windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event. If a pulse-type DDoS attack is identified, the corresponding protection mechanism will be triggered.

2. A FastFlood detection device based on the Flink stream processing engine, characterized in that, The device includes: The traffic acquisition and allocation module is used by the Flink stream processing engine to collect traffic data from various data sources in real time and allocate it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis, and by setting the window size and sliding step, changes in traffic can be captured. The traffic statistics and analysis module is used to perform statistical analysis on traffic data within each slide-out window; The attack identification module compares statistical analysis results with preset thresholds to determine whether they exhibit typical characteristics of a pulse DDoS attack. For slip-out window traffic statistics below the preset threshold, a label is added, and the average of these statistics is used as the baseline traffic. For slip-out window traffic statistics above the preset threshold, if the traffic statistics are more than three times the threshold, a normal detection process is initiated. The average traffic within the slip-out window is compared with the preset threshold; if it exceeds the threshold, an alarm is triggered and recorded as a normal attack event. If the traffic statistics are more than three times the threshold, a Fastflood detection process is initiated. If the traffic in this slip-out window is more than three times the baseline traffic of the previous two slip-out windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event. If a pulse DDoS attack is identified, the corresponding protection mechanism is triggered.

3. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program in accordance with the method of claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that performs the method of claim 1.

Images