FastFlood detection method and device based on Flink stream processing engine

Abstract

The invention discloses a FastFlood detection method and device based on an Flink flow processing engine, and the method comprises the steps: collecting flow data from various data sources in real time through the Flink flow processing engine, and distributing the flow data to a corresponding sliding time window; sliding the time window to allow the window to slide on the time axis, and capturing the flow change by setting the window size and the sliding step length; carrying out statistical analysis on the flow data in each sliding-out window; the statistical analysis result is compared with a preset threshold value, and whether typical features of the pulse type DDoS attack are presented or not is judged; and if the pulse type DDoS attack is identified, triggering a corresponding protection mechanism. According to the method and the device, the limitation of an operator-level network scale can be spanned, seamless coverage and efficient data acquisition can be realized, attack characteristics can be accurately extracted in a complex and changeable network environment, and a high-sensitivity real-time detection model can be constructed.

Description

Technical Field

[0001] The present invention relates to the field of DDoS attack detection, and in particular to a FastFlood detection method and device based on a Flink stream processing engine. Background Art

[0002] The anti-D detection system of operator group A has exchanges and cooperation with university laboratories. The anti-D side needs to come up with a fast detection solution, that is, to generate attack alarms more quickly and identify pulse attacks. The target requirement of this task is to generate alarms within 30 seconds to ensure the timeliness of the attack to the greatest extent. Different from the previous one-minute time limit, the technical requirement is to use a real-time stream processing engine to analyze network traffic in real time and report alarm events in a timely manner.

[0003] Currently, Operator Group A has an existing DDos attack detection system, but it does not support FastFlood, which means fast detection, also known as the detection of high-frequency pulse attack types. High-frequency pulse attacks are attacks with very high traffic in a short period of time, usually lasting only 1 second or even shorter, but the attack traffic is very large. FastFlood is this type of attack. Therefore, based on the existing implementation, it is necessary to design a model to identify pulse attacks. Summary of the invention

[0004] In view of the severe challenges faced by rapid DDoS detection in current large-scale network environments, namely, the precision bottleneck of wide-area traffic monitoring and the lack of instant identification of pulse attacks, the present invention provides a FastFlood detection method and device based on the Flink stream processing engine. By integrating big data processing, advanced feature engineering and optimized training of the stream processing engine, intelligent rapid detection of pulse-type DDoS attacks with massive traffic is performed. It can not only overcome the limitations of the scale of operator-level networks and achieve seamless coverage and efficient data collection, but also accurately extract attack features in complex and changeable network environments and build a highly sensitive real-time detection model.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions:

[0006] In one embodiment of the present invention, a FastFlood detection method based on the Flink stream processing engine is proposed, and the method includes:

[0007] The Flink stream processing engine collects traffic data from various data sources in real time and distributes it to corresponding sliding time windows. The sliding time window allows the window to slide on the time axis and capture traffic changes by setting the window size and sliding step size.

[0008] In each slide-out window, statistical analysis is performed on the traffic data;

[0009] The statistical analysis results are compared with the preset thresholds to determine whether they present typical characteristics of a pulse-type DDoS attack; if a pulse-type DDoS attack is identified, the corresponding protection mechanism is triggered.

[0010] Furthermore, for the sliding window traffic statistics data that is lower than a preset threshold, a label is added, and the average value of the statistical sliding window traffic data is used as the baseline traffic.

[0011] Furthermore, for the traffic statistics of the sliding window that are higher than the preset threshold, if the traffic statistics are more than three times lower than the threshold, the normal detection process is entered, and the average traffic in the sliding window is compared with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event; if the traffic statistics are more than three times higher than the threshold, the Fastflood detection process is entered. If the traffic in the sliding window is higher than three times the baseline traffic of the previous two sliding windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

[0012] In one embodiment of the present invention, a FastFlood detection device based on the Flink stream processing engine is also proposed, and the device includes:

[0013] The traffic collection and distribution module is used by the Flink stream processing engine to collect traffic data from various data sources in real time and distribute it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis and capture traffic changes by setting the window size and sliding step size.

[0014] Traffic statistics analysis module, used to perform statistical analysis on traffic data in each slide-out window;

[0015] The attack identification module is used to compare the statistical analysis results with the preset threshold and determine whether the typical characteristics of a pulse-type DDoS attack are present; if a pulse-type DDoS attack is identified, the corresponding protection mechanism is triggered.

[0016] Furthermore, for the sliding window traffic statistics data that is lower than a preset threshold, a label is added, and the average value of the statistical sliding window traffic data is used as the baseline traffic.

[0017] Furthermore, for the traffic statistics of the sliding window that are higher than the preset threshold, if the traffic statistics are more than three times lower than the threshold, the normal detection process is entered, and the average traffic in the sliding window is compared with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event; if the traffic statistics are more than three times higher than the threshold, the Fastflood detection process is entered. If the traffic in the sliding window is higher than three times the baseline traffic of the previous two sliding windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

[0018] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned FastFlood detection method based on the Flink stream processing engine is designed.

[0019] In one embodiment of the present invention, a computer-readable storage medium is further proposed, wherein the computer-readable storage medium stores a computer program for executing a FastFlood detection method based on a Flink stream processing engine.

[0020] Beneficial effects:

[0021] 1. The real-time processing capability of Flink in the present invention enables FastFlood detection to analyze network traffic data in real time and promptly detect and respond to traffic anomalies. This is crucial for protecting the network from threats such as flood attacks.

[0022] 2. In the present invention, Flink provides a variety of windowing operation functions, such as rolling windows, sliding windows, and session windows, etc. These functions allow users to flexibly group, aggregate, and calculate network traffic data in FastFlood detection according to actual needs, so as to more accurately identify abnormal behaviors.

[0023] 3. In the present invention, Flink supports distributed operation and can be easily expanded to thousands of nodes. This enables the FastFlood detection device to process large-scale network traffic data and meet the needs of high concurrency and large data volume. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of a FastFlood detection method based on the Flink stream processing engine according to an embodiment of the present invention;

[0025] Figure 2 It is a structural diagram of a FastFlood detection device based on a Flink stream processing engine of the present invention;

[0026] Figure 3It is a schematic diagram of the computer device structure of the present invention. DETAILED DESCRIPTION

[0027] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and design the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] Those skilled in the art will appreciate that the embodiments of the present invention may be designed as a device, apparatus, equipment, method or computer program product. Therefore, the present disclosure may be specifically designed in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0029] According to an embodiment of the present invention, a FastFlood detection method based on the Flink stream processing engine is proposed. The Flink stream processing engine is used to effectively identify and protect against pulse-type DDoS attacks through a carefully designed time window mechanism and real-time traffic analysis technology. In the Flink stream processing engine, the time window is a key mechanism for processing stream data, which is used to cut an infinite data stream into limited data blocks for processing. In view of the characteristics of pulse-type DDoS attacks, the following time window design can be adopted: the sliding time window allows the window to slide on the time axis, and by setting the window size and sliding step, the changes in the data stream can be flexibly captured. When identifying pulse-type DDoS attacks, a shorter window size (60 seconds) and a relatively small sliding step (5 seconds) are set. In this way, the window can frequently update data and reflect traffic changes in a timely manner. In addition to the time factor, it is also necessary to combine the traffic threshold as the trigger condition of the window. When the traffic in the window exceeds the preset threshold, the attack identification process is triggered.

[0030] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0031] The FastFlood detection method based on the Flink stream processing engine proposed in the present invention includes:

[0032] 1. The Flink stream processing engine collects traffic data in real time from various data sources.

[0033] 2. Use the window allocator of the Flink stream computing engine (such as Flink's SlidingTimeWindows) to allocate the collected traffic data to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis and capture traffic changes by setting the window size and sliding step.

[0034] 3. In each slide-out window, perform statistical analysis on the traffic data, such as calculating the total traffic, average traffic, traffic peak, etc.

[0035] 4. Compare the statistical analysis results with the preset threshold and determine whether they show typical characteristics of a pulse-type DDoS attack (such as a sudden increase in traffic followed by a rapid decrease); if the traffic statistics in a certain slide-out window exceed the preset threshold and show typical characteristics of a pulse-type DDoS attack, it is identified as a pulse-type DDoS attack. Once a pulse-type DDoS attack is identified, the corresponding protection mechanism can be triggered, such as automatically adjusting the network configuration, starting the traffic cleaning device, etc., to reduce or eliminate the impact of the attack.

[0036] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0037] In order to explain the FastFlood detection method based on the Flink stream processing engine more clearly, a specific embodiment is described below. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0038] Example:

[0039] The following uses the Flink stream processing engine as an example, using netflow data as the data source of network traffic to simulate the fast detection (FastFlood) capability. Figure 1 shown.

[0040] 1. Flink receives netflow data from Kafka in real time and parses the received netflow data into the following format:

[0041] XFlowRecord:

[0042] ipVer(ip version);

[0043] String srcAddr (source IP address, such as 192.168.1.1);

[0044] String dstAddr (destination IP address, such as 192.168.1.1);

[0045] long packets (number of packets);

[0046] long octets(byte count);

[0047] int srcPort(source port);

[0048] int dstPort(destination port);

[0049] int protocol(protocol);

[0050] int attackType; (attack characteristics)

[0051] }

[0052] 2. Set the initial threshold based on experience. No explanation is given here, assuming that all preparations for detecting pulse attacks have been completed.

[0053] 3. Build the Flink operating environment, set a 60s sliding time window with a sliding step of 5s, group by destination IP, count the average flow in each sliding window according to IP+attack type identification, record the flow bps of each sliding window, identify the window flow below the threshold, and count the average of the window flow in real time as the baseline flow of the IP and the attack type. Record the flow rate of each 5s sliding window in the StoreDstIpDataSink output method, and add the lowerThreshold label to entries below the threshold. A sliding window is a fixed-length time window. For example, a 60-second sliding window has a window length of 60s and will slide backwards over time. The concept of step size is related to the sliding window. For example, a step size of 5s means that the sliding window slides every five seconds, that is, 0-60, 5-65, 10-70, 15-75, and so on. The sliding window is the window data that slides every five seconds, such as 5-65 seconds. The data of the sliding window is the data of the 0-60s window, also known as the expired window data. StoreDstIpDataSink output method: The method of marking the traffic data of each sliding window with the "lowerThreshold" label and recording it in the database.

[0054] StreamExecutionEnvironment env=StreamExecutionEnvironment.getExecutionEnvironment();

[0055] String topic = "flow_test";

[0056] env.addSource(createKafkaConsumer(env,topic)).map(item->JSON.parseObject((String)item,XFlowRecord.class))

[0057] .keyBy((KeySelector<XFlowRecord,String> )XFlowRecord::getDstAddress) / / Group calculation based on destination IP as key

[0058] .window(SlidingProcessingTimeWindows.of(Time.sessions(60),Time.seconds(5))) / / Define a sliding window of length 60s, sliding every 5s

[0059] .apply(new

[0060] DstIpAttackTypeThresholdHoldCompareFunction()) / / Simulate the Flink window calculation module to calculate the bps size of the target IP attack type

[0061] .addSink(newStoreDstIpDataSink()) / / Classify and store window data below and above the threshold

[0062] .setParallelism(10);

[0063] env.execute("fastFlood-test");

[0064] 4. Record the window flow that is higher than the set threshold. There are several cases: First, it is higher than the set threshold, but the flow is not more than 3 times the threshold. In this case, it is a common attack event. It is necessary to calculate the average flow rate in the sliding window and compare it with the threshold. If it is greater than the threshold, an alarm is triggered and the FastFlood mark is not recorded. Second, it is higher than the set threshold and more than 3 times higher, then enter the Fastflood detection process. If the flow of the sliding window is higher than 3 times the baseline flow of the previous two windows, it is determined to be a pulse attack and an attack event is generated. The processing logic is as follows: Figure 1 shown.

[0065] Figure 1 The processing logic mainly shows the logic of the Flink stream processing engine for processing window data. The first one is DstIpAttackTypeThresholdHoldCompareFunction: This method is mainly for processing data in a sliding window with a step size of 5s. The bps of the data in the window is compared with the set threshold, and the comparison result determines whether to use the normal detection process or the fast detection process (FastFlood).

[0066] StoreDstIpDataSink: This method is mainly used to store data results in a sliding window with a step size of 5s. The storage record should be marked with lowerThreshold to indicate whether the window flow is higher than the threshold, so as to take out the flow window result below the threshold in the DstIpAttackTypeThresholdHoldCompareFunction method as the baseline flow to compare with the window flow in the FastFlood detection process.

[0067] 5. The business after the attack is not explained here.

[0068] Based on the same inventive concept, the present invention also proposes a FastFlood detection device based on the Flink stream processing engine. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware designed for a predetermined function. Although the device described in the following embodiments is preferably designed with software, the design of hardware, or a combination of software and hardware is also possible and conceivable.

[0069] Figure 2 Schematic diagram of the FastFlood detection device based on the Flink stream processing engine of the present invention. Figure 2 As shown, the device comprises:

[0070] The traffic collection and distribution module 101 is used by the Flink stream processing engine to collect traffic data from various data sources in real time and distribute it to the corresponding sliding time window; the sliding time window allows the window to slide on the time axis, and captures traffic changes by setting the window size and sliding step.

[0071] The traffic statistics analysis module 102 is used to perform statistics analysis on the traffic data in each slide-out window.

[0072] The attack identification module 103 is used to compare the statistical analysis results with a preset threshold and determine whether a typical feature of a pulse-type DDoS attack is present; if a pulse-type DDoS attack is identified, a corresponding protection mechanism is triggered.

[0073] For the sliding window traffic statistics data that are lower than the preset threshold, a label is added, and the mean value of the sliding window traffic data is used as the baseline traffic.

[0074] For the traffic statistics of the sliding window that is higher than the preset threshold, if the traffic statistics are more than three times lower than the threshold, the normal detection process is entered, and the average traffic in the sliding window is compared with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event; if the traffic statistics are more than three times higher than the threshold, the Fastflood detection process is entered. If the traffic in the sliding window is higher than three times the baseline traffic of the previous two sliding windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

[0075] It should be noted that although several modules of the FastFlood detection device based on the Flink stream processing engine are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be concretized in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for concretization.

[0076] Based on the above invention concept, Figure 3 As shown, the present invention also proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned FastFlood detection method based on the Flink stream processing engine is designed.

[0077] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned FastFlood detection method based on the Flink stream processing engine.

[0078] The FastFlood detection method and device based on the Flink stream processing engine proposed in the present invention have the following highlights:

[0079] 1. Real-time detection and response:

[0080] Flink's real-time processing capabilities enable FastFlood detection to analyze network traffic data in real time, and promptly detect and respond to traffic anomalies. This is crucial for protecting the network from threats such as flood attacks.

[0081] 2. Flexible window operation:

[0082] Flink provides a variety of windowing operations, such as rolling windows, sliding windows, and session windows, etc. These functions allow users to flexibly group, aggregate, and calculate network traffic data in FastFlood detection according to actual needs, thereby more accurately identifying abnormal behaviors.

[0083] 3. High Scalability:

[0084] Flink supports distributed operation and can be easily expanded to thousands of nodes. This enables the FastFlood detection device to process large-scale network traffic data and meet the needs of high concurrency and large data volumes.

[0085] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0086] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A FastFlood detection method based on the Flink stream processing engine, characterized in that: The method includes: The Flink stream processing engine collects traffic data from various data sources in real time and distributes it to corresponding sliding time windows. The sliding time window allows the window to slide on the time axis and capture traffic changes by setting the window size and sliding step size. In each slide-out window, statistical analysis is performed on the traffic data; The statistical analysis results are compared with the preset thresholds to determine whether they present typical characteristics of a pulse-type DDoS attack; if a pulse-type DDoS attack is identified, the corresponding protection mechanism is triggered.

2. According to the FastFlood detection method based on the Flink stream processing engine according to claim 1, it is characterized in that: For the sliding window traffic statistics data that are lower than the preset threshold, a label is added, and the mean value of the sliding window traffic data is used as the baseline traffic.

3. The FastFlood detection method based on the Flink stream processing engine according to claim 1, characterized in that: For the traffic statistics of the sliding window that are higher than the preset threshold, if the traffic statistics are more than three times lower than the threshold, the normal detection process is entered to compare the average traffic in the sliding window with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event; If the traffic statistics are more than three times higher than the threshold, the Fastflood detection process is entered. If the traffic in the slide-out window is three times higher than the baseline traffic of the previous two slide-out windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

4. A FastFlood detection device based on the Flink stream processing engine, characterized in that: The device includes: The traffic collection and distribution module is used by the Flink stream processing engine to collect traffic data from various data sources in real time and distribute it to the corresponding sliding time window. The sliding time window allows the window to slide on the time axis and capture traffic changes by setting the window size and sliding step size. Traffic statistics analysis module, used to perform statistical analysis on traffic data in each slide-out window; The attack identification module is used to compare the statistical analysis results with the preset threshold and determine whether the typical characteristics of a pulse-type DDoS attack are present; if a pulse-type DDoS attack is identified, the corresponding protection mechanism is triggered.

5. The FastFlood detection device based on the Flink stream processing engine according to claim 4, characterized in that: For the sliding window traffic statistics data that are lower than the preset threshold, a label is added, and the mean value of the sliding window traffic data is used as the baseline traffic.

6. The FastFlood detection device based on the Flink stream processing engine according to claim 4, characterized in that: For the traffic statistics of the sliding window that are higher than the preset threshold, if the traffic statistics are more than three times lower than the threshold, the normal detection process is entered to compare the average traffic in the sliding window with the preset threshold. If it is greater than the preset threshold, an alarm is triggered and recorded as a normal attack event; If the traffic statistics are more than three times higher than the threshold, the Fastflood detection process is entered. If the traffic in the slide-out window is three times higher than the baseline traffic of the previous two slide-out windows, it is determined to be a pulse DDoS attack and recorded as a Fastflood attack event.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 3.

Images