Techniques to detect advanced application layer flood attack tools

Abstract

A method and system are provided for detecting an application layer flood denial of service (DDoS) attack initiated by an attacker using an advanced application layer flood attack tool. The method processes application layer transactions received during a current time window to detect rate-based anomalies in traffic directed to a protected entity; processing the received application layer transaction to determine a rate-independent anomaly based on a plurality of application attributes (AppAttached) observed in the application layer transaction received during a current time window, where the rate-independent anomaly is determined based on a consecutive update baseline of AppAttached, where the AppAttached represents an application behavior of a protected entity modeled based on the application layer transaction; determining whether an application layer flood DDoS exists in the current time window based on the detected rate-based anomaly and the detected rate-independent anomaly; and triggering a mitigation action when the application layer flood DDoS exists.

Description

Technical Field

[0001] The present disclosure generally relates to techniques for detecting application layer Denial of Service (DoS) based attacks. Background Art

[0002] Today, online businesses and organizations are vulnerable to malicious attacks. Recently, a wide range of attack techniques and tools have been used to carry out cyberattacks, targeting the information maintained by online businesses, their IT infrastructure, and actual service availability. Hackers and attackers are constantly evolving their attack strategies to cause irreversible damage and overcome currently deployed protection mechanisms.

[0003] A popular type of network attack is a DoS / DDoS attack, which attempts to render a computer or network resource unavailable or idle. Common techniques for executing a DoS / DDoS attack involve saturating a target victim resource (e.g., a computer, web server, API server, web application, etc.) with a large number of external requests or traffic. As a result, the target victim becomes overloaded and is unable to allocate resources and properly respond to legitimate traffic. When an attacker sends many application requests or other requests to its victim service or application, each victim resource is affected by the DoS attack. DDoS attacks are performed by taking control of many machines and other entities connected to the internet and directing them to attack as a group, thereby increasing the destructive potential of the attack.

[0004] One type of DDoS attack is known as an "application layer DDoS attack." This is a form of DDoS attack in which the attacker targets application-layer processes, resources, or entire applications. The attacker overuses specific functions or features of the application, rendering them inoperable. This causes the application to become unresponsive to legitimate requests or even terminate or crash. A major subcategory of application-layer DDoS attacks is the HTTP flood attack.

[0005] In an HTTP flood attack, attackers manipulate HTTP requests, including GET and POST requests, as well as other unwanted HTTP requests, to attack or overload victim server, service, or application resources. These attacks are typically carried out by one or more attack tools designed to generate and send a large number of legitimate-looking HTTP requests to the victim server. HTTP requests can be plaintext or encrypted using the HTTPS protocol. The content of these requests may be randomized or pseudo-randomized to simulate legitimate web client behavior and circumvent anti-DoS mitigation measures. Examples of such tools include Challenge Collapsar (CC), Shaphyra, the Mirai botnet, the Meris botnet, HMDDoS, Blood, KillNet, Akira, Xerxes, web stress testers, and DDoS attackers (DDosSers). Attackers use a variety of methods to obfuscate their identities; attack traffic may originate from anonymous proxies, Tor nodes, IoT devices, legitimate home routers, or hosted web servers.

[0006] Recently, hackers have developed a plethora of new, sophisticated tools that are now being used in a variety of deadly, high-volume HTTP flood attacks. The need for simple, yet accurate, technical solutions to mitigate HTTP flood attacks has become both practical and urgent. Modern online services require application-based anti-DoS (Anti-DoS) solutions that can accurately detect HTTP flood attacks in real time and accurately distinguish between incoming HTTP requests from attackers and legitimate clients while maintaining extremely low false positive and false negative rates. Attackers are constantly improving their attack tools by generating legitimate-looking HTTP requests, making mitigation and more specific detection of application-level attacks extremely challenging.

[0007] Detecting HTTP flood DDoS attacks performed by these tools is a complex problem that cannot be solved with simple DDoS attack detection schemes. Distinguishing legitimate HTTP requests from malicious ones is a complex and complex task. Specifically, increasing demand for access to a web application (or resource) by legitimate users could reasonably lead to HTTP flood traffic. Such behavior could be caused by sudden spikes in access (flash crowds) or transient traffic surges (flash floods). In other words, any effective detection scheme should be able to distinguish between bursts of normal traffic and attack traffic. This complexity stems from the fact that many attack tools behave differently and generate different attack patterns. Furthermore, attack tools send HTTP requests with a legitimate structure (e.g., headers and payloads defined by the corresponding HTTP standards and following industry conventions), but some parts of their request content are randomized. For example, the values ​​of HTTP headers, random query parameters, and cookie keys and values ​​can all be randomly selected. Furthermore, due to the large number of requests (e.g., thousands or tens of thousands of requests per second) and the evolving content of the requests, coupled with the extensive use of randomization, existing DDoS detection schemes cannot effectively detect and alert HTTP flood application layer DDoS attacks, and cannot effectively distinguish them from bursty normal traffic.

[0008] Accurate HTTP flood detection methods require an accurate understanding of legitimate HTTP traffic baselines. Over the past few years, numerous new web technologies have emerged, resulting in a wide variety of legitimate web traffic behaviors, flow rates, web technologies (such as API applications and mobile apps), and network deployments. This diverse traffic makes establishing an accurate baseline of legitimate HTTP traffic challenging, making HTTP flood attack detection a significant technical challenge.

[0009] Therefore, it would be advantageous to provide an effective security solution for detecting HTTPS flood attacks. Summary of the Invention

[0010] An overview of several example embodiments of the present disclosure is provided below. This overview is provided for the convenience of the reader to provide a basic understanding of these embodiments and does not fully limit the breadth of the present disclosure. This overview is not an extensive review of all contemplated embodiments and is neither intended to identify key or important elements of all embodiments nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to a more detailed description that will be presented later. For convenience, the terms "some embodiments" or "certain embodiments" may be used herein to refer to a single embodiment or multiple embodiments of the present disclosure.

[0011] Some embodiments disclosed herein include a method for detecting an application layer flood denial of service (DDoS) attack launched by an attacker using an advanced application layer flood attack tool. The method includes processing application layer transactions received during a current time window to detect rate-based anomalies in traffic directed to a protected entity; processing the received application layer transactions to determine a rate-independent anomaly based on a plurality of application attributes (AppAttributes) observed in the application layer transactions received during the current time window, wherein the rate-independent anomaly is determined based on a continuously updated baseline of the AppAttributes, wherein the AppAttributes represent application behavior of the protected entity, the application behavior being modeled based on the application layer transactions; determining whether an application layer flood DDoS exists in the current time window based on the detected rate-based anomaly and the detected rate-independent anomaly; and triggering a mitigation action when the application layer flood DDoS occurs.

[0012] Some embodiments disclosed herein include a system for detecting an application layer flood denial of service (DDoS) attack launched by an attacker using an advanced application layer flood attack tool. The system includes processing circuitry; and a memory containing instructions that, when executed by the processing circuitry, configure a controller to: process application layer transactions received during a current time window to detect rate-based anomalies in traffic directed to a protected entity; process the received application layer transactions to determine rate-independent anomalies based on a plurality of application attributes (AppAttributes) observed in the application layer transactions received during the current time window, wherein the rate-independent anomalies are determined based on a continuously updated baseline of the AppAttributes, wherein the AppAttributes represent application behavior of the protected entity, the application behavior being modeled based on the application layer transactions; determine whether an application layer flood DDoS attack exists in the current time window based on the detected rate-based anomalies and the detected rate-independent anomalies; and trigger a mitigation action when the application layer flood DDoS attack occurs. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the appended claims.The foregoing and other objects, features and advantages of the invention will become readily apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0014] Figure 1 is a diagram for describing various embodiments for detecting HTTP flood attacks according to some embodiments.

[0015] Figure 2 is a flowchart illustrating a workflow of a detection system according to some embodiments.

[0016] Figure 3 is the structure of a window buffer or a baseline buffer according to an embodiment.

[0017] Figure 4 FIG. 4 shows a distribution density function of query arguments types of AppAttributes according to an embodiment.

[0018] Figure 5A 、 Figure 5B and Figure 5C is a window buffer used to demonstrate the process of updating a baseline according to an embodiment.

[0019] Figure 6 is a flow chart illustrating a method for detecting HTTP flood attacks according to an embodiment.

[0020] Figure 7 is a flow chart describing a method for detecting rate-based anomalies according to an embodiment.

[0021] Figure 8 is an example flow chart describing a method for detecting rate-independent anomalies according to an embodiment.

[0022] Figure 9 is a block diagram of an apparatus for performing the disclosed embodiments.

[0023] Figure 10A 、 Figure 10B 、 Figure 10C and Figure 10D The establishment and updating of baseline and window buffers according to an embodiment is shown. DETAILED DESCRIPTION

[0024] The embodiments disclosed herein are merely examples of many possible advantageous uses and implementations of the innovative concepts presented herein. In general, the statements in this application specification do not necessarily limit any of the various claimed inventions. Furthermore, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be plural, and vice versa, without loss of generality. In the accompanying drawings, like numbers refer to like parts throughout the several views.

[0025] Various disclosed embodiments include a method for detecting HTTP flood DDoS attacks. The disclosed method distinguishes between malicious traffic and legitimate (burst) traffic to achieve efficient detection of HTTP flood attacks. This detection is based on comparing a baseline distribution of application attributes (AppAttributes) learned during normal times with the distribution of AppAttributes measured during an attack.

[0026] AppAttributes represents the application behavior presented in the input HTTP requests and output HTTP responses of the application. In one embodiment, an HTTP flood attack period is triggered when an indication of a significant increase in the request rate and an indication of a significant change in application behavior appear in the AppAttributes distribution. The baseline of AppAttributes models the legitimate application behavior of the protected entity. The disclosed method can be performed by a device deployed in a traffic line, or other deployment can enable traffic observation during normal periods and attack periods by a device implementing the disclosed method. Various disclosed embodiments will be described with reference to HTTP flood DDoS attacks and / or HTTPS flood DDoS attacks, but the technology disclosed herein can be used to characterize flood DDoS attacks generated by other types of application layer protocols.

[0027] The disclosed embodiments enable rapid detection of HTTP flood DDoS attacks while enabling the proper operation of protected entities during burst access events. It should be understood that during such attacks or when burst access occurs, the number of requests directed to protected entities (e.g., web applications, API applications, etc.) can be millions to tens of millions of requests per second. Therefore, human operators are unable to process this information and provide an indication of whether the received or observed traffic is legitimate.

[0028] Figure 1 1 is a schematic diagram 100 illustrating various embodiments for detecting HTTP flood attacks according to some embodiments. HTTP flood attacks are a type of DDoS network attack. In schematic diagram 100, client device 120 and attack tool 125 communicate with server 130 via network 140. To illustrate the disclosed embodiments, client device 120 is a legitimate client (operated by a real legitimate user or other legitimate web client entity), attack tool 125 is a client device (e.g., operating as a bot in a botnet), and server 130 is a "victim server," i.e., the attacked server can be any type of web-based application, API-based application, etc.

[0029] The legitimate client 120 can be a web browser or other type of legitimate web application client or network user agent executed on a computing device such as a server, mobile device, tablet computer, IoT device, laptop, PC, TV system, smart watch, etc.

[0030] Attack tool 125 conducts malicious attacks on victim server 130, specifically HTTP and / or HTTPS flood attacks. Attack tool 125 generates and sends "legitimate-looking" HTTP requests with the goal of overwhelming the resources of its victim, a legitimate online server (such as victim server 130). The HTTP requests generated by the attacker have the correct structure and content required by the HTTP protocol and its common practices, and because of this, these requests appear "legitimate" even though they are generated by an attacker with malicious intent. When organizing the attack, the attacker uses a large amount of randomization or pseudo-randomization. In some cases, the attacker constructs a large set of different "legitimate" requests and randomly selects HTTP requests as attack requests to send within a selected time period. It should be noted that the attacker generates a large number of different HTTP requests so that they can evade fingerprinting and mitigation by simple web filtering or other existing attack mitigation methods. Therefore, it is not feasible for human operators to detect this attack.

[0031] Attack tool 125 can be an HTTP flood attack tool that can be deployed in the following ways: as a botnet; as an HTTP flood attack tool using a web proxy; without using a web proxy (for example, directly using a hosting server to generate the attack); or using a legitimate home router running as a web proxy. This allows the attacker to disguise their Internet identity and, in this way, eliminates any possibility of simply identifying their HTTP requests by simply knowing their source IP address in advance. Attack tool 125 can also be deployed as a web stress tester (WEBstresser), DDoS attack initiator (DDoSers) and other "rental DDoS attack (DDos for hire)" form of attack.

[0032] The attack tool 125 generates requests with legitimate structure and content. To achieve a "legitimate structure," the attacker-generated HTTP request may include one or more legitimate URLs within the protected application (or victim server 130), a set of common HTTP headers, and contain one or more query parameters. The attack tool 125 may always include specific HTTP headers (standard or non-standard) or query parameters in the HTTP requests it generates, or may randomly decide to include or exclude them in each generated request or request set. The attack tool 125 may also randomly decide the HTTP method and URL for each HTTP transaction sent to attack the victim server.

[0033] The requests generated by the attack tool 125 may also contain legitimate and different content. In order to make the requests it generates appear "legitimate," the HTTP requests generated by the attack tool may have HTTP headers with legitimate values ​​(e.g., a user agent may be randomly selected from a predefined list of legitimate user agents, and a referrer may be randomly selected from a predefined list of legitimate common websites (e.g., facebook.com, google.com)).

[0034] These overall operations of attack tool 125 can result in a set of tens of thousands or even millions of different attacker HTTP requests. The attacker uses randomization to select the actual HTTP request sent to the victim in each request transmission. Therefore, simply identifying millions of different attacker requests "as is" would be a very tedious and almost impossible task. It is worth noting that there are many variations and variants of these tools, but they still follow similar operations, so the HTTP requests they generate are as described above. Advanced attack tools are designed to bypass simple layer 7 filtering mitigations by generating a large number of different and "legitimate-looking" HTTP requests. In this way, no major and frequent set of HTTP requests can be simply characterized as being issued by attack tool 125.

[0035] It should be noted that an attacker can create an HTTP flood attack using a lower level of randomization, for example, by consistently using the GET method as the HTTP method, or by not using any type of randomization. The disclosed embodiments are intended to detect HTTP flood attacks regardless of the level of randomization implemented by the attack tool used.

[0036] Compared to attacker requests, legitimate client device(s) 120 generate requests that are more diverse in structure and content. Legitimate client HTTP requests may have more HTTP headers, both standard and non-standard, redirect to multiple URLs within protected application 130, have more key-value pairs in cookies, use more query parameters, and so on. All of these attributes appear in HTTP requests and responses, depending on the application's activity (e.g., logging in, simply browsing to the homepage or login page, going to a specific area of ​​the application, etc.). Based on the higher diversity and unique content distribution of legitimate requests, it is possible to detect HTTP flood attacks and distinguish them from legitimate sudden traffic spikes.

[0037] The attack tool 125 is intended and designed in its fundamental sense to give the attacker the ability to overwhelm as many online WEB assertions as possible. This is because such an attack tool is neither directed nor targeted to attack a specific target. In most cases, the HTTP requests carefully crafted and generated by the attack tool are not entirely targeted at the attacked application. Therefore, the attacker's requests may appear to be legitimate, but most do not necessarily include any application-specific attributes. Specific application attributes (referred to herein as AppAttributes) can be any part of the input HTTP request and their corresponding HTTP response that is unique to a specific protected application or victim server 130, and thereby can identify the legitimate "application behavior" of the protected application or victim server 130.

[0038] AppAttributes can be understood as "things that can be introduced only by legitimate clients 120 and not by, for example, attack tools 125." The present invention leverages this capability to distinguish legitimate traffic surges or sudden access or traffic spikes from attacker-induced floods. Attack tools have little knowledge of legitimate application behavior, so the "closeness" of attack traffic to normal, legitimate application behavior is low. However, sudden access spikes are legitimate and therefore have legitimate properties, making their behavior somewhat "close" to normal traffic.

[0039] It should be noted that the embodiments disclosed herein apply when multiple attack tools are simultaneously executing attacks against the victim server 130. Similarly, a large number of legitimate client devices 120 can be running simultaneously so as to be delivered along with the services offered by the server 130. One or more client devices 120 and one or more attack tools 125 can simultaneously reach the victim server 130. The network 140 can be, but is not limited to, a local area network (LAN), a wide area network (WAN), the Internet, a content delivery network (CDN), a cloud network, a cellular network and a metropolitan area network (MAN), a wireless network, an Internet of Things network, or any combination thereof. Similarly, the victim server 130 can be constructed from one or more servers.

[0040] According to the disclosed embodiment, a detection system 110 is deployed between a client device 120, an attack tool 125, and a victim server 130. This deployment can be an always-on deployment, or any other deployment that enables the detection system 110 to observe incoming HTTP requests and their corresponding responses during normal times and during active attack periods. The detection system 110 can be connected to a mitigation system 170 configured to characterize and mitigate attack traffic. An example embodiment of the mitigation system 170 is discussed in U.S. Patent Application No. 17 / 456,332, assigned to a common assignee.

[0041] The detection system 110 and the victim server 130 can be deployed in a cloud computing platform and / or on-premise so that they are set up together or as a combination. The cloud computing platform can be, but is not limited to, a public cloud, a private cloud, or a hybrid cloud. Example cloud computing platforms include Amazon Web Services (AWS), Cisco MetaCloud, and others. Microsoft Azure Google Cloud Platform, etc. In one embodiment, when installed in the cloud, the detection system 110 can be run as SaaS or as a managed security service provided as a cloud service. In one embodiment, when installed locally, the detection system 110 can be run as a managed security service.

[0042] In one embodiment, detection system 110 and / or mitigation system 170 are integrated into a WAF (Web Application Firewall) device or a dedicated Layer 7 DDoS detection and mitigation device. In another embodiment, detection system 110 and / or mitigation system 170 are integrated into any form of web proxy (reverse proxy, forward proxy, etc.) or web server. In yet another embodiment, detection system 110 and / or mitigation system 170 can be integrated into a web cache system (such as a CDN, etc.).

[0043] Victim server 130 is an entity protected from malicious threats (protected entity). Server 130 can be a physical entity or a virtual entity (e.g., a virtual machine, a software container, a serverless function, an API gateway, etc.). Victim server 130 can be a web server (e.g., an original server, a compromised server, a compromised online web server, a compromised web application, an API server, a mobile application, etc.). Protected entities can include applications, services, processes, etc. executed on server 130.

[0044] According to the disclosed embodiment, the detection system 110 is configured to detect changes or rapid increases in the transmission rate or RPS requests per second of traffic directed to the server 130 during normal times and during the initiation of active attacks. Following such detection, an indication that a potential DDoS attack is ongoing can be provided. Indications of a potential attack may also be caused by sudden spikes in access traffic, as the transmission rate of packets and / or HTTP requests increases. The detection indication may be a rate-based anomaly that may indicate a transition from normal period conditions. This transition may be from normal times (normal traffic, when there is no active attacker sending traffic to the victim) to a potential attack, or to a sudden spike in access conditions.

[0045] In one embodiment, rate-based anomalies are determined by rate-based traffic parameters. Examples of such parameters include, but are not limited to, requests per second (RPS) at layer 7 or HTTP requests per second. In another embodiment, such parameters may include layer 3 attributes (e.g., bits per second and packets per second), layer 4 attributes (e.g., TCP SYNs per second and TCP connections per second (new and existing connections)), and the like. Rate-based anomalies are characterized by a sharp increase in traffic rate attributes within a short period of time. To this end, a rate-based anomaly threshold is defined as an RPS level (or other layer 3 or layer 4 attribute) representing an RPS that is considered a legitimate or normal traffic level, such that when the threshold is exceeded, it indicates that the traffic state may be transitioning from a normal period to a potential attack period, or that an RPS anomaly exists. When the RPS fails to meet the anomaly threshold, it indicates that a rate-based anomaly may exist.

[0046] In an example embodiment, an anomaly threshold based on the rate is calculated for each time window and protected entity by using a combination of alpha filters configured in short (e.g., 1 hour) and medium (e.g., 8 hours) durations. Other durations of alpha filters are also applicable. The RPS baseline and anomaly threshold will be discussed below. It should be emphasized that the comparison with the RPS anomaly threshold is performed for each time window. For example, such a time window is set to 10 seconds. The time window allows for stable detection of attacks that occur and run for a short period of time.

[0047] According to the disclosed embodiment, in order to distinguish between legitimate sudden traffic peaks and attack traffic floods, the detection system 110 is configured to examine application transactions. A transaction is a request (e.g., an HTTP request) sent to the victim server 130 and a corresponding response to the request (e.g., an HTTP response) sent by the victim server 130 in response to the HTTP request it received. The detection system 110 is configured to analyze the received transactions and determine whether the rate-independent parameters in the transactions indicate normal or abnormal behavior of the application. Abnormal behavior indicates an HTTP flood DDoS attack or is a rate-independent anomaly. In contrast, normal behavior indicates a legitimate sudden access peak situation.

[0048] In one embodiment, the rate-independent parameters are AppAttributes, whose baselines are developed during normal times and monitored during potential attack periods. In another embodiment, the AppAttributes uniquely characterize legitimate behavior in two categories: HTTP requests sent to a protected entity, and corresponding HTTP responses sent by the protected entity. In yet another embodiment, the AppAttributes can be of different types. AppAttribute types can be pre-selected, and the same set of types can be used when developing baselines and detecting potential anomalous behavior.

[0049] As discussed in detail below, the detection system 110 is configured to warn of a potential HTTP flood DDoS attack based in part on a comparison between an AppAttributes buffer generated for a current time window (hereinafter referred to as the "window buffer") and a baseline AppAttributes buffer calculated over a past time window (hereinafter referred to as the "baseline buffer"). Since requests generated by the attack tool do not exhibit a near-baseline AppAttributes distribution (unlike the AppAttributes distribution of legitimate clients), it is expected that a different AppAttributes distribution will be displayed during the attack period window. Taking advantage of this fact, the operation of the detection system 110 will be able to effectively distinguish between sudden access spike conditions and attack scenarios. It should be noted that the baseline AppAttributes represent the legitimate or normal behavior of the victim application. The unique baseline AppAttributes distribution of the victim application 130 learned during normal times describes the unique behavior of the protected application that occurs in its requests and responses.

[0050] exist Figure 1 In the example deployment shown, the detection system 110 can be connected in series with traffic between the client device 120 and the attack tool 125 toward the victim server 130. In this deployment, the detection system 110 is configured to process incoming traffic from the client device 120 and the attack tool 125, as well as process outgoing traffic or the return path from the victim server 130 back to the device 120 and, in an attack scenario, back to the attack tool 125.

[0051] In another configuration, the detection system 110 can be an always-on deployment. In such a deployment, the detection system 110 and the mitigation system 170 are part of a cloud protection platform (not shown). In yet another embodiment, a hybrid deployment may be proposed, wherein the detection system 110 is deployed on the cloud, while the mitigation system 170 is deployed locally near the victim server 130.

[0052] It should be noted that although Figure 1A client device 120, an attack tool 125, and a victim server 130 are described in the foregoing, but for simplicity only, the embodiments disclosed herein can be applied to multiple clients and servers. The clients can be located in different geographical locations. The servers can be part of one or more data centers, server frameworks, private clouds, public clouds, hybrid clouds, or a combination thereof. In some configurations, the victim server 130 can be deployed in a data center, a cloud computing platform, or a local organization, etc. The cloud computing platform can be a private cloud, a public cloud, a hybrid cloud, or any combination thereof. In addition, Figure 1 The deployment shown may include a content delivery network (CDN) connected between the client 120 and the attack tool 125 and connected to the server 130. In one embodiment, the detection system 110 and the mitigation system 170 are deployed as part of the CDN.

[0053] The detection system 110 may be implemented in software, hardware, or any combination thereof. The detection system 110 may be a physical entity (example block diagrams are discussed below) or a virtual entity (eg, a virtual machine, a software container, a micro-entity, a function, etc.).

[0054] Figure 2 An example flow chart 200 illustrating the operation of a detection system according to some embodiments is shown. Detection of HTTP flood attacks is performed during predefined time windows, where an indication can be provided during each window. A baseline is deployed over time based on transactions received during a normal period. There are two detection paths: rate-based anomaly detection (labeled 210) and rate-independent anomaly detection (labeled 220). In one embodiment, rate-based features are features related to traffic that tend to increase during periods of increased traffic. In another embodiment, rate-independent features are features related to traffic that do not increase during periods of increased traffic.

[0055] The rate-based path 210 will be discussed with reference to a specific rate parameter, namely, the total number of requests per second (RPS) arriving at the victim server.

[0056] Transactions 205 received by the detection system 110 are fed into two paths for processing. This may include receiving the traffic by a web proxy to decrypt the traffic and, when using HTTPS, present the incoming request and its corresponding response. Figure 2In the embodiment described in

[15] , the web proxy is part of the detection system 110. In another embodiment, the web proxy can be external to the detection system 110 (e.g., a CDN, a network server, etc.). In the latter case, the detection system 110 is configured with a predefined interface to the web proxy to further receive web transactions to or from the victim server 130. This processing is performed in each time window, but is continuous during the operation of the system 110. The duration window is a few seconds, for example, 10 seconds long.

[0057] In the rate-based path, an attempt is made to detect potential attacks due to an increase in the number of transactions and their rate. Specifically, at block 210, the RPS for the current window (n+1, where 'n' is an integer) is calculated. The value of RPS[n+1] is calculated as follows:

[0058] WinRPS[n+1] = total number of requests / time window duration Equation 1

[0059] Wherein, the total number of requests is the number of requests received within the time window [n+1]. Then, the short baseline and the mid baseline are calculated in blocks 212 and 213, respectively. The baselines SrtBL[n] and MidBL[n] are the mean and variance values ​​when different alpha filters are applied to the mean and variance values ​​of the RPS. In one embodiment, the mean and variance (var) values ​​can be calculated using the alpha filter as follows:

[0060] RPS_mean [n+1] =RPS_mean [n] (1–α)+RPS [n+1] α Equation 2

[0061] RPS_var [n+1] =RPS_var [n] (1–α)+RPS [n+1] 2 α+RPS_mean [n] 2 (1–α)–RPS_mean [n+1] 2

[0062] Equation 3 Among them, the "α" value of the "short" baseline (SrtBL[n]) is smaller than the α value of the "medium" baseline (MidBL[n]).

[0063] The SrtBL[n] and MidBL[n] baselines are obtained by using RPS_mean [n+1] and RPS_var [n+1] Definition: Accordingly, RPS_var is the variance of the measured RPS.

[0064] In one embodiment, a short baseline is used to track hourly incremental changes in RPS (particularly for 24x7 legitimate web traffic patterns), and thus averaged over the last hour using alpha fitting. A medium ("mid") baseline is used to track bursty RPS traffic patterns. In one embodiment, the duration of the alpha filter can be configured to dynamically adapt to other types of patterns and attack detection sensitivity. In another embodiment, the detection system 110 supports multiple alpha filters to cover other types of patterns and multiple integration periods or averaging periods.

[0065] At the RPS anomaly detection block 214, each of the SrtBL[n] and MidBL[n] baselines is used to construct an RPS anomaly threshold (TH). During normal times, at the end of each time window (e.g., every 10 seconds), the RPS anomaly threshold (TH) calculated for the previous time window is compared with the current window RPS to detect RPS anomalies. In one embodiment, for window n+1, the threshold for RPS anomaly can be defined as follows:

[0066] RPS abnormal TH[n+1]=max{(RPS abnormal TH) α-short ,(RPS abnormal TH) α-mid Equation 4

[0067] in:

[0068] (RPS abnormal threshold) α = max{(averageFactor)*RPS_mean,RPS_mean+STD_Factor*

[0069] SQRT(RPS_var)} Equation 5

[0070] Where RPS_mean calculated in Equation 2 is the mean value of RPS. RPS_var is the variance of RPS calculated in Equation 3. α (short and medium), MAF or minimum attack factor, STD_Factor values ​​are predefined. The minimum attack factor is defined as the minimum multiple of the attacker's RPS relative to the legitimate RPS level to be considered an HTTP flood attack. For example, for MAF=3, only window RPS amounts that are higher than 3 times RPS_mean can be considered RPS anomalies. Another possible case of RPS anomaly is when the window RPS is higher than the baseline and STD_Factor multiplied by the calculated standard deviation (square root of RPS_var). It should be noted that RPS is just an example of a rate-based parameter and other parameters such as Layer 3 and Layer 4 attributes can be used in this article.

[0071] The value of "α" is preconfigured, where higher values ​​of "α" give more weight to the current window RPS or WinRPS[n+1] value compared to the previously received RPS. Lower values ​​of "α" give more weight to past RPS values ​​compared to the RPS received in the current window, which means a slower reaction to relatively rapid (hourly) changes in RPS. In one example, for the intermediate filter, the short "α" value is set to 0.005, while the medium "α" value is set to 0.0008.

[0072] When the value of the current window RPS WinRPS[n+1] exceeds the RPS abnormality threshold [n+1], an abnormality based on the RPS rate is determined.

[0073] In rate-independent path 220, the current time window is checked for deviations from the AppAttributes baseline behavior. According to the disclosed embodiment, each protected application is allocated a combination of two buffers: baseline and window. Each set of baseline and window buffers contains a buffer for each AppAttributes type. Examples of AppAttributes types are provided herein.

[0074] Application attributes (or AppAttributes) are attributes (primarily keys) in HTTP request and response headers that uniquely identify legitimate behavior of a victim server application. This behavior can be inferred by analyzing the application's legitimate HTTP requests and their corresponding HTTP responses. In one embodiment, AppAttributes can be described as a "feature vector" that includes several AppAttributes types.

[0075] In another embodiment, the AppAttributes type can include the key of a non-standard HTTP header (or an unknown HTTP header) in an HTTP request, the key of each cookie in the cookie header in an HTTP request, the key of each query parameter in the request URL, the HTTP request size, the HTTP response size, the HTTP response status code, attributes of the source client IP address (e.g., geographic location (e.g., country and city) and ASN assignment, etc.) appearing in the X-Forwarded-For header or Layer 3 IP header, the user-agent value, the TLS fingerprint value calculated from the client hello message in the incoming traffic, the distribution of durations (in seconds) between HTTP requests and their corresponding HTTP responses, and the key of each cookie in the cookie header of the HTTP response set. When selecting an AppAttributes type, it is important to select an application attribute whose value and distribution uniquely identify legitimate traffic (as seen in legitimate traffic) but cannot be introduced by an attack tool. This is because transactions generated by an attack tool are generic, as the attack tool does not target a specific victim server application. Therefore, it is assumed that requests crafted by an attack tool will not have an AppAttributes distribution similar to the distribution of legitimate client requests. In one embodiment, AppAttribute provides a metric to differentiate between sudden access spike conditions and attack scenarios.

[0076] Figure 3 An example AppAttributes buffer 300 is shown as a window buffer or baseline buffer. The buffer 300 includes an AppAttribute key value field (311), an occurrence (occ) field (312), a weight field (313), and a timestamp / age field (314).

[0077] The AppAttribute key value field (311) includes the name of the key value (e.g., query parameter (args) name, non-standard HTTP header name, cookie name), HTTP request size range, HTTP response size range, HTTP response status code (e.g., 200), source IP address attribute (geo, ANS, etc.), user agent value, TLS fingerprint attribute, set-cookie cookie name, etc. The Occ field (312) indicates the number of occurrences or the average number of occurrences of the AppAttributes specific key value over a period of time. The Weight field (313) includes the frequency of occurrence of the AppAttributes specific key value relative to other AppAttributes (of the same type) that appear in the same HTTP request received. For example, the weight value can be set to:

[0078] Weight = 1 / (Total number of AppAttributes present in the transaction) Equation 6 The timestamp field (314) indicates the last time the AppAttribute key value was observed in the received HTTP request.

[0079] For each AppAttributes type, multiple buffers are created, one buffer for each AppAttribute key value. The buffers are maintained in the memory of the system 110, and in one embodiment, the number of buffers is limited to allow secure storage management and to prevent an attacker from exhausting the memory of the detection system 110 during an active attack (when a large number of new application attribute (AppAttribute) key values ​​appear). In many cases, the attack tool does not know the complete list of legal AppAttributes of each type, so in many cases, the attack tool 125 uses randomized AppAttributes key values. During an active volumetric attack, it is expected that a very large number of new AppAttributes keys will appear, tens of thousands or even millions. The baseline buffer is constructed as a set of AppAttributes buffers 300, such as Figure 3 As shown. The baseline represents a histogram (statistical distribution) of the number of occurrences and / or weights of the AppAttributes key values ​​from the first to the last time window. In one embodiment, a set of buffers in the baseline buffer are pre-set with key values. In one embodiment, such AppAttributes key values ​​can be learned during the learning phase of the system 110 and thereafter learned by continuous learning during normal periods. The baseline includes adding new key value buffers to existing buffers observed during normal periods. The baseline buffer is updated every 10 second window (only when no active attack is detected) based on the data aggregated on the window buffer.

[0080] It should be noted that the baseline represents the behavior of AppAttributes appearing in protected applications during normal periods. In one embodiment, the normal period baseline is compared with the AppAttributes appearance behavior during the abnormal window, thereby achieving rate-independent application abnormal behavior detection.

[0081] The window buffer is also constructed as Figure 3A set of AppAttributes buffers 300 is shown, and it represents only a histogram of the number of AppAttributes occurrences for the last time window. That is, the window buffer matches the baseline buffer in structure. For each AppAttributes type, the window buffer is divided into two sections. One section contains the AppAttributes keys pre-assigned to the AppAttributes from the baseline buffer. The second section pre-allocated AppAttributes buffers to be allocated for new AppAttributes key values. The structure of the buffers disclosed herein ensures that previously observed AppAttributes are taken into account and prevents newly arriving AppAttributes from overwhelming the window buffer. Updates to the window AppAttributes buffers include: parsing input transactions (requests and responses) to identify the AppAttributes and adding or updating the key values ​​in the corresponding buffers; incrementing the occ field (312) and, if applicable, updating the weight field (312) based on equation 6 above; and setting the timestamp field (313) to the timestamp of the current window time. Updates to the window buffers are performed for each time window. In one embodiment, each buffer is updated by each input HTTP transaction. In another embodiment, the corresponding buffer is updated by the sampled input HTTP transaction. The sampling may be for 1 out of N transactions, the first N transactions in a time window, etc. In yet another embodiment, the sampling rate N may be different for normal period scenarios and attack period scenarios to better adapt to the number of HTTP requests sent to the protected entity.

[0082] In one embodiment, the baseline buffer is updated as long as there is no RPS anomaly and no active HTTP flood attack is detected. In addition, an aging mechanism is applied to the baseline buffer, wherein values ​​in the buffer that exceed an aging threshold (based on their timestamp values) are removed.

[0083] Figure 10A The structures of the baseline buffer and the window buffer and the process of establishing and updating these buffers are shown. For ease of description, a group of buffers for a single AppAttributes type (eg, query parameter key, cookies key, etc.) is shown in FIG10 .

[0084] The window buffer 1100 includes two sections: a baseline value section 1110 and a first value section 1120. Each of these sections is constructed from a set of buffers, each of which is constructed from a set of fields, such as Figure 3As shown in Figure 11, the values ​​in segment 1110 are copied from baseline buffer 1130 at the beginning of each window. Segment 1120 records new values ​​observed for the first time during the time window, i.e., values ​​not present in the baseline segment. The size of segment 1120 is limited so as not to saturate system resources during an active attack. That is, segment 1120 is configured to record or buffer up to a predefined number of new values.

[0085] The baseline buffer 1130 includes an AppAttributes baseline for a specific AppAttributes type. The baseline buffer 1130 is composed of a set of buffers, each of which is composed of a set of fields, such as Figure 3 As shown. The values ​​in baseline buffer 1130 are updated at the end of each time window to include the values ​​buffered in window buffer segments 1110 and 1120 throughout the last time window. It should be noted that baseline buffer 1130 is only updated if no attacks were detected during the time window. Baseline buffer 1130 is aggregated using AppAttributes values ​​learned during the learning phase. However, once the system is deployed and running, system baseline buffer 1130 is continuously updated at the end of each time window using values ​​observed during normal times. In one embodiment, the time window is set to 10 seconds.

[0086] Figure 10B It is shown that the window buffer 1100 is updated at the end of each time window using the AppAttributes values ​​learned and aggregated in the baseline window. Figure 10C It is shown that the window buffer 1100 is updated during each time window according to the values ​​present in the input HTTP request and response.It is assumed here that a new AppAttributes value is observed within a time window. Figure 10D The figure shows that at the end of each time window in which normal activity is observed, the baseline buffer 1300 is updated with the AppAttributes present in the web transactions received during the time window that just ended. For AppAttributes that exist in the baseline, their occ and weight fields need to be updated. For new AppAttributes, the window occ and weight fields should be allocated for the new buffer in the baseline buffer.

[0087] refer to Figure 2At block 221, a window buffer (WinAppAttBuf[n+1]) is calculated at the end of the window based on the transactions received at time window (n+1). The WinAppAttBuf[n+1] buffer is a histogram that is fed to block 222 to update the baseline buffer with the current window (n+1) AppAttribute values ​​and fed to block 223 to determine if there is a rate-independent anomaly based on the AppAttributes. The actual update of the baseline buffer is only performed when no active attack is detected, as it is undesirable to allow an attacker to affect the AppAttributes normal baseline value in any way.

[0088] Specifically, the determination of an AppAttributes anomaly is based on attack proximity, which indicates how close the WinAppAttBuf[n+1] window buffer is statistically to the BLATTBuf[n] baseline buffer. Each AppAttributes buffer can be represented as a single bar, and the entire buffer of a particular type can be represented as a histogram, where the AppAttribute key value is the X-axis of the histogram and the number of occurrences or weights represents the Y-axis of the histogram. From these histograms from the window and baseline buffers, an AppAttributes probability density function or distribution can be calculated to represent the probability of each AppAttribute occurring. For example, Figure 4 , where the X-axis shows the AppAttributes key value and the Y-axis shows the probability of encountering each AppAttribute in a request. Bar 410 is the baseline buffer and bar 420 is the window buffer.

[0089] In one embodiment, the probability (Pr) is determined based on the weight values ​​in the buffer. In one embodiment, each AppAttribute (e.g., AppAtt i The probability that ) is located at the i-th buffer in the buffer can be calculated as follows:

[0090]

[0091] In one embodiment, the attack proximity represents the statistical distance between the AppAttributes window distribution and the AppAttributes baseline distribution for each AppAttributes type.

[0092] In one embodiment, for each AppAttributes type, the statistical distance between the AppAttributes window distribution and the AppAttributes baseline distribution can be calculated using methods such as total variation distance, Bhattacharya coefficient, Kolmogorov metric, Kantorovich metric, or other methods for calculating the average of probability metrics.

[0093] In one embodiment, the attack proximity is calculated using the Total Variation method. The attack proximity is calculated as the sum of the total variation measures calculated for each of the various AppAttributes types. The total variation measure for an AppAttributes type is the sum of the measured distances between the window probability of a particular type and the baseline probability. This measured distance is Figure 4 Marked as D KeyVal#i (i=1…,r, r is the key value or the number of buffers in the buffer.) The AppAttributes total variation measure can be calculated as follows:

[0094] AppAttribyte Total Variation Measure = ∑ AppAtt#i D AppAtt#i Equation 8

[0095] in:

[0096] Metric distance D AppAtt#i It can be calculated as follows:

[0097] D AppAtt#i =|BaselineP AppAtt#i -WindowP AppAtt#i |

[0098] Among them, BaselineP AppAtt#i and WindowP AppAtt#i are the baseline probability and window probability of AppAttributes i, respectively. In one embodiment, the AppAttributes total variation metric can take values ​​ranging from 0 (the two distributions are identical) to 2 (the two distributions are completely different from each other). In one embodiment, the total variation is used to calculate each type of AppAttributes metric. In another embodiment, the AppAttributes metric is calculated using other probability metrics, such as the Bhattacharyya index.

[0099] Next, attack proximity is calculated as the sum of all AppAttribute total variation measures across all AppAttributes types:

[0100]

[0101] In another embodiment, the attack proximity is calculated as a weighted sum of various AppAttributes types, where each type can be given a different weight to represent its importance relative to other AppAttributes types. The higher the weight, the more important the particular AppAttributes type is.

[0102] For AppAttributes types that represent size (such as HTTP request and response sizes), the histogram is set to a static range. For example, for HTTP request and response sizes, the size range is in bytes, such as fixed ranges such as 1-100; 101-1000; 1001-10000; and above 10000. For HTTP response sizes, a size of "0" represents a web transaction with no response, or an HTTP request without any corresponding HTTP response. In another embodiment, a dynamic and adaptive histogram range can be proposed to accurately represent the size distribution during normal periods.

[0103] When a particular input web transaction does not include any value for a particular AppAttributes type, the AppAttributes will be considered to have a "None" value. For example, when an HTTP request does not contain a cookie, the cookieAppAttributes key value is considered to be "None", and its Occ and Weight fields will be updated and incremented by 1 accordingly.

[0104] Back to Figure 2 At block 223, the calculated attack proximity is compared with a proximity threshold. When the attack proximity exceeds the proximity threshold, an AppAttributes exception or a rate-independent exception is set for the current window (n+1). When both the AppAttributes exception and the RPS exception are set, an HTTP flood DDoS attack is declared. When only the RPS exception is set, the increase in traffic may be due to a sudden spike in traffic.

[0105] In one embodiment, the proximity threshold is pre-configured. In another embodiment, the proximity threshold may be calculated as follows:

[0106] Proximity threshold = number of valid application parameters * total variation factor (TotalVariationFactor)

[0107] In other words, the proximity threshold is the product of the number of valid AppAttributes and a preconfigured parameter (TotalVariationFactor). Valid AppAttributes are considered to reduce false positives caused by misidentifying sudden traffic spikes as attacks. An AppAttribute is considered valid if both the window and baseline "None" value distributions are below the predefined threshold.

[0108] In one embodiment, if the "None" probability of the window or baseline distribution is less than a predefined "None threshold" (e.g., 0.5), then the AppAttributes type is considered "valid" for attack detection. If both the window and baseline "None" probabilities are above the "None threshold," then the AppAttribute is considered invalid and cannot be used as part of the attack proximity calculation. Therefore, the proximity threshold should be adjusted to accommodate different numbers of valid AppAttributes types.

[0109] In one embodiment, an anomaly detection method is used to dynamically learn the proximity threshold during normal times. Another embodiment uses an alpha filter or exponential averaging to calculate the attack proximity mean and variance. The attack proximity threshold is then calculated as the calculated mean plus a predefined multiplier multiplied by the calculated attack proximity standard deviation.

[0110] At block 230, the alarm logic determines whether a flooding DDoS attack has been detected. Specifically, the alarm logic processes rate-independent and rate-based indicators to determine which type of alarm to output. Specifically, when both the rate-independent abnormality and rate-based abnormality indicators are set, an alarm regarding an HTTP flood attack is generated and output. When the rate-based abnormality indicator is set and the rate-independent normal indicator is output, an alarm regarding a sudden peak in access traffic is output. In all other combinations, no additional alarms are set.

[0111] Figures 5A to 5C This is an example window buffer used to demonstrate the process of updating a baseline. During normal periods when no anomalies are reported, the baseline buffer is updated. The buffers used in this process are the window and baseline. The window attribute buffer holds the aggregate of the current values ​​observed during each time window. The baseline buffer holds the normal baseline. Both the baseline and window buffers are structured as a set of buffers, such as Figure 3The number of buffers in each of the two types of buffers can be different but predefined. In one embodiment, the number of buffers in the window buffer that can be created in response to new AppAttributes is predefined. This is done to prevent the creation of a large number of buffers during an attack, thereby saturating system resources. In this example, the AppAttributes type is a URL query parameter.

[0112] like Figure 5A As shown, buffers 510-1 to 510-3 are buffers set for the AppAttributes included in the baseline. At time '0' when the time window begins, buffers 510-1, 510-2, and 510-3 are set to include the "_ga", "_gl", and "page" key values. These key values ​​are the existing AppAttributes copied from the respective baseline buffers (see also section 1110 in Figure 10), where occ and weight are set to zero. Buffers 510-4 and 510-5 remain empty and are reserved for new AppAttributes that may appear in the time window (see also section 1120 in Figure 10). It should be noted that the window buffer is continuously updated.

[0113] For the baseline buffer, at the end of each time window (when no other anomalies are observed), for existing AppAttributes, the "occ" field of the corresponding buffer is incremented by the "ooc" from the window buffer, and the weight and timestamp fields are also updated. The number of buffers that can be allocated to newly observed AppAttributes is predetermined. Then, the corresponding "occ", "weight", and "age" fields are updated accordingly.

[0114] According to the above example, at time window t=1, the HTTP request includes the following URL query parameters:

[0115] / / www.domain.com / url?_ga=2.89&_gl=1093

[0116] "_ga" and "_gl" are included in the URL parameters and will be included in the window AppAttributes. At the end of t=1, if no attack is detected, the value of the baseline buffer is updated to include Figure 5B The values ​​of the corresponding buffers are shown.

[0117] At the next time window t=2, the HTTP request includes the following URL query parameters:

[0118] / / www.domain.com / url?_ga=2.89&_gl=1093&location=zambia

[0119] Here, a query parameter named "location" appears in the request. Therefore, a new AppAttribute is recognized and should be added to the new AppAttributes buffer allocated within the window buffer. Since there are two free buffers (510-4 and 510-5), buffer 510-4 is updated to include the AppAttributes "location". Then, the fields of buffers 510-1 through 510-4 are updated, as shown in the following example: Figure 5C As shown in Figure 2. At the end of time window t = 2, assuming no active attacks are detected, the baseline window buffer is updated with all aggregated values ​​in the window buffer and, in this case, with the new AppAttributes for the query parameters. New buffers are allocated from the baseline AppAttributes only if there are free buffers available in the baseline buffer. The baseline buffer is also updated to remove expired AppAttributes.

[0120] In one embodiment, for each AppAttributes type, the 'ooc' and 'weight' of all input HTTP requests and their corresponding responses are accumulated in a window buffer by simple addition. At the end of the time window, the aggregated AppAttributes 'ooc' and 'weight' from the window buffer are aggregated into the corresponding baseline buffer by simple addition. In another embodiment, the window buffer 'ooc' and 'weight' are aggregated into the baseline buffer using exponential averaging with an alpha filter, as described in the following equation:

[0121] weight_mean [n+1] = weight _mean [n] (1 – α) + weight [n+1] ·α Equation 10

[0122] You can set α to the average value of the last hour, for example, the "α" value is set to 0.005.

[0123] According to the disclosed embodiment, the end of an attack is detected when several consecutive windows meet the following conditions:

[0124] 1. RPS (requests per second) returns to the normal RPS baseline value;

[0125] 2. The RPS is lower than the RPS abnormal threshold (RPS TH); or

[0126] 3. The attack proximity is below the proximity threshold.

[0127] To avoid oscillations between active attack and attack end detection, some margin can be applied. For example, an attack starts when RPS > baseline RPS, but the same attack ends only when RPS < 0.5 * baseline RPS.

[0128] Figure 6 An example flowchart 600 is shown that illustrates a method for detecting HTTP flood attacks according to an embodiment. In one embodiment, the method may be performed using the detection system 110. The method illustrated in flowchart 600 is performed during a normal period when no active attacks are detected.

[0129] At S610, a transaction is received during a time window. The duration of the time window is predetermined, and the window begins and ends during the execution of S610. The received transaction is a web transaction and typically includes an HTTP request to / from a protected entity hosted by a victim server and its corresponding response.

[0130] At S620, the received transactions are processed to determine whether there is a rate-based anomaly in the traffic directed to the victim server. In one embodiment, the rate-based anomaly is determined for at least one rate-based parameter, including but not limited to the RPS of the total transactions received during the time window. Figure 7 The execution of S620 is further discussed.

[0131] At S625, a check is performed to determine whether a rate-based anomaly is determined. If so, a rate-based anomaly indication is set; otherwise, a rate-based normal indication is output to S640. The rate-based normal indication is provided to S640 to enable learning of a baseline for rate-based and rate-independent behavior. Both the anomaly and normal indications are provided to S640.

[0132] At S630, the received transaction is processed to determine whether there is a rate-independent anomaly in the traffic directed to the victim server. In one embodiment, the rate-independent anomaly is determined due to multiple AppAttributes, including but not limited to: non-standard HTTP headers (or unknown headers); cookies; query parameters in the URL; HTTP request size, HTTP response size; challenge response, HTTP response status code, user agent value, client IP characteristics and other attributes, input traffic TLS fingerprinting, etc. Figure 8 The execution of S630 is further discussed.

[0133] At S635, it is checked whether a rate-independent abnormality is determined, and if so, a rate-independent abnormality indication is set; otherwise, a rate-independent normal indication is output. Both the abnormality and normal indications are provided to S640.

[0134] At S640, the rate-independent and rate-based indications are processed to determine the type of alarm to output. Specifically, when both the rate-independent and rate-based abnormality indications are set, an attack alarm is generated and output. Furthermore, when the rate-based abnormality indication is set and the rate-independent normal indication is output, an alarm for sudden peak traffic is output. For all other combinations, no other alarms are set.

[0135] In one embodiment, when an attack alert is generated, mitigation actions can be taken. Mitigation actions may include blocking the request, responding with a blocking page response, reporting and passing the request to the protected entity, and so on. In one embodiment, the characteristics of the attacker represented by the dynamic application signature can be provided to the mitigation resource. In other words, the general structure of the HTTP request generated by the attacker is provided to the mitigation resource. This will enable the definition and implementation of new mitigation strategies and actions against the attacker. Examples of mitigation actions are provided above.

[0136] In one embodiment, mitigation actions include blocking an attack tool at the source when it is repeatedly characterized as matching a dynamic application signature. For example, if a client identified by an IP address or X-Forwarded-For HTTP header issues a high frequency of HTTP requests that match a dynamic application signature, the client can be considered an attacker (or attack tool). After the client is identified as an attacker, all future HTTP requests from the identified attacker will be blocked without performing any signature matching operations.

[0137] At S650, the time window is reset and execution returns to S610 where transactions received during the new time window are processed.

[0138] In one embodiment, when an attack is detected, the method may also determine an end-of-attack condition. This condition is detected when a preconfigured number of consecutive time-windowed rate-independent or rate-based normal indications are output. A normal condition is satisfied when the windowed request rate per second (window_RPS) or attack proximity falls below a normal baseline threshold, an RPS anomaly threshold, or a proximity threshold multiplied by a predefined coefficient (e.g., 0.8) to avoid oscillation.

[0139] When the attack is detected to be over, a grace period of several time windows is assigned to allow a safe transition back to normal times.

[0140] Figure 7An example flow chart S620 describing a method for detecting rate-based anomalies according to an embodiment is shown. The method will be discussed with reference to a specific example in which rate-based anomalies are determined with respect to a packets per second (RPS) parameter.

[0141] At S710 , the RPS of transactions received during the current window is calculated as the number of transactions (eg, HTTP requests) received divided by the duration of the time window (in seconds).

[0142] At S720 , a short RPS baseline and a medium RPS baseline are determined. These baselines are the mean and variance of the RPS values ​​measured for the previous time window when no active attack is occurring and different alpha filters are applied. The short RPS baseline and the medium RPS baseline are discussed in more detail above. It should be emphasized that S720 is only performed during normal periods when no attacks are detected.

[0143] At S730 , an RPS abnormality threshold is set based on the RPS baseline. In one embodiment, the threshold is set to the maximum value between the short RPS threshold and the medium RPS threshold calculated based on the short RPS baseline and the medium RPS baseline.

[0144] At S740, the RPS (WinRPS[n+1]) of the current time window determined at S710 is compared with the RPS abnormality threshold (RPS_TH). If the RPS exceeds the threshold, a rate-based abnormality is determined (S750). Then, execution returns to S625 ( Figure 6 ).

[0145] Figure 8 An example flow chart S630 describing a method for detecting rate-independent anomalies according to an embodiment is shown. The rate-independent anomalies are determined based on AppAttributes.

[0146] At S810, transactions received during the current window are processed to at least identify AppAttributes in the transactions, various types of AppAttributes, and for each AppAttributes type, extract or parse at least a key from the request.

[0147] At S820, the window buffer for the current window is updated based on the AppAttributes identified in the received transaction. This includes: for each previous AppAttributes key value, incrementing the occ field value and the weight field value in the respective buffers. For the first occurrence of the AppAttributes key value, a new buffer is started and the value is recorded in it. The new buffer for storing the newly occurring value is dedicated. In both options, the weight and timestamp fields are updated as described above.

[0148] At S830, the baseline buffer is updated based on the window buffer. The baseline buffer is updated only during normal periods (i.e., when the RPS is outputting a normal indication). The baseline is updated based on simple addition or exponential averaging (e.g., through an alpha filter). The baseline is updated for newly appearing AppAttributes only when no anomalies are detected. Updating the baseline buffer is discussed above.

[0149] At S840, the attack proximity is determined. The attack proximity represents the statistical closeness of the window buffer (for the current time window) to the baseline buffer. The determination of the attack proximity is discussed above.

[0150] At S850, the attack proximity is compared with the proximity threshold. If the attack proximity exceeds the proximity threshold, a rate-independent anomaly is determined (S860). Regardless of the result, execution returns to S635 ( Figure 6 ).

[0151] Figure 9 9 is an example block diagram of a system 110 implemented according to an embodiment. Detection system 110 includes processing circuitry 910 coupled to memory 915, storage 920, and network interface 940. In another embodiment, the components of system 110 may be communicatively coupled via bus 950.

[0152] The processing circuit 910 may be implemented as one or more hardware logic components and circuits. For example, but not limited to, illustrative types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), general-purpose microprocessors, microcontrollers, digital signal processors (DSPs), or any other hardware logic components that can perform calculations or other information operations.

[0153] The memory 915 may be volatile (eg, RAM), non-volatile (eg, ROM, flash memory), or a combination thereof. In one configuration, computer-readable instructions for implementing one or more embodiments disclosed herein may be stored in the storage device 920 .

[0154] In another embodiment, the memory 915 is configured to store software. Software should be broadly interpreted as any type of instruction, whether referred to as software, firmware, middleware, microcode, hardware description language or other. Instructions may include code (e.g., source code format, binary code format, executable code format, or any other suitable code format). When executed by one or more processors, these instructions cause the processing circuit 910 to perform the various processes described herein. Specifically, when executed, the instructions cause the processing circuit 910 to perform the embodiments described herein. The memory 915 may be further configured as a storage buffer.

[0155] Storage device 920 may be a magnetic storage device, an optical storage device, etc., and may be implemented as, for example, flash memory or other storage technology, a CD-ROM, a digital versatile disk (DVD), or any other medium that can be used to store the desired information.

[0156] Processing circuitry 910 is configured to detect and cause detection of an HTTPS flood attack (as described herein).

[0157] The network interface 940 allows the device to communicate with at least a server and a client. Figure 9 The particular architecture shown is used, and other architectures may equally be used without departing from the scope of the disclosed embodiments. Additionally, the system 110 may use Figure 9 The arrangement shown is constructed.

[0158] The various embodiments disclosed herein can be implemented as hardware, firmware, software or any combination thereof. In addition, the software is preferably implemented as an application program tangibly contained in a program storage unit or a computer-readable medium, and the program storage unit or the computer-readable medium is composed of a combination of components or certain devices and / or devices. The application program can be uploaded to a machine including any suitable architecture and executed by the machine. Preferably, the machine is implemented on a computer platform with hardware such as one or more central processing units (CPUs), memory and input / output interfaces. The computer platform can also include an operating system and microinstruction code. The various processes and functions described herein can be a part of the microinstruction code or a part of the application program or any combination thereof, which can be executed by the CPU (regardless of whether such a computer or processor is clearly shown). In addition, various other peripheral units can be connected to the computer platform, for example, additional data storage units and printing units. In addition, non-transitory computer-readable media are any computer-readable media other than temporary propagation signals.

[0159] All examples and conditional language cited herein are for teaching purposes to help the reader understand the principles of the disclosed embodiments and the concepts contributed by the inventors to promote this field, and should be interpreted as not being limited to these specifically cited examples and conditions. In addition, all statements of the principles, aspects, and embodiments of the disclosed embodiments described herein and their specific examples are intended to cover their structural and functional equivalents. In addition, the equivalent technical solutions covered are intended to include currently known equivalents and equivalents developed in the future, that is, any technical elements that can perform the same function, regardless of their structure.

[0160] As used herein, the phrase "at least one of" followed by a list of items means that any of the listed items may be used alone, or any combination of two or more of the listed items may be used. For example, if a system is described as including "at least one of A, B, and C," the system may include: A alone; B alone; C alone; 2A; 2B; 2C; 3A; a combination of A and B; a combination of B and C; a combination of A and C; a combination of A, B, and C; a combination of 2A and C; a combination of A, 3B, and 2C, and so on.

[0161] It should be understood that any reference to technical elements herein using terms such as "first," "second," etc. generally does not limit the number or order of these elements. Instead, these expressions are generally used herein as a convenient way to distinguish between two or more elements or instances of elements. Thus, a reference to a first and a second element does not mean that only two elements can be used there, nor does it mean that the first element must precede the second element in some way. Furthermore, unless otherwise indicated, a group of elements includes one or more elements.

Claims

1. A method for detecting an application layer flood denial of service (DDoS) attack, wherein the attack is initiated by an attacker using an advanced application layer flood attack tool, the method comprising: processing application layer transactions received during a current time window to detect rate-based anomalies in traffic directed to a protected entity; processing the received application layer transactions to determine a rate-independent anomaly based on a plurality of application attributes (AppAttributes) observed in the application layer transactions received during the current time window, wherein the rate-independent anomaly is determined based on a continuously updated baseline of the AppAttributes, wherein the AppAttributes represent application behavior of the protected entity, the application behavior being modeled based on the application layer transactions; Determining whether an application layer flooding DDoS exists in the current time window based on the detected rate-based anomaly and the detected rate-independent anomaly; and When the application layer flood DDoS occurs, a mitigation action is triggered.

2. The method according to claim 1, wherein Detection of rate-based anomalies in traffic directed to protected entities also includes: calculating a current rate-based parameter for the current time window, defined as a number of input transactions received per second during the time window; calculating at least one rate-based baseline, each of the rate-based baselines being calculated using an alpha filter configured for a specific time period; calculating a rate-based anomaly threshold for each of the at least one rate-based baseline; and When the calculated value of the current rate-based parameter exceeds one of the rate-based anomaly thresholds, the rate-based anomaly is lowered.

3. The method according to claim 1, further comprising: Buffering AppAttributes derived from application layer transactions received during the current time window into a window buffer.

4. The method according to claim 3, further comprising: Calculates the baseline for AppAttributes using the baseline buffer.

5. The method according to claim 4, wherein Each of the baseline buffer and the window buffer is for a single AppAttributes type.

6. The method according to claim 5, wherein: The AppAttributes type is any of the following: query parameter (args) name, non-standard HTTP header name, cookie name, HTTP request size range, HTTP response size range, HTTP response status code, source IP address attribute, user-agent value, TLS fingerprint attribute, and set-cookie name.

7. The method according to claim 5, wherein: Each buffer of a single AppAttributes type includes: a key field that specifies the key value, an occurrence (occ) field that counts the number of occurrences, a weight field that indicates the relative weight, and an age (age) field that indicates the last time the AppAttributes type was observed in an input application layer transaction.

8. The method according to claim 3, wherein: Each AppAttributes type buffer of the window buffer includes a first segment containing previously observed AppAttributes values ​​and a second segment containing first observed AppAttributes values, wherein the first observed AppAttributes values ​​are observed in transactions received during the current time window and are not specified in the first segment.

9. The method according to claim 8, further comprising: The window buffer is updated based on AppAttributes values ​​observed in transactions received during a current time window, wherein the window buffer is updated per AppAttributes type.

10. The method according to claim 9, further comprising: At the end of the current time window, the baseline buffer is updated with the AppAttributes in the corresponding window buffer, wherein the baseline buffer is updated only during normal periods.

11. The method according to claim 3, wherein: Determining the rate-independent anomaly further includes: For each AppAttributes type, determining an attack proximity, wherein the attack proximity indicates how statistically close the AppAttributes in the window buffer are to the AppAttributes in the baseline buffer; Calculate the total attack proximity across all AppAttributes types based on their respective window buffers and baseline buffers; Calculate proximity threshold; comparing the attack proximity to the proximity threshold; and When the attack proximity exceeds the proximity threshold, it is determined that the rate is irrelevant.

12. The method according to claim 11, further comprising: The attack proximity is determined using a distribution density function.

13. The method according to claim 12, wherein: The attack proximity for each AppAttribute type is a function of a total variation between a baseline distribution density and a window distribution density, wherein the baseline distribution density function is determined based on the baseline buffer and the window is determined based on the window buffer.

14. The method according to claim 12, wherein: The proximity threshold is a function of the number of valid AppAttributes types multiplied by a pre-configured parameter.

15. The method according to claim 1, wherein Determining whether the flood DDoS has exited further includes: When the rate-independent indication and the rate-based anomaly indication are set during the time window, an alert regarding the attack is generated.

16. The method according to claim 1, further comprising: Determine the attack end conditions; and The mitigation action is triggered to end.

17. The method according to claim 16, wherein When any of the following conditions occurs, the attack end condition is determined to be satisfied: The rate independence indication is below a corresponding rate independence threshold; or The rate-based indication is below one of the rate-based thresholds.

18. The method according to claim 1, wherein An application layer transaction includes any of the following: HTTP request, HTTP response, HTTPs request, and HTTPs response.

19. The method according to claim 18, wherein Application layer transactions include samples of the following: actual HTTP requests, HTTP responses corresponding to the actual HTTP requests, HTTPs requests and their corresponding HTTPs responses, wherein different sampling rates can be used for normal period scenarios and attack period scenarios.

20. A non-transitory computer-readable medium having stored thereon instructions for causing a processing circuit to execute a program, the program comprising: Detect application layer flood denial of service (DDoS) attacks launched by attackers using advanced application layer flood attack tools, including: processing application layer transactions received during a current time window to detect rate-based anomalies in traffic directed to a protected entity; processing the received application layer transactions to determine a rate-independent anomaly based on a plurality of application attributes (AppAttributes) observed in the application layer transactions received during the current time window, wherein the rate-independent anomaly is determined based on a continuously updated baseline of the AppAttributes, wherein the AppAttributes represent application behavior of the protected entity, the application behavior being modeled based on the application layer transactions; Determining whether an application layer flooding DDoS exists in the current time window based on the detected rate-based anomaly and the detected rate-independent anomaly; and When the application layer flood DDoS occurs, a mitigation action is triggered.

21. A system for detecting an application layer flood denial of service (DDoS) attack, wherein the attack is initiated by an attacker using an advanced application layer flood attack tool, the system comprising: processing circuits; and a memory containing instructions that, when executed by the processing circuitry, configure the controller to: processing application layer transactions received during a current time window to detect rate-based anomalies in traffic directed to a protected entity; processing the received application layer transactions to determine a rate-independent anomaly based on a plurality of application attributes (AppAttributes) observed in the application layer transactions received during the current time window, wherein the rate-independent anomaly is determined based on a continuously updated baseline of the AppAttributes, wherein the AppAttributes represent application behavior of the protected entity, the application behavior being modeled based on the application layer transactions; Determining whether an application layer flooding DDoS exists in the current time window based on the detected rate-based anomaly and the detected rate-independent anomaly; and When the application layer flood DDoS occurs, a mitigation action is triggered.

22. The system of claim 21, wherein: The system is further configured to: calculating a current rate-based parameter for a current time window, defined as the number of input transactions received per second during said time window; calculating at least one rate-based baseline, each of the rate-based baselines being calculated using an alpha filter configured for a particular time period; calculating a rate-based anomaly threshold for each of at least one rate-based baseline; and When the calculated value of the current rate-based parameter exceeds one of the anomaly rate-based thresholds, the rate-based anomaly is lowered.

23. The system of claim 21, wherein: The system is further configured to: Buffering AppAttributes derived from application layer transactions received during the current time window into a window buffer.

24. The system of claim 23, wherein the system is further configured to: Calculates the baseline for AppAttributes using the baseline buffer.

25. The system of claim 24, wherein: Each of the baseline buffer and the window buffer is for a single AppAttributes type.

26. The system of claim 25, wherein: The AppAttributes type is any of the following: query parameter (args) name, non-standard HTTP header name, cookie name, HTTP request size range, HTTP response size range, HTTP response status code, source IP address attribute, user-agent value, TLS fingerprint attribute, and set-cookie name.

27. The system of claim 25, wherein: Each buffer of a single AppAttributes type includes: a key value field that specifies the key value, an occurrence (occ) field that counts the number of occurrences, a weight field that indicates the relative weight, and a timestamp field that indicates the last time the AppAttributes type was observed in an input application layer transaction.

28. The system of claim 23, wherein: Each AppAttributes type buffer of the window buffer includes a first segment containing previously observed AppAttributes values ​​and a second segment containing first observed AppAttributes values, wherein the first observed AppAttributes values ​​are observed in transactions received during the current time window and are not specified in the first segment.

29. The system of claim 28, wherein the system is further configured to: The window buffer is updated based on AppAttributes values ​​observed in transactions received during a current time window, wherein The window buffer is updated according to each AppAttributes type.

30. The system of claim 29, wherein the system is further configured to: At the end of the current time window, the baseline buffer is updated with the AppAttributes in the corresponding window buffer, where The baseline buffer is updated only during normal periods.

31. The system of claim 23, wherein: The system is further configured to: For each AppAttributes type, determining an attack proximity, wherein the attack proximity indicates how statistically close the AppAttributes in the window buffer are to the AppAttributes in the baseline buffer; Calculates the total attack proximity across all AppAttributes types based on their respective window buffers and baseline buffers; Calculate proximity threshold; comparing the attack proximity to the proximity threshold; and When the attack proximity exceeds the proximity threshold, it is determined that the rate is irrelevant.

32. The system of claim 31, wherein: The system is further configured to determine the attack proximity using a distribution density function.

33. The system of claim 32, wherein: The attack proximity for each AppAttribute type is a function of a total variation between a baseline distribution density and a window distribution density, wherein the baseline distribution density function is determined based on the baseline buffer and the window is determined based on the window buffer.

34. The system of claim 32, wherein: The proximity threshold is a function of the number of valid AppAttributes types multiplied by a pre-configured parameter.

35. The system of claim 21, wherein: The system is further configured to: When the rate-independent indication and the rate-based anomaly indication are set during the time window, an alert regarding the attack is generated.

36. The system of claim 21, wherein: The system is further configured to: Determine attack termination conditions; and The mitigation action is triggered to end.

37. The system of claim 36, wherein: The system is further configured to determine that the attack end condition is satisfied when any of the following situations occurs: The rate independence indication is below a corresponding rate independence threshold; or The rate-based indication is below one of the rate-based thresholds.

38. The system of claim 21, wherein: An application layer transaction includes any of the following: HTTP request, HTTP response, HTTPs request, and HTTPs response.

39. The system of claim 38, wherein: Application layer transactions include samples of actual HTTP requests and their corresponding HTTP responses, as well as HTTPs requests and their corresponding HTTPs responses, wherein different sampling rates can be used for normal period scenarios and attack period scenarios.

Images