Http white flow filtering method and filtering system based on online learning algorithm
By combining online learning algorithms and binary tree models, the problems of online updates and high complexity in HTTP white traffic filtering in traditional hardware WAFs are solved, achieving efficient and low-cost white traffic filtering in hardware WAFs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING VENUS INFORMATION SECURITY TECH
- Filing Date
- 2023-03-02
- Publication Date
- 2026-08-04
AI Technical Summary
Traditional hardware WAFs struggle to learn and update online when identifying and filtering HTTP traffic, and their high time and space complexity makes them difficult to deploy in hardware WAFs.
An HTTP white traffic filtering method based on online learning algorithms is adopted. By preprocessing and feature extraction of real-time HTTP traffic, calculating anomaly scores using a binary tree model, and updating the model using quality estimation theory, white traffic can be filtered out.
In industrial-grade massive data scenarios, it reduces time and space complexity, can efficiently filter white traffic in hardware WAF, alleviate data drift problems, and reduce labor costs.
Smart Images
Figure CN116208413B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of information security, specifically relating to an HTTP white traffic filtering method and system based on online learning algorithms. Background Technology
[0002] With the development and maturation of internet technology, cyberattacks have become increasingly frequent, especially attacks targeting web application services, which account for the vast majority of cyberattacks. Therefore, Web Application Firewalls (WAFs) have emerged. WAFs protect web service security by detecting HTTP / HTTPS network requests, identifying and blocking web attacks such as SQL injection, cross-site scripting, web trojan uploads, command / code injection, CC attacks, and cross-site request forgery. WAFs can generally be categorized into three types: software WAFs, hardware WAFs, and cloud WAFs, with hardware WAFs being the most common. Hardware WAFs exist as independent hardware devices and can be deployed in the network in transparent bridging mode, bypass mode, and reverse proxy forms to provide security protection for backend web applications.
[0003] In traditional hardware-based Web Application Firewalls (WAFs), intrusion detection technology intercepts web intrusions by maintaining a large rule base. On the one hand, hard rules are easily bypassed by agile hackers, and rule sets based on past knowledge are insufficient to deal with unknown attacks. On the other hand, the level of attack and defense is rising, and the construction and maintenance of rules by the defender has a high threshold and high human resource costs.
[0004] Next-generation web intrusion detection technologies based on data-driven and machine learning methods have overcome the shortcomings of traditional rule-based methods, bringing new developments and breakthroughs to defenders. Most of these methods rely on supervising classification models learned from fixed black-and-white sample datasets. However, supervising methods are difficult to implement for identifying and filtering HTTP white traffic because different business scenarios generate different HTTP white traffic, and sample labels are hard to obtain. Therefore, the most common approach in academia and industry is to establish a white traffic baseline using unsupervised anomaly detection methods to filter white traffic. Traditional unsupervised anomaly detection methods include density-based Local Anomaly Factor (LOF) algorithms, tree-based Isolation Forest (iForest) algorithms, and distance-based KNN algorithms. These algorithms have the following problems: First, the models used in these algorithms are usually static models based on a fixed dataset, which cannot be learned and updated online; second, most of these algorithms are based on density estimation theory, which has high time and space complexity, making them difficult to deploy in hardware WAF scenarios. Summary of the Invention
[0005] To at least partially overcome the problems existing in related technologies, this application provides an HTTP white traffic filtering method and system based on an online learning algorithm.
[0006] According to a first aspect of the embodiments of this application, this application provides an HTTP white traffic filtering method based on an online learning algorithm, which includes the following steps:
[0007] Preprocess the real-time HTTP traffic text data stream to obtain HTTP messages;
[0008] Extract the necessary fields from the HTTP message to obtain text data, and then convert the text data into a numerical vector to form a feature data stream;
[0009] The process of using online learning algorithms to filter white traffic from feature data streams is as follows:
[0010] Create a binary tree model using 1 to n samples from the feature data stream;
[0011] Determine whether the samples after the nth sample in the feature data stream have filled the nearest window. If so, update the current t binary tree models using the samples that have filled the nearest window. Otherwise, calculate the outlier score of the binary tree model based on the quality estimation theory.
[0012] The final anomaly score determines whether the corresponding feature data stream is legitimate traffic; if so, it is allowed to pass directly.
[0013] In the above-mentioned HTTP white traffic filtering method based on online learning algorithms, the preprocessing of real-time HTTP traffic text data stream includes decryption, data cleaning, and filtering of the real-time HTTP traffic text data stream.
[0014] In the above HTTP white traffic filtering method based on online learning algorithms, the process of creating a binary tree model using 1 to n samples from the feature data stream is as follows:
[0015] Create a single binary tree model, which includes internal nodes and leaf nodes. The attributes of the internal nodes include the specific feature d of the partition, the split point value p corresponding to the specific feature d, the reference data quality referenceMass, the recent data quality latestMass, and the depth k of the current node.
[0016] Create t binary tree models.
[0017] Furthermore, the process of creating a single binary tree model is as follows:
[0018] Randomly select a feature and calculate the minimum value (minValue) and maximum value (maxValue) of the samples within the baseline window on that feature;
[0019] The split point value p is calculated based on the minimum value minValue and the maximum value maxValue:
[0020] p=0.5*(minValue+maxValue),
[0021] The sample data within the baseline window is divided into left and right subtrees based on the split point value.
[0022] Furthermore, the specific process of updating the current t binary tree models using samples that fill the nearest window is as follows:
[0023] Set the nearest window to the same size as the base window;
[0024] The (n+1)th to 2nth samples used to fill the nearest window flow through the baseline window and then enter the binary tree model;
[0025] Assign the latestMass value of all current nodes to the referenceMass value, and simultaneously reset the latestMass value of all current nodes to zero, thus completing the update of the binary tree model.
[0026] In the above HTTP white traffic filtering method based on online learning algorithms, the process of calculating the anomaly score of the binary tree model based on quality estimation theory is as follows:
[0027] The outlier score of the i-th binary tree is:
[0028] score i =1-[node.referenceMass*pow(2,node.depth) / maxScore],
[0029] In the formula, node.referenceMass represents the reference data quality of the node, node.depth represents the current depth of the node; pow(2,node.depth) represents 2 raised to the power of node.depth; maxScore represents the maximum score.
[0030] maxScore=w*pow(2,maxDepth),
[0031] In the formula, w represents the size of the nearest window;
[0032] The final anomaly score, finalScore, is the average score of multiple binary tree models.
[0033] finalScore = averageofscore i ,
[0034] In the formula, finalScore is a number between 0 and 1.
[0035] Furthermore, the process of determining whether the corresponding feature data stream is white traffic based on the final anomaly score is as follows:
[0036] Set an anomaly score threshold δ. The feature data stream corresponding to the final anomaly score finalScore that is greater than or equal to δ is determined as an anomaly traffic, and the feature data stream corresponding to the final anomaly score finalScore that is less than δ is determined as a legitimate traffic.
[0037] According to a second aspect of the embodiments of this application, this application provides an HTTP white traffic filtering system based on an online learning algorithm, which includes a preprocessing module, a feature engineering module, and an online learning module;
[0038] The preprocessing module is used to preprocess the real-time HTTP traffic text data stream to obtain HTTP messages;
[0039] The feature engineering module is used to extract the required fields from the HTTP message and convert the obtained text data into a numerical vector to form a feature data stream;
[0040] The online learning module uses an online learning algorithm to perform white traffic filtering on the feature data stream, the process of which is as follows:
[0041] Create a binary tree model using 1 to n samples from the feature data stream;
[0042] Determine whether the samples after the nth sample in the feature data stream have filled the nearest window. If so, update the current t binary tree models using the samples that have filled the nearest window. Otherwise, calculate the outlier score of the binary tree model based on the quality estimation theory.
[0043] The final anomaly score determines whether the corresponding feature data stream is legitimate traffic; if so, it is allowed to pass directly.
[0044] According to a third aspect of the embodiments of this application, this application also provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the HTTP white traffic filtering method based on the online learning algorithm described in any of the preceding claims.
[0045] As can be seen from the above specific embodiments of this application, it has at least the following beneficial effects: The HTTP white traffic filtering method based on online learning algorithms provided in this application, in industrial-grade massive data scenarios, calculates anomaly scores based on the situation of few but distinct abnormal traffic, using quality estimation theory, and filters out white traffic according to the anomaly scores. The constant time and space complexity allows this application to be practically applied to hardware WAFs. Furthermore, the HTTP white traffic filtering method based on online learning algorithms continuously updates the model, which can alleviate data drift problems and greatly reduce labor costs.
[0046] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the claims made in this application. Attached Figure Description
[0047] The accompanying drawings, which are part of the specification of this application, illustrate embodiments of the present application and are used together with the description of the specification to illustrate the principles of the present application.
[0048] Figure 1 This is a flowchart illustrating an HTTP white traffic filtering method based on an online learning algorithm, provided as an embodiment of this application.
[0049] Figure 2 This is a schematic diagram of the structure of a single binary tree model in the HTTP white traffic filtering method based on online learning algorithms provided in the embodiments of this application.
[0050] Figure 3 This is a schematic diagram of the structure of an HTTP white traffic filtering system based on an online learning algorithm, provided in an embodiment of this application. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the spirit of the content disclosed in this application will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of this application, any person skilled in the art can make changes and modifications based on the technology taught in this application without departing from the spirit and scope of this application.
[0052] The illustrative embodiments and descriptions provided in this application are for explaining the application, but are not intended to limit the application. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.
[0053] The terms “first,” “second,” etc., used in this document are not intended to specifically refer to order or sequence, nor are they used to limit this application; they are merely used to distinguish elements or operations described using the same technical terms.
[0054] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.
[0055] The term "and / or" as used herein includes any or all of the things mentioned.
[0056] The term "multiple" in this article includes "two" and "more than two"; the term "multiple groups" in this article includes "two groups" and "more than two groups".
[0057] Certain terms used to describe this application will be discussed below or elsewhere in this specification to provide additional guidance to those skilled in the art in describing the application.
[0058] During the research and development process, the inventors of this application discovered that although traditional anomaly detection methods have good experimental results in establishing white flow baselines, they are still impractical in hardware WAF applications. The reason is that...
[0059] (1) In the industry, on the one hand, hardware WAFs are usually deployed in a relatively forward position in the network, which determines that their traffic is usually tens of thousands of messages per second; on the other hand, due to cost reasons, hardware WAFs usually do not have high configuration of computing and storage resources. Therefore, many existing methods are not designed from an industrial perspective, and have high time and space complexity, making them difficult to apply in practice to hardware WAFs.
[0060] (2) In the industry, network traffic data drifts with changes in customer business and time. Traditional anomaly detection methods train an inherent model based on static datasets, which drifts with dynamic data. The model needs to be updated and maintained regularly for different customers, resulting in extremely high manpower costs.
[0061] Starting from the situation where outliers are "few but different", this application proposes an HTTP white traffic filtering method based on mass estimation theory and online learning algorithm for network traffic data in industrial-scale, drifting scenarios.
[0062] like Figure 1 As shown, the HTTP white traffic filtering method based on online learning algorithms provided in this application includes the following steps:
[0063] S1. Preprocess the real-time HTTP traffic text data stream to obtain HTTP messages.
[0064] The preprocessing process includes decrypting, cleaning, and filtering real-time HTTP traffic text data streams.
[0065] S2. Extract the required fields from the HTTP message to obtain text data, and convert the text data into a numerical vector to form a feature data stream.
[0066] S3. Utilize online learning algorithms to perform white traffic filtering on the feature data stream. The specific process is as follows:
[0067] S31. Create a binary tree model using 1 to n samples from the feature data stream, including:
[0068] S311. Create a single binary tree model, where the single binary tree model includes internal nodes and leaf nodes. The attributes of the internal nodes include the specific feature d of the partition, the split point value p corresponding to the specific feature d, the reference data quality referenceMass, the latest data quality latestMass, and the depth k of the current node. The process is as follows:
[0069] First, randomly select a feature and calculate the minimum value (minValue) and maximum value (maxValue) of the samples within the baseline window w1 on that feature.
[0070] Secondly, the split point value p is calculated based on the minimum value minValue and the maximum value maxValue:
[0071] p=0.5*(minValue+maxValue) (1)
[0072] Finally, the sample data within the baseline window w1 is divided into left and right subtrees based on the split point value, resulting in the left and right subtrees.
[0073] For example, such as Figure 2 As shown, x and y represent features, 0.5 and 0.25 represent split point values, the leaf node values 2, 33, and 3 represent reference mass, the leaf node depth k is 2, and the latest mass of all nodes is 0.
[0074] S312. Using the method in step S311, randomly select a feature and create t binary tree models.
[0075] S32. Determine whether the samples after the nth sample in the feature data stream have filled the nearest window w2. If so, use these samples to update the current t binary tree models. Otherwise, proceed to step S33 to calculate the anomaly score of the binary tree model.
[0076] The size of the nearest window w2 is equal to the size of the reference window w1, and w represents the size of the nearest window w2 or the reference window w1. When the nearest window w2 or the reference window w1 is filled, it can hold at most n samples, that is, the size w of the nearest window w2 or the reference window w1 is equal to the number of samples n.
[0077] The specific process of updating the current t binary tree models using samples that fill the nearest window w2 is as follows:
[0078] The (n+1)th to 2nth samples used to fill the nearest window w2 flow through the baseline window w1 and then enter the binary tree model.
[0079] Assign the latestMass value of all current nodes to the referenceMass value, and simultaneously reset the latestMass value of all current nodes to zero, thus completing the update of the binary tree model.
[0080] S33. Based on the quality estimation theory, calculate the anomaly score of the binary tree model.
[0081] Abnormal traffic data, compared to normal traffic data (i.e., HTTP white traffic data), is characterized by being "less frequent but different". Therefore, abnormal traffic data is represented in a binary tree model as follows:
[0082] Abnormal traffic data typically falls on shallower leaf nodes in a binary tree model; abnormal traffic data typically falls on leaf nodes with a smaller reference mass in a binary tree model.
[0083] The outlier score of the i-th binary tree is:
[0084] score i =1-[node.referenceMass*pow(2,node.depth) / maxScore] (2)
[0085] In equation (2), node.referenceMass represents the reference data quality of a node, node.depth represents the current depth of a node; pow(2,node.depth) represents a power function, i.e., 2 raised to the power of node.depth; maxScore represents the maximum score, i.e., all samples in the baseline window w1 fall into the same node's reference data quality and reach the maximum depth, mathematically expressed as:
[0086] maxScore=w*pow(2,maxDepth) (3)
[0087] In equation (3), w represents the size of the nearest window w2.
[0088] The final anomaly score, finalScore, is the average score of multiple binary tree models, mathematically expressed as:
[0089] finalScore = averageofscore i (4)
[0090] In equation (4), finalScore is a number between 0 and 1.
[0091] S34. Determine whether the corresponding feature data stream is legitimate traffic based on the final anomaly score. If it is, allow it to pass directly without sending it to the subsequent parsing and detection modules; otherwise, send the abnormal traffic to the subsequent parsing, decoding, and detection modules.
[0092] The process of determining whether the corresponding HTTP traffic text data stream is legitimate traffic based on the final anomaly score is as follows:
[0093] An anomaly score threshold δ is set. Feature data streams with a final anomaly score (finalScore) greater than or equal to δ are classified as abnormal traffic, while feature data streams with a final anomaly score (finalScore) less than δ are classified as normal traffic, i.e., legitimate traffic. The anomaly score threshold δ can be set to a value of 0.5.
[0094] The HTTP white traffic filtering method based on online learning algorithms provided in this application has the following theoretical time and space complexity:
[0095] Time complexity: O(t*(h+w)), where t represents the number of binary trees, h represents the maximum tree depth, and w represents the size of the baseline window w1 or the size of the nearest window w2. This time complexity is constant.
[0096] Space complexity: O(t*2^h), which is also a constant.
[0097] Constant time and space complexity are suitable for resource-constrained hardware WAFs, and they offer high detection efficiency and speed.
[0098] The HTTP white traffic filtering method based on online learning algorithms provided in this application, in industrial-grade massive data scenarios, calculates anomaly scores using quality estimation theory based on the scenario of few but diverse abnormal traffic, and filters out white traffic according to the anomaly scores. The HTTP white traffic filtering method based on online learning algorithms continuously updates the model, which can alleviate the data drift problem.
[0099] like Figure 3As shown, based on the HTTP white traffic filtering method based on online learning algorithm provided in this application, this application also provides an HTTP white traffic filtering system based on online learning algorithm, which includes a preprocessing module, a feature engineering module, and an online learning module.
[0100] The preprocessing module preprocesses the real-time HTTP traffic text data stream to obtain HTTP messages. The feature engineering module extracts the necessary fields from the HTTP messages and converts the resulting text data into numerical vectors to form a feature data stream. The online learning module uses online learning algorithms to filter out unwanted traffic from the feature data stream.
[0101] It should be noted that the HTTP white traffic filtering system based on online learning algorithms provided in the above embodiments and the HTTP white traffic filtering method based on online learning algorithms belong to the same concept. For details of its implementation process, please refer to the method embodiments, which will not be repeated here.
[0102] In an exemplary embodiment, this application also provides a computer storage medium, which is a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor to perform the HTTP white traffic filtering method based on an online learning algorithm in any embodiment of this application.
[0103] The embodiments of this application described above can be implemented in various hardware, software codes, or combinations thereof. For example, embodiments of this application may also represent program code executing the above methods in a data signal processor. This application may also relate to various functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array. The processor described above can be configured to perform specific tasks according to this application, which are accomplished by executing machine-readable software code or firmware code defining the specific methods disclosed in this application. The software code or firmware code can be developed to represent different programming languages and different formats or forms. It can also represent software code compiled for different target platforms. However, the different code styles, types, and languages of the software code performing tasks according to this application and other types of configuration code do not depart from the spirit and scope of this application.
[0104] The above description is merely an illustrative embodiment of this application. Any equivalent changes and modifications made by those skilled in the art without departing from the concept and principles of this application shall fall within the scope of protection of this application.
Claims
1. A method for filtering HTTP white traffic based on an online learning algorithm, characterized in that, Includes the following steps: Preprocess the real-time HTTP traffic text data stream to obtain HTTP messages; Extract the necessary fields from the HTTP message to obtain text data, and then convert the text data into a numerical vector to form a feature data stream; The process of using online learning algorithms to filter white traffic from feature data streams is as follows: Create a binary tree model using 1 to n samples from the feature data stream; Determine whether the samples after the nth sample in the feature data stream have filled the nearest window. If so, update the current t binary tree models using the samples that have filled the nearest window; otherwise, calculate the outlier score of the binary tree model based on quality estimation theory. The process of calculating the outlier score of the binary tree model based on quality estimation theory is as follows: The outlier score of the i-th binary tree is: scorei=1−[node.referenceMass*pow(2,node.depth) / maxScore], In the formula, node.referenceMass represents the reference data quality of the node, node.depth represents the current depth of the node; pow(2,node.depth) represents 2 raised to the power of node.depth; maxScore represents the maximum score. maxScore=w*pow(2,maxDepth), In the formula, w represents the size of the nearest window; The final anomaly score, finalScore, is the average score of multiple binary tree models. finalScore=average of scorei, In the formula, finalScore is a number between 0 and 1; The final anomaly score determines whether the corresponding feature data stream is legitimate traffic; if so, it is allowed to pass directly.
2. The HTTP white traffic filtering method based on online learning algorithm according to claim 1, characterized in that, The preprocessing of real-time HTTP traffic text data streams includes decryption, data cleaning, and filtering.
3. The HTTP white traffic filtering method based on online learning algorithm according to claim 1, characterized in that, The process of creating a binary tree model using 1 to n samples from the feature data stream is as follows: Create a single binary tree model, which includes internal nodes and leaf nodes. The attributes of the internal nodes include the specific feature d of the partition, the split point value p corresponding to the specific feature d, the reference data quality referenceMass, the recent data quality latestMass, and the depth k of the current node. Create t binary tree models.
4. The HTTP white traffic filtering method based on online learning algorithm according to claim 3, characterized in that, The process of creating a single binary tree model is as follows: Randomly select a feature and calculate the minimum value (minValue) and maximum value (maxValue) of the samples within the baseline window on that feature; The split point value p is calculated based on the minimum value minValue and the maximum value maxValue: p = 0.5 * (minValue + maxValue) The sample data within the baseline window is divided into left and right subtrees based on the split point value.
5. The HTTP white traffic filtering method based on online learning algorithm according to claim 4, characterized in that, The specific process of updating the current t binary tree models using samples that fill the nearest window is as follows: Set the nearest window to the same size as the base window; The (n+1)th to 2nth samples used to fill the nearest window flow through the baseline window and then enter the binary tree model; Assign the latestMass value of all current nodes to the referenceMass value, and simultaneously reset the latestMass value of all current nodes to zero, thus completing the update of the binary tree model.
6. The HTTP white traffic filtering method based on online learning algorithm according to claim 5, characterized in that, The process of determining whether the corresponding feature data stream is white traffic based on the final anomaly score is as follows: Set an abnormal score threshold , will be greater than or equal to The feature data stream corresponding to the final anomaly score (finalScore) is determined to be abnormal traffic, and those scores less than [a certain value] will be [increased]. The feature data stream corresponding to the final anomaly score (finalScore) is determined to be white traffic.
7. An HTTP white traffic filtering system based on an online learning algorithm, characterized in that, It includes a preprocessing module, a feature engineering module, and an online learning module; The preprocessing module is used to preprocess the real-time HTTP traffic text data stream to obtain HTTP messages; The feature engineering module is used to extract the required fields from the HTTP message and convert the obtained text data into a numerical vector to form a feature data stream; The online learning module uses an online learning algorithm to perform white traffic filtering on the feature data stream, the process of which is as follows: Create a binary tree model using 1 to n samples from the feature data stream; Determine whether the samples after the nth sample in the feature data stream have filled the nearest window. If so, update the current t binary tree models using the samples that have filled the nearest window. Otherwise, calculate the outlier score of the binary tree model based on the quality estimation theory. The process of calculating the outlier score of the binary tree model based on quality estimation theory is as follows: The outlier score of the i-th binary tree is: scorei=1−[node.referenceMass*pow(2,node.depth) / maxScore], In the formula, node.referenceMass represents the reference data quality of the node, node.depth represents the current depth of the node; pow(2,node.depth) represents 2 raised to the power of node.depth; maxScore represents the maximum score. maxScore=w*pow(2,maxDepth), In the formula, w represents the size of the nearest window; The final anomaly score, finalScore, is the average score of multiple binary tree models. finalScore=average of scorei, In the formula, finalScore is a number between 0 and 1; The final anomaly score determines whether the corresponding feature data stream is legitimate traffic; if so, it is allowed to pass directly.
8. A storage medium, characterized in that, It stores a computer program, characterized in that, when the computer program is executed by a processor, it implements the HTTP white traffic filtering method based on the online learning algorithm as described in any one of claims 1-6.