A traffic data cleaning method and system

Abstract

The present application relates to the technical field of flow data filtering, and particularly relates to a flow data cleaning method and system. The method comprises the following steps: step S1: long-period time sequence feature extraction and Laplace feature mapping dimension reduction are performed on historical HTTP request flow samples to obtain a dimension-reduced feature vector set; step S2: low-frequency attack mode analysis is performed based on the dimension-reduced feature vector set to obtain an attack feature parameter set; step S3: a time attenuation factor is introduced to analyze real-time HTTP requests, obtain an attack risk assessment, and clean data based on the attack risk assessment. The full-link solution from feature mining, pattern recognition to real-time cleaning significantly improves the detection and cleaning capability for distributed low-frequency attacks, and effectively guarantees the network security of users.

Description

Technical Field

[0001] This invention relates to the field of traffic data filtering technology, and in particular to a traffic data cleaning method and system. Background Technology

[0002] With the widespread adoption of internet applications and the rapid development of network services, DDoS attacks, a highly covert and low-bandwidth-consumption attack method, pose a serious threat to various network services. Some network attacks simulate normal user behavior, sending large amounts of seemingly legitimate traffic data requests in a distributed manner, consuming server resources and preventing legitimate users from accessing target websites or services.

[0003] Existing defense measures mainly rely on static rules or simple statistical features for filtering, such as request frequency limits, IP blacklists, and access frequency threshold controls. For example, there is a technical method with existing patent publication number CN120358098A. However, this method does not address traffic data for long-cycle requests, making it difficult to identify the associated behaviors of distributed low-frequency attacks. Such attacks will not be removed during data cleaning, thus affecting users' network security. Summary of the Invention

[0004] To address the aforementioned shortcomings, the present invention aims to propose a traffic data cleaning method and system that solves the problem of insufficient detection of low-frequency attacks in existing data cleaning methods.

[0005] To achieve this objective, the present invention adopts the following technical solution: a traffic data cleaning method, comprising the following steps:

[0006] Step S1: Perform long-term time-series feature extraction and Laplacian feature mapping dimensionality reduction on historical HTTP request traffic samples to obtain a set of dimensionality-reduced feature vectors;

[0007] Step S2: Analyze low-frequency attack patterns based on the reduced-dimensional feature vector set to obtain a set of attack feature parameters;

[0008] Step S3: Introduce a time decay factor to analyze real-time HTTP requests, obtain an attack risk assessment, and clean the data based on the attack risk assessment.

[0009] Preferably, the long-cycle time-series characteristics include request distribution density greater than a time threshold, request interval variance across time windows, and low-frequency peak period.

[0010] Preferably, the set of dimensionality-reduced feature vectors is obtained in the following way:

[0011] Step S11: Preprocess the historical HTTP request traffic samples to obtain the raw data;

[0012] Step S12: Randomly select 5% of the historical attack sets as anchor points, construct a local similarity matrix, and calculate the similarity between the original data and the anchor points;

[0013] Step S13: Expand the base projection matrix using the local similarity matrix, and use the expanded base projection matrix to perform dimensionality reduction calculation on the original data to obtain the processed data;

[0014] Step S14: Divide the processed data based on the session interval duration to obtain low-frequency data and long-term data;

[0015] Step S15: Perform feature enhancement on low-frequency data to obtain enhanced data, and construct the dimensionality-reduced feature vector set using the enhanced data and long-term data.

[0016] Preferably, the preprocessing in step S11 includes:

[0017] The unified timestamp format is the ISO 8601 standard;

[0018] Standardized URL encoding;

[0019] Filter illegal characters and empty fields;

[0020] Generate based on the processed data, using the set of feature vectors as the original data;

[0021] The eigenvectors are expressed as follows:

[0022] ;

[0023] in The time interval between the current request and the last request in the same session is: SessionCount is the total number of requests in the current session; Duration is the session duration of the current request; H is the Shannon entropy value of the URL path; VersionDepth is the depth of the User-Agent version number; and NestingLevel is the nesting level of the Cookie field.

[0024] Preferably, the expression for the local similarity matrix is ​​as follows:

[0025] ;

[0026] ;

[0027] in These represent the i-th original data and the j-th anchor data, respectively. This is the bandwidth parameter.

[0028] Preferably, the expression for expanding the base projection matrix using the local similarity matrix in step S13 is as follows:

[0029] ;

[0030] ;

[0031] in Based on projection, This is the regularization coefficient, with a value ranging from 0.1 to 1. For historical similarity matrix, The historical eigenvector matrix, It is the inverse of the historical eigenvalue matrix. For feature vectors, The projection matrix is ​​obtained from historical training.

[0032] Preferably, the specific steps of step S2 are as follows:

[0033] Step S21: Calculate the dispersion of each feature vector in the dimensionality-reduced feature vector set, and store the feature vectors with dispersion less than the dispersion threshold in the first subset;

[0034] Step S22: Use a fully connected network with two layers of ReLU activation and add a time interval matching loss function to obtain the generator network. Add a random noise vector to the first subset to obtain the first processing subset.

[0035] The first processing subset is input into the generator network to obtain attack time feature samples;

[0036] Step S23: Evaluate the feature importance of the attack time feature samples, and obtain the set of attack time feature samples whose feature importance meets the condition as the attack feature parameter set;

[0037] The formula for obtaining feature importance assessment is as follows:

[0038] ;

[0039] in The importance evaluation of the j-th dimension feature, where N is the total number of attack time feature samples. For the i-th feature vector in the first processing subset, for In the output of the discriminator model, Let represent the eigenvalue of the j-th dimension of the i-th eigenvector, and corr represent the correlation between the eigenvectors over the time interval.

[0040] Preferably, step S3 is as follows:

[0041] Obtain the dimensionality-reduced feature vector of the real-time HTTP request and use it as the input vector;

[0042] The input vector is corrected by a time decay factor to obtain the corrected vector;

[0043] Obtain the similarity between the correction vector and the attack feature parameter set. When the similarity is greater than the similarity threshold, clean the traffic data of the real-time HTTP request.

[0044] The formula for obtaining the correction vector is as follows:

[0045] ;

[0046] in The time decay factor, The time interval between the current request and the last request in the same session. This is the reduced-dimensional feature vector of the current real-time HTTP request.

[0047] A traffic data cleaning system is provided to implement the traffic data cleaning method, including a dimensionality reduction module, a feature acquisition module, and a cleaning module.

[0048] The dimensionality reduction module is used to extract long-term time-series features and perform Laplacian feature mapping to reduce the dimensionality of historical HTTP request traffic samples, resulting in a set of dimensionality-reduced feature vectors.

[0049] The feature acquisition module performs low-frequency attack pattern analysis based on the reduced-dimensional feature vector set to obtain an attack feature parameter set;

[0050] The cleaning module is used to analyze real-time HTTP requests by introducing a time decay factor, obtain an attack risk assessment, and clean the data based on the attack risk assessment.

[0051] One of the above technical solutions has the following advantages or beneficial effects: the end-to-end solution from feature mining and pattern recognition to real-time cleaning significantly improves the detection and cleaning capabilities of distributed low-frequency attacks, effectively ensuring the security of users' network use. Attached Figure Description

[0052] Figure 1 This is a flowchart of one embodiment of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of one embodiment of the system of the present invention. Detailed Implementation

[0054] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0055] In the description of embodiments of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0056] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0057] like Figures 1-2 As shown, a traffic data cleaning method includes the following steps:

[0058] Step S1: Perform long-term time-series feature extraction and Laplacian feature mapping dimensionality reduction on historical HTTP request traffic samples to obtain a set of dimensionality-reduced feature vectors;

[0059] Step S2: Analyze low-frequency attack patterns based on the reduced-dimensional feature vector set to obtain a set of attack feature parameters;

[0060] Step S3: Introduce a time decay factor to analyze real-time HTTP requests, obtain an attack risk assessment, and clean the data based on the attack risk assessment.

[0061] Because existing traffic data cleaning processes lack data on long-term requests, the associated behaviors of distributed low-frequency attacks are difficult to identify. These attacks are not removed during data cleaning, thus impacting user network security. To address this, this invention first extracts long-term temporal features from historical HTTP request traffic samples, overcoming the limitations of traditional short-term analysis. This captures the dispersed characteristics of low-frequency attacks over time (such as attack request interval patterns and cross-period correlation patterns). Combined with Laplace eigenmaps dimensionality reduction technology, the data dimensionality is reduced while preserving key temporal structure information, forming a more representative set of dimensionality-reduced feature vectors, providing efficient feature input for subsequent analysis. Then, the feature vectors are dimensionality-reduced and low-frequency attack pattern analysis is performed to uncover the essential characteristic parameters of attack behavior, constructing a library containing low-frequency attack features. This makes the previously concealed distributed attack behavior explicit in the long-term feature dimension.

[0062] Finally, in step S3, a time decay factor is introduced to dynamically analyze real-time HTTP requests. Because low-frequency attacks have long time intervals, directly matching real-time HTTP requests with attack characteristic parameters for similarity could easily lead to misidentification of long-term session traffic data, resulting in users' long-interval access being blocked. Adding a time decay factor reduces the false positive rate of short-term inactive sessions, improves the success rate of attack identification, and prevents normal users' long-interval access from being blocked.

[0063] This invention forms a complete solution from feature mining and pattern recognition to real-time cleaning, which significantly improves the detection and cleaning capabilities of distributed low-frequency attacks and effectively protects users' network security.

[0064] Preferably, the long-cycle time-series characteristics include request distribution density greater than a time threshold, request interval variance across time windows, and low-frequency peak period.

[0065] Preferably, the set of dimensionality-reduced feature vectors is obtained in the following way:

[0066] Step S11: Preprocess the historical HTTP request traffic samples to obtain the raw data. By preprocessing the historical HTTP request traffic samples, the data quality is ensured and noise interference is eliminated, providing a high-quality raw data foundation for subsequent analysis and avoiding misjudgment of attack patterns due to data quality issues.

[0067] Step S12: Randomly select 5% of the historical attack sets as anchor points and construct a local similarity matrix. Calculate the similarity between the original data and the anchor points. Randomly select 5% of the historical attack sets as anchor points and calculate the similarity between the original data and the anchor points (e.g., cosine similarity, Euclidean distance). Using a small number of representative historical attack sets as reference points, quickly capture local features in the data related to known attack patterns. The advantage is that the similarity calculation guided by anchor points reduces the complexity of global feature calculation while retaining sensitivity to key attack features. Of course, when the administrator's equipment configuration is high, the percentage of anchor points selected can be appropriately increased to better obtain attack features.

[0068] Step S13: Expand the base projection matrix using a local similarity matrix, and use the expanded base projection matrix to perform dimensionality reduction calculations on the original data to obtain the processed data; use the local similarity matrix to perform Nystrom expansion on the base projection matrix, integrating incremental data features into the historical model, forming an expanded projection matrix to reduce the dimensionality of the original data. This incremental learning mechanism achieves streaming data processing capabilities while preserving the topology-preserving properties of the Laplacian eigenmap. The base projection matrix is ​​the linear transformation matrix in the Laplacian eigenmap that maps high-dimensional original data to a low-dimensional feature space. Its core purpose is dimensionality reduction and feature extraction.

[0069] Step S14: Divide the processed data based on session interval duration to obtain low-frequency data and long-term data. Before proceeding to step S14, it is necessary to statistically analyze the session interval duration based on the habits of most users and calculate the average time between adjacent sessions. This average session interval duration will be used as the criterion for division. For example, if the interval between adjacent sessions in a sample is greater than this average time, it indicates that the access frequency of this sample's traffic data is low, belonging to low-frequency data, and requires additional attention.

[0070] If the interval between adjacent sessions in a sample is less than this average time, it indicates that the traffic data in this sample is accessed frequently and belongs to long-term data. Data augmentation techniques can then be used to increase the attention given to low-frequency data, in order to better identify the characteristics of low-frequency attacks.

[0071] Step S15: Perform feature enhancement on the low-frequency data to obtain enhanced data, and construct the dimensionality-reduced feature vector set using the enhanced data and long-term data. The expression for feature enhancement of low-frequency data in step S15 is as follows:

[0072] .

[0073] Preferably, the preprocessing in step S11 includes:

[0074] The unified timestamp format is the ISO 8601 standard;

[0075] Normalized URL encoding (percent sign encoding conversion);

[0076] Filter illegal characters and empty fields;

[0077] Generate based on the processed data, using the set of feature vectors as the original data;

[0078] The eigenvectors are expressed as follows:

[0079] ;

[0080] in The time interval between the current request and the last request in the same session is: SessionCount is the total number of requests in the current session; Duration is the session duration of the current request; H is the Shannon entropy value of the URL path; VersionDepth is the depth of the User-Agent version number; and NestingLevel is the nesting level of the Cookie field.

[0081] Preferably, the expression for the local similarity matrix is ​​as follows:

[0082] ;

[0083] ;

[0084] in These represent the i-th original data and the j-th anchor data, respectively. This is the bandwidth parameter (controls the similarity decay rate; the default value is the feature standard deviation).

[0085] Preferably, the expression for expanding the base projection matrix using the local similarity matrix in step S13 is as follows:

[0086] ;

[0087] ;

[0088] in Based on projection, This is the regularization coefficient, with a value ranging from 0.1 to 1. For historical similarity matrix, The historical eigenvector matrix, It is the inverse of the historical eigenvalue matrix. For feature vectors, The projection matrix is ​​obtained from historical training.

[0089] Historical similarity matrix It uses similarity measurement methods such as Gaussian kernel function to calculate the similarity between all sample pairs (or sample pairs filtered by k nearest neighbors) in historical HTTP request traffic samples.

[0090] Historical eigenvector matrix and the inverse of the historical eigenvalue matrix The eigenvalues ​​are obtained by performing eigenvalue decomposition on historical HTTP request traffic samples using the Laplace matrix.

[0091] The projection matrix obtained from historical training is a matrix concatenated by selecting the eigenvectors corresponding to the first few largest eigenvalues ​​after performing eigenvalue decomposition on historical HTTP request traffic samples using the Laplacian matrix.

[0092] Preferably, the specific steps of step S2 are as follows:

[0093] Step S21: Calculate the dispersion of each feature vector in the dimensionality-reduced feature vector set, and store the feature vectors with dispersion less than the dispersion threshold in the first subset;

[0094] Step S22: Use a fully connected network with two layers of ReLU activation and add a time interval matching loss function to obtain the generator network. Add a random noise vector to the first subset to obtain the first processing subset.

[0095] The first processing subset is input into the generator network to obtain attack time feature samples;

[0096] Step S23: Evaluate the feature importance of the attack time feature samples, and obtain the set of attack time feature samples whose feature importance meets the condition as the attack feature parameter set;

[0097] The formula for obtaining feature importance assessment is as follows:

[0098] ;

[0099] in The importance evaluation of the j-th dimension feature, where N is the total number of attack time feature samples. For the i-th feature vector in the first processing subset, for In the output of the discriminator model, Let represent the feature value of the j-th dimension of the i-th feature vector, and corr represent the correlation between feature vectors over time intervals. The discriminator model used is Conditional GAN ​​(CGAN) Discriminator.

[0100] When analyzing low-frequency attack patterns, the dispersion of each feature vector in the dimensionality-reduced feature vector set is calculated first (using the ratio of standard deviation to mean as a quantitative indicator). Feature vectors with dispersion below a preset threshold (0.5 recommended) are then filtered and stored in the first subset. This operation is based on the core principle that low-frequency attack features have high stability, effectively filtering out random fluctuation noise.

[0101] However, due to the limited number of low-frequency attack features and the low percentage of anchor points selected in the earlier step S12, in order to overcome the limitation of insufficient traditional samples, a first processing subset is generated by injecting normally distributed random noise vectors into the first subset. This first subset is then synthesized by a generation network to increase the number of attack time feature samples and improve the coverage of low-frequency attack features.

[0102] Attack event feature samples are not necessarily features of low-frequency attacks. Therefore, feature importance assessment is needed to select them. In this invention, a range is set. When the feature importance assessment score falls within this range, it is considered an attack time feature sample of a low-frequency attack. Finally, all attack time feature samples of low-frequency attacks are statistically analyzed to construct an attack feature parameter set.

[0103] Preferably, step S3 is as follows:

[0104] Obtain the dimensionality-reduced feature vector of the real-time HTTP request and use it as the input vector;

[0105] The input vector is corrected by a time decay factor to obtain the corrected vector;

[0106] Obtain the similarity between the correction vector and the attack feature parameter set. When the similarity is greater than the similarity threshold, clean the traffic data of the real-time HTTP request.

[0107] The formula for obtaining the correction vector is as follows:

[0108] ;

[0109] in This represents the time decay factor, where the value of the time decay factor ranges from 0.001 to 0.01. The time interval between the current request and the last request in the same session. This is the reduced-dimensional feature vector of the current real-time HTTP request.

[0110] In step S3, step S1 needs to be executed on the real-time HTTP request to obtain the dimensionality-reduced feature vector of the real-time traffic data, which serves as the input vector. If the input vector is directly matched with the attack feature parameter set for similarity judgment, it is easy to misidentify long-term session traffic data, leading to the blocking of users' long-interval accesses. Therefore, a time decay factor needs to be added to the input vector for correction. The time decay factor dynamically balances real-time performance and historical correlation, avoiding both over-reliance on old data leading to missed new attacks and misjudging normal, dispersed requests by ignoring long-term patterns. The corrected feature vector better reflects actual attack trends, improving the accuracy of similarity matching.

[0111] It is worth mentioning that there are multiple attack time feature samples in the attack feature parameter set. The correction vector needs to be calculated similarly to each attack time feature sample. When there is an attack time feature sample with a similarity greater than the similarity threshold, the real-time HTTP request will be cleaned.

[0112] A traffic data cleaning system is provided to implement the traffic data cleaning method, including a dimensionality reduction module, a feature acquisition module, and a cleaning module.

[0113] The dimensionality reduction module is used to extract long-term time-series features and perform Laplacian feature mapping to reduce the dimensionality of historical HTTP request traffic samples, resulting in a set of dimensionality-reduced feature vectors.

[0114] The feature acquisition module performs low-frequency attack pattern analysis based on the reduced-dimensional feature vector set to obtain an attack feature parameter set;

[0115] The cleaning module is used to analyze real-time HTTP requests by introducing a time decay factor, obtain an attack risk assessment, and clean the data based on the attack risk assessment.

[0116] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0117] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A traffic data cleaning method, characterized in that, Comprising the following steps: Step S1: long-period timing feature extraction and Laplace feature mapping dimension reduction are performed on the historical HTTP request traffic sample to obtain a set of dimension-reduced feature vectors; Step S2: low-frequency attack mode analysis is performed based on the set of dimension-reduced feature vectors to obtain a set of attack feature parameters; The specific steps of step S2 are as follows: Step S21: the dispersion of each feature vector in the set of dimension-reduced feature vectors is calculated, and the feature vectors with a dispersion less than a dispersion threshold are stored in a first subset; Step S22: a double-layer ReLU activated full connection network is used, and a time interval matching loss function is added to obtain a generation network, a random noise vector is added to the first subset to obtain a first processing subset; The first processing subset is input into the generation network to obtain an attack time feature sample; Step S23: feature importance evaluation is performed on the attack time feature sample, and a union of attack time feature samples that meet the feature importance condition is obtained as the set of attack feature parameters; The formula for obtaining feature importance evaluation is as follows: ； wherein importance evaluation of the jth dimension feature, N is the total number of attack time feature samples, is the ith feature vector in the first processing subset, is at the output of the discriminator model, denotes the jth dimension feature value of the ith feature vector, and corr is the correlation degree of the feature vector in the time interval; Step S3: a time decay factor is introduced to analyze real-time HTTP requests to obtain attack risk assessment, and the data is cleaned based on the attack risk assessment; The steps of step S3 are as follows: Obtain the dimension-reduced feature vector of the real-time HTTP request as the input vector; The input vector is modified by the time decay factor to obtain a modified vector; Obtain the similarity between the modified vector and the set of attack feature parameters, and when the similarity is greater than a similarity threshold, clean the traffic data of the real-time HTTP request; The formula for obtaining the modified vector is as follows: ； wherein is a time decay factor, is the time interval between the current request and the last request on the session, is the reduced dimension feature vector of the current real-time HTTP request.

2. The flow data cleaning method of claim 1, wherein, The long-period timing feature includes request distribution density greater than a time threshold, request interval variance across time windows, and low-frequency peak period.

3. The flow data cleaning method of claim 2, wherein, The set of dimension-reduced feature vectors is obtained as follows: Step S11: pre-process the historical HTTP request traffic sample to obtain raw data; Step S12: randomly obtain 5% from the historical attack set as anchor points, and construct a local similarity matrix to calculate the similarity between the raw data and the anchor points; Step S13: use the local similarity matrix to expand the basic projection matrix, use the expanded basic projection matrix to perform dimension reduction calculation on the raw data to obtain processed data; Step S14: divide the processed data based on session interval duration to obtain low-frequency data and long-term data; Step S15: obtain enhanced data by enhancing the low-frequency data, and construct the set of dimension-reduced feature vectors based on the enhanced data and the long-term data.

4. The flow data cleaning method of claim 3, wherein, The pre-processing of step S11 includes: Unify the timestamp format to ISO 8601 standard; Standardize URL encoding; Filter illegal characters and null fields; Generate based on the processed data, with the set of feature vectors as the raw data; The expression of the feature vector is as follows: ； wherein is the time interval between the current request and the last request on the session, SessionCount is the total number of requests for the current session, Duration is the session duration of the current request, H is the Shannon entropy value of the URL path, VersionDepth is the User-Agent version number depth, and NestingLevel is the number of Cookie field nesting levels.

5. The method of claim 3, wherein, The expression of the local similarity matrix is as follows: ； ； wherein respectively represent the i-th original data and the j-th anchor data, is a bandwidth parameter.

6. The method of claim 3, wherein, The expression of using the local similarity matrix to expand the basic projection matrix in step S13 is as follows: ； ； wherein is a base projection, is a regularization coefficient, and takes a value of 0.1~1, is a historical similarity matrix, is a historical feature vector matrix, is an inverse of a historical eigenvalue matrix, is a feature vector, is a projection matrix obtained by historical training.

7. A traffic data cleansing system characterized by, The application discloses a traffic data cleaning method and device, and a computer device and a computer program product. The method comprises a dimension reduction module, a feature acquisition module and a cleaning module. The dimension reduction module is used for extracting long-period timing features and performing Laplace feature mapping dimension reduction on historical HTTP request traffic samples to obtain a dimension reduction feature vector set. The feature acquisition module performs low-frequency attack mode analysis based on the dimension reduction feature vector set to obtain an attack feature parameter set. The cleaning module is used for introducing a time attenuation factor to analyze real-time HTTP requests, obtaining an attack risk assessment, and cleaning data based on the attack risk assessment.

Images