A traffic data anomaly detection method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a traffic data anomaly detection method, device and equipment, and a storage medium, relating to the technical field of network security. The method comprises: collecting normal traffic data of an access website, preprocessing all the normal traffic data to obtain all training data; combining an elastic adaptive deep self-encoding module and a sparse key point interactive attention module to establish a traffic data anomaly detection model; training the traffic data anomaly detection model according to all the training data, so that the traffic data anomaly detection model self-learns the features of all the training data; combining the features of all the training data self-learned by the traffic data anomaly detection model to generate a normal traffic data rule set; and judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set to obtain a detection result of the to-be-detected traffic data. The embodiments of the present application can comprehensively and accurately detect traffic data anomalies, effectively avoiding false detection and missed detection problems.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a traffic data anomaly detection method and device, equipment and a storage medium. BACKGROUND

[0002] A website application level intrusion prevention system (Web Application Firewall, WAF) device can detect anomalies in traffic data accessing a website and block abnormal traffic data to ensure network security. Currently, a commonly used traffic data anomaly detection method is to configure a rule library in the WAF device, so that the WAF device judges whether the traffic data is abnormal traffic data based on the rule library. However, the rule library mainly relies on manual professional experience configuration, and it is difficult to comprehensively and accurately detect anomalies in traffic data, and false positives and false negatives are likely to occur. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a traffic data anomaly detection method, device, equipment and storage medium, which can comprehensively and accurately detect anomalies in traffic data and effectively avoid false positives and false negatives.

[0004] In a first aspect, the embodiments of the present application provide a traffic data anomaly detection method, comprising:

[0005] Collect normal traffic data accessing a website, pre-process all the normal traffic data to obtain all training data;

[0006] Combine an elastic adaptive deep auto-encoding module and a sparse key point interactive attention module to establish a traffic data anomaly detection model;

[0007] Train the traffic data anomaly detection model according to all the training data, so that the traffic data anomaly detection model learns the features of all the training data;

[0008] Combine the features of all the training data learned by the traffic data anomaly detection model to generate a normal traffic data rule set;

[0009] Judge whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set to obtain a detection result of the to-be-detected traffic data.

[0010] In the implementation process, by combining the elastic adaptive deep self-encoding module and the sparse key point interactive attention module, a traffic data anomaly detection model is established, all normal traffic data of legal access websites are preprocessed as training data to train the traffic data anomaly detection model, the traffic data anomaly detection model is self-learned the features of all training data, the features of all training data are combined to generate a normal traffic data rule set, and based on the normal traffic data rule set, the abnormality of the to-be-detected traffic data is detected, which can comprehensively and accurately detect the abnormality of the traffic data, and effectively avoid the problems of false detection and missed detection.

[0011] Further, before the step of judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the to-be-detected traffic data, the method further comprises:

[0012] real-time acquiring traffic data accessing the website as the to-be-detected traffic data;

[0013] judging whether the to-be-detected traffic data is unknown traffic data according to all the normal traffic data;

[0014] when the to-be-detected traffic data is unknown traffic data, recording the to-be-detected traffic data.

[0015] In the implementation process, before the step of detecting the abnormality of the to-be-detected traffic data based on the normal traffic data rule set, the to-be-detected traffic data is judged whether it is unknown traffic data according to all the normal traffic data, and the to-be-detected traffic data judged as unknown traffic data is recorded, which can provide the recorded unknown to-be-detected traffic data and its detection result after detecting the abnormality of the unknown to-be-detected traffic data, and provide a reference basis for the user to provide its detection result, so as to further realize comprehensive and accurate detection of the abnormality of the traffic data, and effectively avoid the problems of false detection and missed detection.

[0016] Further, before the step of judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the to-be-detected traffic data, the method further comprises:

[0017] when the number of all the recorded to-be-detected traffic data reaches a preset threshold, all the recorded to-be-detected traffic data are taken as new training data;

[0018] updating the traffic data anomaly detection model according to all the new training data through a continuous scalable learning module, so that the traffic data anomaly detection model re-learns the features of all the new training data;

[0019] Update the normal traffic data rule set in combination with the features of all the new training data that the traffic data anomaly detection model relearns.

[0020] In the implementation process described above, by using the continuous scalable learning module to train the traffic data anomaly detection model with all unknown traffic data when too much unknown traffic data is obtained, the model learns the features of all unknown traffic data to update the normal traffic data rule set, which can comprehensively and accurately detect unknown traffic data and effectively avoid false positives and false negatives.

[0021] Further, the training of the traffic data anomaly detection model based on all the training data to enable the model to learn the features of all the training data includes:

[0022] For each training data, the training data is compressed by an encoder to extract the target features of the training data, and the data is reconstructed by a decoder based on the target features of the training data to obtain reconstructed data, so that the elastic adaptive deep auto-encoding module learns the features of the training data with the goal of minimizing reconstruction error; wherein the elastic adaptive deep auto-encoding module includes the encoder and the decoder;

[0023] The key features of the training data are extracted by the sparse key point interactive attention module, and an attention map is generated based on the key features of the training data, so that the sparse key point interactive attention module learns the features of the training data with the goal of minimizing the attention map loss;

[0024] In combination with the features of the training data learned by the elastic adaptive deep auto-encoding module and the sparse key point interactive attention module, the training data is detected to obtain the detection result of the training data, and the model parameters of the traffic data anomaly detection model are adjusted based on the detection result of the training data.

[0025] In the implementation process described above, by training the traffic data anomaly detection model based on all the training data, the elastic adaptive deep auto-encoding module and the sparse key point interactive attention module learn the features of all the training data, which can further comprehensively and accurately detect traffic data and effectively avoid false positives and false negatives.

[0026] Further, the determination of whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set to obtain the detection result of the to-be-detected traffic data includes:

[0027] Based on the normal traffic data rule set, it is judged whether the to-be-detected traffic data conforms to any normal traffic data rule in the normal traffic data rule set, and if yes, the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is not abnormal traffic data.

[0028] In the implementation process, when it is determined that the to-be-detected traffic data conforms to any normal traffic data rule in the normal traffic data rule set, it is determined that the to-be-detected traffic data is not abnormal traffic data, so that the normal traffic data can be quickly detected, and the abnormal detection efficiency is improved.

[0029] Further, after the to-be-detected traffic data is judged to be abnormal traffic data based on the normal traffic data rule set, the detection result of the to-be-detected traffic data is obtained, and the method further comprises:

[0030] When the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is abnormal traffic data, the to-be-detected traffic data is intercepted.

[0031] In the implementation process, when it is determined that the to-be-detected traffic data is abnormal traffic data, the to-be-detected traffic data is intercepted, so that the abnormal traffic data can be intercepted in time, and the network security is ensured.

[0032] Further, the preprocessing comprises data cleaning, data conversion, and data normalization processing.

[0033] In the implementation process, all normal traffic data is preprocessed by selecting data cleaning, data conversion, and data normalization processing, and all training data is obtained, so that the traffic data anomaly detection model can be effectively trained according to all training data, and the traffic data anomaly detection model can self-learn the features of all training data.

[0034] In a second aspect, an embodiment of the present application provides a traffic data anomaly detection device, comprising:

[0035] A data processing unit is configured to collect normal traffic data of algorithmically accessed websites, preprocess all the normal traffic data, and obtain all training data.

[0036] A model establishing unit is configured to combine an elastic adaptive deep self-encoding module and a sparse key point interactive attention module to establish a traffic data anomaly detection model.

[0037] A feature learning unit is configured to train the traffic data anomaly detection model according to all the training data, so that the traffic data anomaly detection model self-learns the features of all the training data.

[0038] a rule generation unit configured to generate a normal traffic data rule set in combination with features of all the training data self-learned by the traffic data anomaly detection model;

[0039] an anomaly detection unit configured to determine whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set, and obtain a detection result of the to-be-detected traffic data.

[0040] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; the memory is coupled to the processor, and the processor implements the traffic data anomaly detection method when executing the computer program.

[0041] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, including a stored computer program; wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the traffic data anomaly detection method when the computer program runs. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0043] Figure 1 a flowchart of a traffic data anomaly detection method provided by the first embodiment of the present application;

[0044] Figure 2 a data flow diagram of a training traffic data anomaly detection model provided by an optional embodiment of the first embodiment of the present application;

[0045] Figure 3 a structural diagram of a traffic data anomaly detection device provided by the second embodiment of the present application;

[0046] Figure 4 a structural diagram of an electronic device provided by the third embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

[0048] It should be noted that in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. At the same time, the step numbers in the text are only for the convenience of explaining the embodiments of the present application, and do not serve as the function of limiting the execution sequence of the steps. The method provided by the embodiments of the present application can be executed by the related terminal device, and hereinafter the WAF device is taken as an example for execution.

[0049] It should be noted that deep learning (Deep Learning) is a branch of machine learning that models the neural network structure of the human brain and can be used for many different tasks, such as image recognition, speech recognition, natural language processing, game strategy, etc. Deep learning uses multiple neural network layer structures to extract abstract feature representations of input data, and each neural network layer contains multiple neurons or nodes, each of which weights the input data and applies an activation function to generate an output result. By continuously using these hierarchical structures, deep learning models can learn more complex feature representations. Compared with traditional machine learning methods, deep learning models usually require more data and more computing resources for training. However, because it can model a large amount of data and can handle complex nonlinear relationships, it has achieved very good performance on many tasks. Some common architectures of deep learning models include: Convolutional Neural Networks (CNN) for image processing; Recurrent Neural Networks (RNN) for sequence data processing; Transformers for natural language processing. At the same time, deep learning is often used in combination with traditional machine learning methods to improve the performance of the model.

[0050] Self-supervised Learning refers to the ability of a machine learning system to learn from data without explicit supervision or guidance. It is also known as unsupervised learning or self-learning. Unlike supervised learning, where a human expert designs and selects appropriate features and algorithms, and then trains and tunes the model using a large amount of labeled data, self-supervised learning allows the model to learn from data on its own, without much human intervention. Self-supervised learning is often implemented using deep learning algorithms, which can automatically learn features from large amounts of data and adjust the model's weights and parameters based on feedback signals, to improve its performance. This makes self-supervised learning more flexible and adaptive to changing environments and tasks. Self-supervised learning is often classified as a type of unsupervised learning, as it can learn from data without explicit labels or target functions. In self-supervised learning, the goal of the model is to discover patterns and structures in the data, rather than directly learning how to perform a specific task or predict a target value. However, some research has incorporated supervised signals into self-supervised algorithms. For example, in deep learning, a model can be trained by adding a classification task output to an autoencoder, to obtain better representation capabilities. This approach can be considered as semi-supervised learning, as it uses both labeled and unlabeled data for learning. Semi-supervised learning is a learning method that uses both labeled and unlabeled data. In semi-supervised learning, the algorithm usually learns from a small set of labeled data and a large set of unlabeled data, to improve the model's performance and generalization ability. Currently, we are used to classifying machine learning algorithms into four categories: supervised learning, semi-supervised learning, unsupervised learning, and reinforcement learning. The first three have been introduced earlier, while reinforcement learning is essentially an algorithm that learns a behavior policy by interacting with the environment to maximize long-term rewards. Reinforcement learning algorithms do not directly tell you how to process input data, but rather learn from experience by learning how to interact with the environment to maximize rewards. In summary, supervised learning, semi-supervised learning, unsupervised learning, and reinforcement learning are all common learning methods in machine learning, but they each handle different problems and methods, so we need to choose the appropriate learning method based on the specific task.

[0051] Anomaly Detection is to find special observations in data that do not conform to expected behavior. Anomaly detection aims to detect those observations that are different from the rest of the data distribution, usually referred to as anomalies, outliers, outliers or noise points. Anomaly detection can be applied in various fields, such as fraud detection in the financial field, disease diagnosis in the medical field, intrusion detection in the network security field, fault detection in the industrial field, etc. Commonly used anomaly detection methods include statistical-based methods (such as box plots, probability distribution models, etc.), machine learning-based methods (such as clustering, support vector machines, deep learning, etc.), and time series-based methods, etc. These methods all try to extract features from data and use some indicators or models to quantify the difference between each observation and the expected behavior, so as to identify outliers. When we only collect positive sample data and fail to cover all negative samples, we can use anomaly detection to handle this kind of task.

[0052] The task studied by the embodiments of the present application is this kind of task. We only have positive samples, but we do not know the negative samples. Common anomaly detection algorithms include statistical-based methods, clustering-based methods, support vector machine (SVM) based methods, density-based methods, and deep learning-based methods. Among them, the autoencoder (Autoencoder) based method is a deep learning method that can be used to learn the representation of input data, and then identify outliers by measuring reconstruction error. Other deep learning-based anomaly detection algorithms include GAN-based anomaly detection, etc.

[0053] Attention Mechanism is a widely used technique in computer vision and natural language processing. Its core idea is to let the model focus on some important information in the input data when processing it, thereby improving the performance of the model. Attention mechanism originated from natural language processing, but has now been extended to other fields such as computer vision. The basic principle of attention mechanism is to highlight important information by learning the weight distribution of input data. These weight distributions can be seen as a probability distribution, which corresponds to the importance of each element (such as words or image regions) in the input data. When calculating the weight distribution, the model will consider the context information of the input data. After obtaining the weight distribution, the model can calculate a context vector by weighted summation, which can capture important information in the input data. In natural language processing, attention mechanism is often applied to sequence-to-sequence (Seq2Seq) models, such as machine translation, abstract generation and other tasks. This model includes an encoder and a decoder, which are responsible for encoding input data into fixed-length context vectors and generating output data from context vectors, respectively. By using attention mechanism, the decoder can focus on the input part related to the current generation position when generating output data. In the field of computer vision, attention mechanism has also been widely applied, such as image classification, target detection and image description generation, etc. In these tasks, attention mechanism can help the model focus on important regions in the image, thereby improving performance. Attention mechanism improves the performance of the model by learning important information in the input data. It has been widely used in computer vision and natural language processing and has achieved significant results in many practical tasks.

[0054] Continual Learning, also known as Incremental Learning or Lifelong Learning, is an important research direction in the field of machine learning. Continual Learning aims to solve a common problem: how to avoid forgetting previously learned knowledge when a model learns new tasks or new data during training. In other words, Continual Learning tries to make the model maintain performance on old tasks while learning new tasks. Traditional deep learning models often encounter the problem of catastrophic forgetting during training. When a model is trained on a new task, it may "forget" the relevant knowledge of previous tasks. This is because the weights in the neural network are constantly updated to adapt to the new task, which may cause the original knowledge to be overwritten. We can solve the problem mentioned earlier in the following ways: (1) Replay, this method stores part or all of the data of previous tasks in a buffer. When training a new task, the model will train these stored old task data at the same time to prevent forgetting. This can be achieved through experience replay or pseudo-sample generation, etc. (2) Regularization: To preserve the knowledge of previous tasks, a regularization term can be introduced during training. For example, the Elastic Weight Consolidation (EWC) method calculates the weights important to previous tasks and adds a regularization term to them to slow down the update speed of these weights on the new task. (3) Dynamic Network Expansion: This method avoids forgetting by expanding the network structure when facing new tasks. For example, adding new neurons or hidden layers, so that the network can learn new tasks while retaining the knowledge of old tasks. The goal of Continual Learning is to make the model learn new tasks without compromising its performance on old tasks. This is a challenging research direction, but it is essential for achieving an intelligent agent similar to a human. Humans can easily switch between different tasks while maintaining their mastery of old tasks without forgetting them when learning new tasks. In practical applications, Continual Learning helps improve the adaptability of models, enabling them to make effective adjustments when facing new tasks or dynamic environments.

[0055] Please refer to Figure 1 , Figure 1 A flowchart of a traffic data anomaly detection method provided by the first embodiment of the present application. The first embodiment of the present application provides a traffic data anomaly detection method, comprising steps S101-S105:

[0056] S101, collect normal traffic data of the accessed website, pre-process all the normal traffic data to obtain all training data;

[0057] S102, establish a traffic data anomaly detection model in combination with an elastic adaptive deep autoencoder module and a sparse key point interactive attention module;

[0058] S103, train the traffic data anomaly detection model according to all the training data, so that the traffic data anomaly detection model learns the features of all the training data;

[0059] S104, generate a normal traffic data rule set in combination with the features of all the training data learned by the traffic data anomaly detection model;

[0060] S105, determine whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set, and obtain a detection result of the to-be-detected traffic data.

[0061] As an example, normal traffic data of the accessed website is collected, and all the normal traffic data includes request header information, URL (uniform resource locator) parameters, request methods, etc. All the normal traffic data is pre-processed through data cleaning, data conversion, data normalization processing, etc. to obtain all the training data.

[0062] An elastic adaptive deep autoencoder (EADA) module is designed. The elastic adaptive mechanism can adapt the elastic adaptive deep autoencoder module to unknown traffic data, automatically adjust the complexity and learning ability of the model, and maintain good performance in different scenarios. An autoencoder (AE) is a type of artificial neural network (ANN) used in unsupervised learning, which is used for representation learning by taking input data as a learning target. For traffic data, the autoencoder can obtain the features of the traffic data by learning the internal structure and rules of the traffic data, so that the autoencoder can automatically capture important information of the traffic data without manual labeling of the traffic data, reducing the dependence on manually designed rules.

[0063] A sparse key interactive attention module (SKIAM) is designed. The attention mechanism used in the sparse key interactive attention model can better focus on important information in the input data, improving the performance of the model in complex tasks. This makes the model more effective in identifying features in sparse sequence data when processing sparse sequence data, reducing the rate of missed detection.

[0064] The elastic adaptive deep auto-encoding module and the sparse key interactive attention module are combined to establish a traffic data anomaly detection model.

[0065] The traffic data anomaly detection model is trained according to all training data, so that the traffic data anomaly detection model learns the features of all training data.

[0066] The traffic data anomaly detection model learns the features of all training data, and generates a normal traffic data rule set. The features learned by the traffic data anomaly detection model include domain name parameters of training data, key fields of training data, and key fields include parameter names, parameter value ranges, parameter combinations, etc. The normal traffic data rule set includes at least one normal traffic data rule. The normal traffic data rule can be based on a specific parameter threshold, a specific field combination, or the relationship between features. The form of the normal traffic data rule can be a simple conditional judgment or a more complex logical expression. For example, the normal traffic data rule set includes: (1) the length of the request URL is within a predetermined length range, such as the length of the request URL being less than 100 characters; (2) the number of request parameters is within a predetermined number range, such as the number of request parameters being less than 10; (3) the request method belongs to a predetermined request method, such as the request method being GET or POST; (4) the value of the target key parameter meets the predetermined value condition, such as the parameter value of the user ID being a number and being between 10000 and 50000; (5) the target parameter combination meets the predetermined mapping condition, such as when the value of parameter A is "x", the value of parameter B must be "y".

[0067] Obtain the traffic data to be detected, and based on the normal traffic data rule set, determine whether the traffic data to be detected is abnormal traffic data, and obtain the detection result of the traffic data to be detected.

[0068] The embodiment of the application combines the elastic adaptive deep self-encoding module and the sparse key point interactive attention module, establishes a traffic data anomaly detection model, pre-processes all normal traffic data of legal access websites as training data to train the traffic data anomaly detection model, enables the traffic data anomaly detection model to self-learn the features of all training data, combines the features of all training data, generates a normal traffic data rule set, performs anomaly detection on the to-be-detected traffic data based on the normal traffic data rule set, can comprehensively and accurately perform anomaly detection on the traffic data, and effectively avoids the problems of false detection and missed detection.

[0069] In optional embodiments, before the step of judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the to-be-detected traffic data, the method further includes the following steps: acquiring traffic data of accessing websites in real time as to-be-detected traffic data; judging whether the to-be-detected traffic data is unknown traffic data according to all normal traffic data; and recording the to-be-detected traffic data when the to-be-detected traffic data is unknown traffic data.

[0070] As an example, considering that the normal traffic data rule set is obtained by the traffic data anomaly detection model self-learning the features of all normal traffic data, anomaly detection on unknown to-be-detected traffic data based on the normal traffic data rule set may not be accurate, the traffic data of accessing websites is acquired in real time as to-be-detected traffic data, each normal traffic data is compared with the to-be-detected traffic data respectively, if the to-be-detected traffic data is consistent with any normal traffic data, it is considered that the to-be-detected traffic data is not unknown traffic data, the normal traffic data rule set can be directly used to judge whether the to-be-detected traffic data is abnormal traffic data, and the detection result of the to-be-detected traffic data is obtained, if the to-be-detected traffic data is inconsistent with each normal traffic data, it is considered that the to-be-detected traffic data is unknown traffic data, the to-be-detected data needs to be recorded first, and then the normal traffic data rule set is used to judge whether the to-be-detected traffic data is abnormal traffic data, and the detection result of the to-be-detected traffic data is obtained.

[0071] The embodiment of the application judges whether the to-be-detected traffic data is unknown traffic data according to all normal traffic data before performing anomaly detection on the to-be-detected traffic data based on the normal traffic data rule set, and records the to-be-detected traffic data that is judged as unknown traffic data, can provide the recorded unknown to-be-detected traffic data and its detection result after performing anomaly detection on the unknown to-be-detected traffic data, provides a reference basis for the user of the detection result, and thus further comprehensively and accurately performs anomaly detection on the traffic data, and effectively avoids the problems of false detection and missed detection.

[0072] In an optional embodiment, before judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set to obtain a detection result of the to-be-detected traffic data, the method further comprises: when the number of all the to-be-detected traffic data recorded reaches a preset threshold, taking all the to-be-detected traffic data recorded as new training data; updating the traffic data anomaly detection model according to all the new training data through a continue scalable learning module (CSLM) to make the traffic data anomaly detection model re-learn the features of all the new training data; and updating the normal traffic data rule set in combination with the features of all the new training data re-learned by the traffic data anomaly detection model.

[0073] As an example, a continue scalable learning module (CSLM) is designed, and the continue learning adopted by the continue scalable learning module can dynamically adjust the structure and parameters of the model according to the size and distribution of new input data to adapt to the changes of the traffic data and maintain the learning ability.

[0074] When the number of all the to-be-detected traffic data recorded reaches a preset threshold, all the to-be-detected traffic data recorded are taken as new training data, the traffic data anomaly detection model is continuously trained according to all the new training data through a continue scalable learning module (CSLM) to update the traffic data anomaly detection model, and the traffic data anomaly detection model re-learns the features of all the new training data.

[0075] Among them, considering that the traffic data generally has time sequence features, the time sequence features can be extracted from all the new training data through the continue scalable learning module using a sliding window, time difference and other methods, so that the model can capture the relationship on the time sequence when training.

[0076] In combination with the features of all the new training data re-learned by the traffic data anomaly detection model, at least one new normal traffic data rule is generated, all the new normal traffic data rules are added to the normal traffic data rule set, and the normal traffic data rule set is updated.

[0077] The embodiments of the present application can comprehensively and accurately detect the unknown traffic data, effectively avoid the false detection and missed detection problems by using the continue scalable learning module to train the traffic data anomaly detection model with all the unknown traffic data to make the traffic data anomaly detection model re-learn the features of all the unknown traffic data to update the normal traffic data rule set when too much unknown traffic data is obtained.

[0078] Please refer to Figure 2 , Figure 2A data flow diagram of the training traffic data anomaly detection model is provided for an optional embodiment of the first embodiment of the present application. In the optional embodiment, the traffic data anomaly detection model is trained according to all the training data, so that the traffic data anomaly detection model learns the features of all the training data, specifically including: for each training data, the training data is compressed by the encoder to extract the target features of the training data, the data is reconstructed according to the target features of the training data by the decoder to obtain the reconstructed data, so that the elastic adaptive deep auto-encoding module learns the features of the training data with the objective of minimizing the reconstruction error; wherein the elastic adaptive deep auto-encoding module includes the encoder and the decoder; the key features of the training data are extracted by the sparse key point interactive attention module, and the attention map is generated according to the key features of the training data, so that the sparse key point interactive attention module learns the features of the training data with the objective of minimizing the attention map loss; the features of the training data learned by the elastic adaptive deep auto-encoding module and the sparse key point interactive attention module are combined to detect the abnormality of the training data, obtain the detection result of the training data, and adjust the model parameters of the traffic data anomaly detection model according to the detection result of the training data.

[0079] As an example, as shown in FIG. 1, after obtaining all the training data, each training data (Original Data) is input into the traffic data anomaly detection model. Figure 2

[0080] The training data is compressed by the encoder in the elastic adaptive deep auto-encoding module to extract the target features (Compression Feature) of the training data, the data is reconstructed according to the target features of the training data by the decoder in the elastic adaptive deep auto-encoding module to obtain the reconstructed data (Reconstruct Data), so that the elastic adaptive deep auto-encoding module learns the features of the training data with the objective of minimizing the reconstruction error (L ce ). Wherein the features of the training data learned by the elastic adaptive deep auto-encoding module include the domain name parameters in the training data.

[0081] ​The attention mechanism adopted by the sparse key point interactive attention module can automatically assign different weights according to the importance of various information in the input data, so that the model can pay more attention to important information in the input data and reduce the missed detection rate. Specific applications include: (1) inputting training data into the sparse key point interactive attention module, the training data including request header information, URL parameters, request method, etc.; (2) key field identification, the sparse key point interactive attention module automatically identifies key fields by learning the patterns of the training data, the key fields including parameter names, parameter value ranges, parameter combinations, etc., and the key fields are usually closely related to the judgment of normal or abnormal traffic data; (3) weight assignment, according to the importance of the key fields in the training data, the sparse key point interactive attention module will automatically assign different weights to these key fields, and the fields with higher weights represent a greater impact on the classification result, so the model will pay more attention to these fields; (4) feature extraction and fusion, according to the assigned weights, the sparse key point interactive attention module extracts features from the key fields, the extracted features can be numerical values, categories or high-dimensional vectors, etc., and fuses the extracted features to form a comprehensive feature vector as the key features of the training data. By extracting the key features of the training data through the sparse key point interactive attention module, an attention map is generated according to the key features of the training data, so that the sparse key point interactive attention module learns the features of the training data with the goal of minimizing the attention map loss (L ra ) as the target. Among them, the features of the training data learned by the sparse key point interactive attention module include key fields in the training data, and the key fields include parameter names, parameter value ranges, parameter combinations, etc.

[0082] In combination with the elastic adaptive deep auto-encoding module and the sparse key point interactive attention module, the features of the training data learned by the sparse key point interactive attention module are used to perform anomaly detection on the training data to obtain the detection result of the training data, i.e., the training data is normal traffic data or the training data is abnormal traffic data, and the model parameters of the traffic data anomaly detection model are adjusted according to the detection result of the training data.

[0083] By training the traffic data anomaly detection model according to all training data, the elastic adaptive deep auto-encoding module and the sparse key point interactive attention module learn the features of all training data, which can further realize comprehensive and accurate anomaly detection of traffic data and effectively avoid false positives and missed detections.

[0084] In an optional embodiment, the judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set comprises: judging whether the to-be-detected traffic data conforms to any normal traffic data rule in the normal traffic data rule set based on the normal traffic data rule set, and if yes, the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is not abnormal traffic data.

[0085] For example, considering that the normal traffic data rule set is generated in combination with features of all training data self-learned by the traffic data anomaly detection model, each normal traffic data rule in the normal traffic data rule set can be used for anomaly detection of traffic data, the judging whether the to-be-detected traffic data conforms to any normal traffic data rule in the normal traffic data rule set based on the normal traffic data rule set, if yes, the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is not abnormal traffic data, otherwise, the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is abnormal traffic data.

[0086] The embodiment of the application can quickly detect normal traffic data by judging that the to-be-detected traffic data is not abnormal traffic data when the to-be-detected traffic data conforms to any normal traffic data rule in the normal traffic data rule set, and is beneficial to improving the anomaly detection efficiency.

[0087] In an optional embodiment, after the judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the to-be-detected traffic data, the method further comprises: when the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is abnormal traffic data, intercepting the to-be-detected traffic data.

[0088] For example, when it is detected that the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is abnormal traffic data, the to-be-detected traffic data is intercepted, and when it is detected that the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is not abnormal traffic data, the to-be-detected traffic data is released.

[0089] The embodiment of the application can timely intercept abnormal traffic data by intercepting the to-be-detected traffic data when it is determined that the to-be-detected traffic data is abnormal traffic data, and is beneficial to ensuring network security.

[0090] In an optional embodiment, the preprocessing comprises data cleaning, data conversion, and data normalization processing.

[0091] For example, all normal traffic data is subjected to preprocessing such as data cleaning, data conversion, and data normalization processing to obtain all training data.

[0092] The data cleaning includes deleting abnormal values or replacing with reasonable preset values. The data conversion includes converting non-numeric data such as IP addresses (Internet Protocol Address), URLs and the like into numeric data. The data conversion method can adopt a hash function, one-hot encoding and the like. The data normalization processing can avoid different data having different dimensions and ranges, and the data normalization method can adopt a minimum-maximum normalization method, a Z-score standardization method and the like.

[0093] The embodiment of the application can effectively train the traffic data anomaly detection model according to all the training data by pre-processing all the normal traffic data through the pre-processing methods such as data cleaning, data conversion and data normalization processing, and guarantee that the traffic data anomaly detection model can self-learn the characteristics of all the training data.

[0094] Please refer to Figure 3 , Figure 3 A structure schematic diagram of a traffic data anomaly detection device provided by the second embodiment of the application is provided. The second embodiment of the application provides a traffic data anomaly detection device, which comprises: a data processing unit 201, configured to collect normal traffic data of algorithmically accessing websites, pre-process all the normal traffic data to obtain all the training data; a model establishing unit 202, configured to establish a traffic data anomaly detection model in combination with an elastic adaptive deep self-encoding module and a sparse key point interactive attention module; a feature learning unit 203, configured to train the traffic data anomaly detection model according to all the training data, so that the traffic data anomaly detection model self-learns the characteristics of all the training data; a rule generating unit 204, configured to generate a normal traffic data rule set in combination with the characteristics of all the training data self-learned by the traffic data anomaly detection model; and an anomaly detection unit 205, configured to judge whether to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set, to obtain a detection result of the to-be-detected traffic data.

[0095] In optional embodiments, the anomaly detection unit 205 is further configured to, before judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set to obtain the detection result of the to-be-detected traffic data, acquire traffic data of algorithmically accessing websites in real time as to-be-detected traffic data; judge whether the to-be-detected traffic data is unknown traffic data according to all the normal traffic data; and when the to-be-detected traffic data is unknown traffic data, record the to-be-detected traffic data.

[0096] In an optional embodiment, the anomaly detection unit 205 is further configured to, before determining whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and obtaining a detection result of the to-be-detected traffic data, when the number of all the recorded to-be-detected traffic data reaches a preset threshold, regarding all the recorded to-be-detected traffic data as new training data; updating the traffic data anomaly detection model according to all the new training data through the continuous scalable learning module, so that the traffic data anomaly detection model relearns the features of all the new training data; and updating the normal traffic data rule set in combination with the features of all the new training data relearned by the traffic data anomaly detection model.

[0097] In an optional embodiment, the training of the traffic data anomaly detection model according to all the training data and the self-learning of the features of all the training data by the traffic data anomaly detection model specifically include: for each training data, compressing the training data through an encoder to extract target features of the training data, reconstructing data according to the target features of the training data through a decoder to obtain reconstructed data, so that the elastic adaptive deep self-encoding module self-learns the features of the training data with the minimum reconstruction error as the target; wherein the elastic adaptive deep self-encoding module includes the encoder and the decoder; extracting key features of the training data through a sparse key point interactive attention module, generating an attention map according to the key features of the training data, so that the sparse key point interactive attention module self-learns the features of the training data with the minimum attention map loss as the target; and performing anomaly detection on the training data in combination with the features of the training data self-learned by the elastic adaptive deep self-encoding module and the sparse key point interactive attention module, to obtain a detection result of the training data, and adjusting model parameters of the traffic data anomaly detection model according to the detection result of the training data.

[0098] In an optional embodiment, the determination of whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and the obtaining of the detection result of the to-be-detected traffic data specifically include: determining whether the to-be-detected traffic data conforms to any normal traffic data rule in the normal traffic data rule set based on the normal traffic data rule set, and if yes, the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is not abnormal traffic data.

[0099] In an optional embodiment, the anomaly detection unit 205 is further configured to, after the determination of whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set and the obtaining of the detection result of the to-be-detected traffic data, when the detection result of the to-be-detected traffic data is that the to-be-detected traffic data is abnormal traffic data, intercepting the to-be-detected traffic data.

[0100] In an optional embodiment, the preprocessing includes data cleaning, data conversion, and data normalization processing.

[0101] The implementation process of the functions and roles of each module in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0102] Please refer to Figure 4 , Figure 4 A structural schematic diagram of an electronic device provided by the third embodiment of the present application. The third embodiment of the present application provides an electronic device 30, which comprises a processor 301, a memory 302, and a computer program stored in the memory 302 and configured to be executed by the processor 301; the memory 302 is coupled to the processor 301, and the processor 301 implements the flow data anomaly detection method as described in the first embodiment of the present application when executing the computer program, and can achieve the same beneficial effects.

[0103] Wherein, the processor 301 reads the computer program from the memory 302 through the bus 303 and executes the computer program, and can implement the method of any embodiment of the flow data anomaly detection method as described in the first embodiment of the present application.

[0104] The processor 301 can process digital signals and can include various computing structures. For example, a complex instruction set computer structure, a reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, the processor 301 can be a microprocessor.

[0105] The memory 302 can be used to store instructions executed by the processor 301 or data related to the execution process of the instructions. These instructions and / or data can include code for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 301 of the embodiments of the present disclosure can be used to execute the instructions in the memory 302 to implement the flow data anomaly detection method as described in the first embodiment of the present application. The memory 302 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memories well known to those skilled in the art.

[0106] The fourth embodiment of the present application provides a computer readable storage medium, which comprises a stored computer program; wherein, when the computer program runs, the device where the computer readable storage medium is located executes the flow data anomaly detection method as described in the first embodiment of the present application, and can achieve the same beneficial effects.

[0107] To sum up, the embodiment of the present application provides a traffic data anomaly detection method, device, equipment and storage medium, the traffic data anomaly detection method comprises: collecting normal traffic data of legal access website, pre-processing all normal traffic data to obtain all training data; combining an elastic adaptive deep self-encoding module and a sparse key point interactive attention module to establish a traffic data anomaly detection model; training the traffic data anomaly detection model according to all training data, so that the traffic data anomaly detection model self-learns the characteristics of all training data; combining the characteristics of all training data self-learned by the traffic data anomaly detection model to generate a normal traffic data rule set; judging whether the to-be-detected traffic data is abnormal traffic data based on the normal traffic data rule set to obtain a detection result of the to-be-detected traffic data. The embodiment of the present application combines the elastic adaptive deep self-encoding module and the sparse key point interactive attention module to establish the traffic data anomaly detection model, pre-processes all normal traffic data of legal access website as training data to train the traffic data anomaly detection model, so that the traffic data anomaly detection model self-learns the characteristics of all training data, generates the normal traffic data rule set in combination with the characteristics of all training data, and performs anomaly detection on the to-be-detected traffic data based on the normal traffic data rule set, which can comprehensively and accurately perform anomaly detection on the traffic data and effectively avoid the problems of false detection and missed detection.

[0108] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can also be implemented by other manners. The apparatus embodiments described above are only schematic, for example, the flow charts and block diagrams in the drawings show the possible implementation architectures, functions and operation of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flow charts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that noted in the drawings. For example, two consecutive blocks can actually be executed in parallel or in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flow charts, and the combination of blocks in the block diagrams and / or flow charts, can be implemented by a dedicated hardware-based system, or by a combination of special-purpose hardware and computer instructions.

[0109] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0110] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0111] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for detecting anomalies in traffic data, characterized in that, include: Collect normal traffic data of legitimate website visits, preprocess all the normal traffic data to obtain all training data; A traffic data anomaly detection model is established by combining an elastic adaptive deep autoencoder module and a sparse keypoint interactive attention module. The traffic data anomaly detection model is trained based on all the training data, enabling the model to learn the features of all the training data itself, specifically including: For each training data set, the training data is compressed by an encoder, the target features of the training data are extracted, and the data is reconstructed by a decoder based on the target features of the training data to obtain reconstructed data. This allows the elastic adaptive deep autoencoder module to learn the features of the training data with the goal of minimizing the reconstruction error. The elastic adaptive deep autoencoder module includes the encoder and the decoder. The sparse keypoint interactive attention module extracts key features from the training data and generates an attention map based on these key features, so that the sparse keypoint interactive attention module can self-learn the features of the training data with the goal of minimizing the attention map loss. Combining the features of the training data learned by the elastic adaptive deep autoencoder module and the sparse keypoint interactive attention module, anomaly detection is performed on the training data to obtain the detection results of the training data, and the model parameters of the traffic data anomaly detection model are adjusted according to the detection results of the training data. By combining the features of all the training data learned by the traffic data anomaly detection model, a normal traffic data rule set is generated; Based on the normal traffic data rule set, it is determined whether the traffic data to be detected is abnormal traffic data, and the detection result of the traffic data to be detected is obtained.

2. The method for detecting abnormal traffic data according to claim 1, characterized in that, Before determining whether the traffic data to be detected is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the traffic data to be detected, the method further includes: Real-time acquisition of traffic data accessing the website is used as the traffic data to be detected; Based on all the normal traffic data, determine whether the traffic data to be detected is unknown traffic data; When the traffic data to be detected is unknown, the traffic data to be detected is recorded.

3. The method for detecting abnormal traffic data according to claim 2, characterized in that, Before determining whether the traffic data to be detected is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the traffic data to be detected, the method further includes: When the number of all the recorded traffic data to be detected reaches a preset threshold, all the recorded traffic data to be detected will be used as new training data. The traffic data anomaly detection model is updated by the continuous scaling learning module based on all the new training data, so that the traffic data anomaly detection model can relearn the features of all the new training data. The normal traffic data rule set is updated by combining the features of all the new training data that the traffic data anomaly detection model has relearned.

4. The method for detecting abnormal traffic data according to claim 1, characterized in that, The step of determining whether the traffic data to be detected is abnormal traffic data based on the normal traffic data rule set, and obtaining the detection result of the traffic data to be detected, specifically includes: Based on the normal traffic data rule set, it is determined whether the traffic data to be detected conforms to any normal traffic data rule in the normal traffic data rule set. If so, the detection result of the traffic data to be detected is that the traffic data to be detected is not abnormal traffic data.

5. The method for detecting abnormal traffic data according to claim 1, characterized in that, After determining whether the traffic data to be detected is abnormal traffic data based on the normal traffic data rule set and obtaining the detection result of the traffic data to be detected, the method further includes: When the detection result of the traffic data to be detected is that the traffic data to be detected is abnormal, the traffic data to be detected is intercepted.

6. The method for detecting abnormal traffic data according to claim 1, characterized in that, The preprocessing includes data cleaning, data transformation, and data normalization.

7. A device for detecting abnormal traffic flow data, characterized in that, include: The data processing unit is used to collect normal traffic data of legitimate website visits, preprocess all the normal traffic data, and obtain all training data. The model building unit is used to combine the elastic adaptive deep autoencoder module and the sparse keypoint interactive attention module to build a traffic data anomaly detection model; A feature learning unit is used to train the traffic data anomaly detection model based on all the training data, enabling the traffic data anomaly detection model to self-learn the features of all the training data. Specifically, this includes: for each training data, compressing the training data using an encoder, extracting target features from the training data, and reconstructing the data using a decoder based on the target features to obtain reconstructed data, so that the elastic adaptive deep autoencoder module self-learns the features of the training data with the goal of minimizing the reconstruction error; wherein the elastic adaptive deep autoencoder module includes the encoder and the decoder; extracting key features of the training data using a sparse keypoint interactive attention module, generating an attention map based on the key features of the training data, so that the sparse keypoint interactive attention module self-learns the features of the training data with the goal of minimizing the attention map loss; combining the features of the training data self-learned by the elastic adaptive deep autoencoder module and the sparse keypoint interactive attention module, performing anomaly detection on the training data to obtain the detection result of the training data, and adjusting the model parameters of the traffic data anomaly detection model based on the detection result of the training data; The rule generation unit is used to combine the features of all the training data learned by the traffic data anomaly detection model to generate a set of rules for normal traffic data. An anomaly detection unit is used to determine whether the traffic data to be detected is abnormal traffic data based on the normal traffic data rule set, and to obtain the detection result of the traffic data to be detected.

8. An electronic device, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor; the memory is coupled to the processor, and the processor, when executing the computer program, implements the traffic data anomaly detection method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform the traffic data anomaly detection method according to any one of claims 1 to 6.

Images