Traffic data anomaly detection method based on uncertainty measurement

Abstract

The present invention belongs to the field of data processing technology, and discloses a flow data anomaly detection method based on uncertainty measurement. First, a flow data anomaly detection system is constructed, and the flow data anomaly detection system includes an automatic encoder, a reconstruction encoder, an output module, and an anomaly recognition module; then, the flow data anomaly detection system is trained using a training set constructed from normal flow data samples; and then, the flow data to be detected is input into the trained flow data anomaly detection system to identify the abnormal situation of the flow data to be detected. The present invention uses the uncertainty differences shown between the flow data distributions to directly judge the degree of flow data anomaly, and significantly improves the detection accuracy of abnormal flow.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, relates to data anomaly detection technology, and in particular to a flow data anomaly detection method based on uncertainty measurement. Background Art

[0002] Traffic data anomaly detection is widely used in many fields, such as traffic data anomaly detection, power data anomaly detection, hydraulic data anomaly detection, etc. These traffic data anomaly detection is one of the key technologies to ensure network security, power dispatching, water conservancy dispatching and other livelihood and economic issues. Traditional traffic data anomaly detection methods mainly include rule-based detection, machine learning methods, deep learning classification, etc. Rule-based detection methods rely on manually set rules and features, but they cannot cope with new or complex attack patterns. Machine learning methods mainly use statistical analysis and rely on modeling the statistical features of traffic data. Although they have certain effects, they cannot accurately capture the changing attack patterns. Deep learning classification methods mainly use supervised classification and are still limited to evolving abnormal traffic detection. In addition, abnormal data is always less than normal data. This data category imbalance problem further affects the effectiveness of these traditional methods in traffic data anomaly detection.

[0003] To address the above problems, anomaly detection methods based on semi-supervised learning have emerged. Unlike supervised detection methods, anomaly detection based on semi-supervised learning aims to fit normal data using models, reducing the reliance on abnormal data in the training set and eliminating the need for manual collection and labeling of abnormal traffic. During the detection phase, abnormal data usually exhibits a pattern that is completely different from normal data, resulting in significant deviations in the model, and the degree of abnormality is represented by a larger anomaly score value.

[0004] like Figure 1 As shown in Figure 1, existing semi-supervised traffic data anomaly detection methods mainly rely on a single reconstruction model, such as the AutoEncoder (AE) model, to construct normal traffic distribution and evaluate anomaly scores by testing the different reconstruction qualities and differences of the sample output in the reconstruction model. However, these methods ignore the impact of the same shortcut problem in the reconstruction model on abnormal data, resulting in the inability to effectively capture the significant reconstruction deviation of abnormal data, resulting in unsatisfactory detection results. Summary of the invention

[0005] The purpose of this invention is to design a novel traffic data anomaly detection method based on uncertainty measurement to address the limitations of semi-supervised traffic data anomaly detection. By focusing on the normal traffic data in the training set and modeling the uncertainty of traffic data propagation during link transmission, accurate and efficient anomaly detection is performed.

[0006] The idea of ​​the present invention is to use the autoencoder model to extract and refine the complex patterns and evidence of normal data, and introduce evidence learning to minimize the marginal likelihood function, guiding the uncertainty parameters to output the parameters that best fit the normal traffic data distribution. The degree of abnormality of traffic data is directly evaluated through the uncertainty probability of normal and abnormal traffic data in the detection model, accurately and efficiently monitoring the degree of abnormality of traffic data, while improving interpretability.

[0007] Based on the above invention ideas, the present invention proposes a flow data anomaly detection method based on uncertainty measurement, which specifically includes the following steps:

[0008] S1, build a traffic data anomaly detection system;

[0009] The flow data anomaly detection system includes an automatic encoder, a reconstruction encoder, an output module and an anomaly recognition module;

[0010] The automatic encoder is used to reconstruct the traffic data to obtain a reconstructed sample;

[0011] The reconstruction encoder is used to embed the reconstructed samples into the uncertainty space to obtain the uncertainty distribution;

[0012] The output module is used to obtain distribution parameters according to the uncertainty distribution;

[0013] The anomaly identification module is used to calculate an anomaly score based on the distribution parameters, and identify abnormal conditions of the flow data based on the anomaly score;

[0014] S2, training the traffic data anomaly detection system using a training set constructed from normal traffic data samples;

[0015] S3, inputting the traffic data to be detected into the trained traffic data anomaly detection system to identify the anomalies of the traffic data to be detected.

[0016] In the above step S1, the autoencoder is based on an Encoder-Decoder architecture, which includes an encoder and a decoder; the encoder is used to extract the intrinsic temporal pattern of the sample and extract the hidden layer feature z as a high-dimensional representation of the flow data; the decoder is used to obtain the reconstructed sample based on the high-dimensional representation. The encoder can use a convolutional neural network, and the decoder can use a transposed convolutional neural network.

[0017] The reconstruction encoder converts the reconstructed samples into an uncertainty space. The reconstruction encoder may use a convolutional neural network.

[0018] The output module uses a fully connected layer (FC) to obtain distribution parameters α, β, and v based on the uncertainty distribution obtained by the reconstruction encoder.

[0019] The anomaly identification module calculates the anomaly score according to the following formula:

[0020]

[0021] Here Var represents the variance function.

[0022] Then the anomaly score is compared with the set threshold. When the anomaly score is greater than the set threshold, the traffic data to be detected is determined to be abnormal traffic data, otherwise it is normal traffic data.

[0023] In the above step S2, in order to improve the training effect of the traffic data anomaly detection system and eliminate the possible traffic data noise, the normal traffic data is first preprocessed to generate an efficient and effective normal traffic data input form, thereby enhancing the pattern capture and anomaly assessment capabilities of the subsequent model. In a specific implementation, for each byte type of raw traffic data sample, the noisy byte segments (such as null values ​​and singular values) are filtered and cleaned to obtain the preprocessed normal traffic data.

[0024] Then, the training set is constructed using the preprocessed normal traffic data. The traffic data anomaly detection system is then trained in the following steps:

[0025] S21 inputs the normal traffic data samples in the training set into the automatic encoder to obtain the reconstructed samples, and extracts the output of the encoder as evidence; then obtains the reconstruction loss based on the normal traffic data samples and the reconstructed samples;

[0026] S22 embeds the reconstructed sample into the uncertainty space through the reconstruction encoder to obtain the uncertainty distribution; and obtains the distribution parameters through the output module; and then combines the evidence obtained in step S21 to obtain the uncertainty learning optimization target;

[0027] S23 determines the total loss according to the reconstruction loss and the uncertainty learning optimization target, and then optimizes the parameters of the flow data anomaly detection system according to the total loss;

[0028] Repeat the above steps S21-S23 until the traffic data anomaly detection system converges.

[0029] In the above step S21, the main purpose is to obtain the reconstructed sample and extract evidence. The normal traffic data sample x is input into the autoencoder, and the autoencoder is used to reconstruct the original input of the normal traffic data sample (that is, to obtain the reconstructed sample), automatically capture and model the intrinsic time series pattern of the normal traffic data sample, and the reconstructed sample is expressed as This step also extracts and refines the complex patterns and evidence of normal traffic data from the autoencoder to obtain a high-dimensional vector representation that can represent the traffic data. Specifically, the output hidden layer feature z is extracted from the hidden layer of the autoencoder (i.e., the encoder part) as a high-dimensional representation of the traffic data. This representation summarizes and refines the evidence of normal traffic data and provides a reliable input for subsequent uncertainty modeling.

[0030] Among them, the reconstruction loss function is defined as:

[0031]

[0032] Among them, ||·||1 represents the L1 regular term, x represents normal traffic data, Represents the reconstructed sample.

[0033] In the above step S22, the purpose is to model based on evidence. Using the uncertainty measurement method based on evidence learning, the extracted high-dimensional vector representation is modeled. The present invention directly puts the prior distribution on the likelihood function, trains the output module (here the fully connected layer is used) network output high-order evidence distribution hyperparameters, and models the pattern of benign samples by minimizing the uncertainty evidence distribution of normal traffic data samples.

[0034] The present invention introduces an additional reconstruction encoder after the autoencoder to import the reconstructed samples into the uncertainty space u. Then, evidence learning is introduced to guide the distribution modeling of the uncertainty space through refined evidence. Specifically, it is assumed that the extracted refined evidence z conforms to the Gaussian distribution commonly used in anomaly detection. It is assumed that the mean μ of the uncertainty space u is a Gaussian prior, and the deviation σ of u is assumed to be an inverse gamma prior:

[0035]

[0036] σ ( ~Γ -1 (α,β)

[0037] Here Γ(·) represents a gamma function; α, β, v represent the parameters of the evidence distribution.

[0038] The modeling goal here is to estimate a posterior q(μ,σ ( )=p(μ,σ ( |z), in order to approximate this posterior, it is decomposed into σ(μ,σ ( )=q(μ)σ(σ ( ), and is represented by normal inverse gamma distribution (Gaussian conjugate prior). The specific distribution expression is:

[0039]

[0040] Among them, Ω = (α, β, v) represents the output parameters of the fully connected layer derived based on the uncertainty space u.

[0041] Uncertainty learning is performed on the parameters of the constructed evidence distribution. According to Bayesian probability theory, "model evidence" or marginal likelihood is defined as the likelihood of an observation z given the evidence distribution parameter Ω, and is calculated by marginalizing over the likelihood parameter Ω:

[0042] In order to optimize the system's representation of normal traffic data, it is necessary to maximize the logarithm of the system's evidence. It is very difficult to calculate the distribution of uncertainty parameters based on the logarithm of evidence. When normal inverse gamma evidence is applied before the likelihood function, their analytical solution can be obtained, which guides the uncertainty parameters to output the parameters of the NIG distribution that best fits the normal traffic data distribution by minimizing its negative value. Through the arrangement, it can be seen that uncertainty learning maximizes the following optimization objectives:

[0043]

[0044] ω=2β(1+v).

[0045] In the above step S23, the total loss can be expressed as:

[0046] L 总 =L sme +λL NLL ;

[0047] Among them, L sme represents the reconstruction loss, L NLL represents the uncertainty learning optimization objective, and λ represents the weight factor.

[0048] According to the total loss, the parameters of the traffic data anomaly detection system (including the autoencoder, reconstruction encoder and fully connected layer) are optimized through the Adam optimization algorithm.

[0049] Repeat the above steps S21-S23 until the traffic data anomaly detection system converges.

[0050] Through the above evidence extraction and modeling, the traffic data anomaly detection system obtained is an effective traffic data uncertainty evaluator, which measures the degree of abnormality of the sample by the uncertainty of the output of the traffic data sample to be detected.

[0051] In step S3, anomaly scoring is performed, which is based on a well-known prior knowledge that there is usually a significant distribution difference between normal and abnormal traffic data. In step S2, sufficient evidence modeling is performed on normal samples, so that the constructed model has a small uncertainty prediction for normal samples, and on the contrary, abnormal samples outside the distribution show a large deviation, that is, a high uncertainty difference. Traditional methods often use the difference between the input and output of the trained autoencoder to indicate anomalies. However, due to the same shortcut characteristics of the autoencoder, abnormal samples will also be well reconstructed, which often produces suboptimal anomaly evaluation. The present invention proposes to directly evaluate the uncertainty difference between positive and abnormal samples to judge the degree of sample abnormality and avoid suboptimal secondary calculations. Specifically, the present invention calculates the variance of the mean of the evidence parameter distribution of the data, that is, the uncertainty estimate to output a specific anomaly score. The anomaly score quantifies the uncertainty judgment of the model on the data. A higher anomaly score indicates a higher probability of anomaly. When the score is higher than the set threshold, it is detected as anomaly, otherwise it is normal. Therefore, our anomaly score is defined as follows:

[0052]

[0053] So far, a traffic data anomaly detection method based on uncertainty measurement has been realized.

[0054] Therefore, the present invention mainly utilizes the autoencoder model (AutoEncoder, AE) to extract and refine the complex patterns and evidence of normal traffic data, and on this basis uses the evidence learning (Evidential Learning) framework to introduce uncertainty measurement to minimize the model uncertainty of normal traffic data. Finally, the uncertainty difference between normal and abnormal traffic data in the detection model is used to directly evaluate the abnormality of the traffic data. While achieving accurate and efficient detection, the demand for abnormal traffic data in the algorithm training stage is eliminated, reducing labor costs.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1) The present invention is based on a semi-supervised learning method, using a trained autoencoder model to extract and refine the complex temporal patterns and evidence of normal traffic data, providing an effective and efficient expression for subsequent semi-supervised modeling;

[0057] 2) The present invention cleverly introduces evidence learning into the semi-supervised traffic data anomaly detection model, and guides the uncertainty parameters to output parameters that are most suitable for the normal traffic data distribution by minimizing the marginal likelihood function; the present invention provides a new uncertainty perspective for semi-supervised modeling and has stronger interpretability.

[0058] 3) The present invention proposes to use the uncertainty difference between the distribution of normal and abnormal samples to directly judge the degree of sample abnormality, thereby avoiding the suboptimal evaluation based on reconstruction differences in traditional semi-supervised traffic data anomaly detection methods and significantly improving the detection accuracy of abnormal traffic.

[0059] 4) The present invention makes full use of the advantages of semi-supervised learning and can detect complex and changeable abnormal traffic data in various application scenarios based on existing normal traffic data.

[0060] 5) The present invention uses uncertainty measurement methods to capture the patterns of normal traffic data, and combines refined high-dimensional features and novel uncertainty perspectives to judge the degree of conformity between the traffic data pattern to be detected and the system model. It not only achieves accurate and efficient anomaly detection, but also provides strong algorithm interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A schematic diagram of semi-supervised training and anomaly scoring for traditional semi-supervised traffic data anomaly detection methods, showing the implementation of common detection tasks;

[0062] Figure 2 A schematic diagram of the principle of a flow data anomaly detection method based on uncertainty measurement provided by an embodiment of the present invention;

[0063] Figure 3 A flow chart of a method for detecting anomalies in flow data based on uncertainty measurement provided in an embodiment of the present invention.

[0064] Explanation of terms

[0065] Semi-supervised learning: A machine learning method that aims to train a model using partially labeled data. In the present invention, semi-supervised traffic data anomaly detection refers to training a model using only normal traffic data and detecting anomalies through distribution deviations between normal and abnormal data.

[0066] Autoencoder: used for reconstruction of input data. Its implementation method is referenced in [Yu Zheng, Xinglin Lian, Zhangxuan Dang, Chunlei Peng, Chao Yang, and Jianfeng Ma. ASemi-Supervised Anomaly Network Traffic Detection Framework via Multimodal Traffic Information Fusion. In CIKM, 2023]. DETAILED DESCRIPTION

[0067] The present invention is further described below in conjunction with the accompanying drawings.

[0068] Example

[0069] This embodiment uses two different real data sets (USTC-TFC2016 data set and DataCon2020 data set) to explain in detail the traffic data anomaly detection method based on uncertainty measurement provided by the present invention. The source of the USTC-TFC2016 data set can be found in the reference [Wei Wang, Ming Zhu, Xuewen Zeng, Xiaozhou Ye, and Yiqiang Sheng. 2017. Malware traffic classification using convolutional neural network for representation learning. In ICOIN, 2017.], and the source of the DataCon2020 data set can be found in the reference [DataCon: open dataset DataCon2020-encryptedmalicious traffic datasetdirection open dataset.

[0070] https: / / datacon.qianxin.com / opendata.】.

[0071] As attached Figure 2 As shown, the traffic data anomaly detection method based on uncertainty measurement provided by this embodiment mainly includes two parts: a training phase and an inference phase. Among them, the training phase mainly models normal traffic data samples from the perspective of uncertainty. First, each sample is cleaned to remove irrelevant or noise data. Then the cleaned samples are sent to the evidence extraction module for reconstruction training. In this process, high-dimensional representations are extracted through the autoencoder model, thereby providing effective input for subsequent uncertainty modeling. Based on the output of the autoencoder, the reconstructed samples are converted into uncertainty space using an encoder, and the distribution parameters of evidence learning are extracted through a fully connected neural network layer. In this process, prior information directly acts on the likelihood function, and the evidence distribution parameters of the model output are optimized by training the neural network. Finally, in the inference phase, the present invention evaluates the degree of anomaly by measuring the variance of the uncertainty parameters of the data in the model. Specifically, normal traffic data samples show lower uncertainty by minimizing uncertainty in the training phase; while abnormal samples show higher uncertainty. The final detection result is obtained by comparing the uncertainty of the sample with a preset threshold. This threshold is usually adjusted according to the distribution of different data sets to achieve the best detection effect.

[0072] Based on the above analysis, the flow data anomaly detection method based on uncertainty measurement provided in this embodiment is as follows: Figure 3 As shown, it includes the following steps:

[0073] S1, build a traffic data anomaly detection system.

[0074] The traffic data anomaly detection system includes an automatic encoder, a reconstruction encoder, an output module and an anomaly recognition module.

[0075] The automatic encoder is used to reconstruct the traffic data x and obtain the reconstructed sample The autoencoder consists of an encoder and a decoder. The encoder uses a convolutional neural network (CNN); the decoder uses a transposed convolutional neural network, that is, the convolutional layer in the convolutional neural network is replaced by a transposed convolutional layer.

[0076] The reconstruction encoder is used to embed the reconstructed samples into the uncertainty space to obtain the uncertainty distribution u. The reconstruction encoder uses a convolutional neural network.

[0077] The output module is used to obtain the distribution parameters α, β, v according to the uncertainty distribution. The output module uses a fully connected layer.

[0078] The anomaly identification module is used to calculate the anomaly score based on the distribution parameters and identify the abnormal situation of the traffic data based on the anomaly score. The anomaly identification module calculates the anomaly score according to the following formula:

[0079]

[0080] Then the anomaly score is compared with the set threshold. When the anomaly score is greater than the set threshold, the traffic data to be detected is determined to be abnormal traffic data, otherwise it is normal traffic data.

[0081] S2, trains the traffic data anomaly detection system using a training set constructed from normal traffic data samples.

[0082] This embodiment includes the following steps:

[0083] S21 inputs the normal traffic data samples in the training set into the automatic encoder to obtain the reconstructed samples, and extracts the output of the encoder as evidence; then the reconstruction loss is obtained based on the normal traffic data samples and the reconstructed samples.

[0084] In this embodiment, for the above two data sets, the mutable fields in the data packets are processed, such as the address fields (such as MAC, IP addresses) in the data link layer and the network layer, and these fields without fixed patterns are removed by the "wipe zero" operation to eliminate possible data noise. A communication link consists of multiple basic network flows, and each network flow consists of multiple data packets. First, check and remove duplicate network flow samples, then extract the first 6 data packets for each network flow, and perform a unified "Reshape" operation on them. Finally, these data packets are spliced ​​according to the channel dimension to form a multi-channel data packet representation to provide effective input data for subsequent network modeling.

[0085] The data is represented as follows:

[0086]

[0087] Among them, a traffic data sample consists of multiple data packets. Represents the tth X packet images, {} represents the channel-by-channel splicing operation, and x represents the traffic data samples in the form of multi-channel packets.

[0088] The training set is constructed using the preprocessed normal traffic data samples, and the test set is constructed using the preprocessed normal and abnormal traffic data samples at the same time.

[0089] The normal traffic data samples in the training set are input into the autoencoder, and the original input of the normal traffic data samples is reconstructed through semi-supervised training to obtain the reconstructed samples, which are expressed as At the same time, the output z is extracted from the hidden layer of the autoencoder (i.e., the encoder part) as a high-dimensional representation of the flow data, that is, as evidence.

[0090] Among them, the reconstruction loss function is defined as:

[0091]

[0092] Among them, ||·||1 represents the L1 regular term, x represents normal traffic data, Represents the reconstructed sample.

[0093] S22 embeds the reconstructed samples into the uncertainty space through the reconstruction encoder to obtain the uncertainty distribution; and obtains the distribution parameters through the output module; and then combines the evidence obtained in step S21 to obtain the uncertainty learning optimization target.

[0094] In this step, the sample will be reconstructed By reconstructing the encoder and embedding it into the uncertainty space, we get the uncertainty distribution u. Then we input u into the output module to get the distribution parameters α, β, and v.

[0095] Then, uncertainty learning is performed in combination with the evidence z obtained in step S21 to obtain the uncertainty learning optimization target, which is expressed as follows:

[0096]

[0097] ω=2β(1+v);

[0098] Wherein, u represents the uncertainty distribution obtained by the reconstruction encoder, and z represents the evidence obtained by the encoder. In this embodiment, the log function takes e as the base.

[0099] S23 determines the total loss based on the reconstruction loss and the uncertainty learning optimization target, and then optimizes the parameters of the traffic data anomaly detection system based on the total loss.

[0100] Therefore, the total loss in this step can be expressed as:

[0101] L 总 =L sme +λL NLL ;

[0102] Among them, L smw represents the reconstruction loss, L NLL represents the uncertainty learning optimization objective, and λ represents the weight factor (for the USTC-TFC2016 dataset, λ=1; for the DataCon2020 dataset, λ=0.01).

[0103] According to the total loss, the parameters of the traffic data anomaly detection system (including the autoencoder, reconstruction encoder and fully connected layer) are optimized through the Adam optimization algorithm.

[0104] Repeat the above steps S21-S23 until the traffic data anomaly detection system converges.

[0105] In this embodiment, when the number of iterations reaches the set threshold, the traffic data anomaly detection system is considered to have converged. During the training process, an early stopping strategy can be used to judge by the test evaluation indicators on the test set. When each indicator no longer improves, the training can be terminated in advance.

[0106] S3, inputting the traffic data to be detected into the trained traffic data anomaly detection system to identify the anomalies of the traffic data to be detected.

[0107] In this embodiment, the flow data samples in the test set are input into the trained flow data anomaly detection system to identify the anomalies of the flow data to be detected.

[0108] The present invention adopts three common indicators to evaluate the effect of the traffic data anomaly detection method: the area under the receiver operating characteristic curve (ROC Curve) (AUC), accuracy (ACC) and F1 accuracy (F1 Score). The AUC value ranges from 0 to 1, and the higher the value, the better the classification ability of the model. ACC indicates the proportion of samples correctly predicted by the model to the total number of samples. F1 accuracy is the harmonic mean of precision and recall, which is used to measure the balance between accuracy and completeness of the model.

[0109] At the same time, the traffic data anomaly detection method (UiD) based on uncertainty measurement provided by the present invention is compared with 5 different baseline models (MMR, GANomaly, ARCADE, MFAD), and the uncertainty modeling and anomaly scoring (UiD-AE) are removed as ablation for comparison. The prediction results are shown in Table 1.

[0110] Table 1: Results of abnormal traffic data detection on two datasets

[0111]

[0112] The remaining methods in the table are described as follows:

[0113] MMR: A method based on multi-scale feature reconstruction, which uses the multi-scale information of the data to judge the degree of anomaly. Its implementation process can be found in the literature [Z. Zhang, Z. Zhao, X. Zhang, C. Sun, X. Chen, Industrial anomaly detection with domain shift: A real-world dataset and masked multi-scale reconstruction, Computers in Industry, 2023.]

[0114] GANomaly: An autoencoder architecture based on Encoder-Decoder-Encoder. It proposes to use the reconstruction difference of the hidden layer to judge anomalies, and introduces an adversarial learning training strategy to enhance the reconstruction method of output quality. Its implementation process can be found in the literature [Samet Akcay, Amir Atapour-Abarghouei, and Toby PBreckon. Ganomaly: Semi-supervised anomaly detection via adversarial training. In ACCV, 2019.]

[0115] ARCADE: An Encoder-Decoder autoencoder architecture based on adversarial training, which mainly introduces the WGAN-GP optimization strategy. This strategy is to solve the problem that WassersteinGAN does not work well in practical applications, especially the limitation of Lipschitz continuity conditions. This method replaces weight clipping with gradient penalty, making the weight distribution more uniform and adversarial training more stable. For its implementation process, please refer to the reference [Willian T Lunardi,Martin Andreoni Lopez,and Jean-Pierre Giacalone.Arcade:Adversarially regularized convolutional autoencoder for network anomaly detection.IEEE Transactions on Network and Service Management,2022.]

[0116] MFAD: This model designs a contrastive learning module to distinguish whether the query event has occurred before. Its implementation process reference [Yu Zheng, Xinglin Lian, Zhangxuan Dang, Chunlei Peng, Chao Yang, and Jianfeng Ma. A Semi-Supervised Anomaly Network Traffic Detection Framework via Multimodal Traffic Information Fusion. In CIKM, 2023.]

[0117] From the experimental results in Table 1, it can be seen that the method of the present invention is superior to the index values ​​of other methods in all three index values, and the model has achieved the best performance improvement on all data sets. Compared with the comparative experiments, the method based on uncertainty metric modeling and anomaly scoring in the present invention avoids the suboptimal limitations of the reconstruction method and achieves excellent detection from a novel uncertainty perspective.

[0118] By comparing the experimental results of the variant UiD-AE of the method of the present invention with other methods, it can be seen that the basic reconstruction framework of the present invention has significant advantages. The multi-channel traffic data sample composition can effectively remove the noise field in the data, and can achieve optimal performance without additional adversarial training. By comparing the results of the present invention and other methods, it can be seen that the method of the present invention successfully captures the complex time series pattern of normal traffic data through semi-supervised modeling based on the uncertainty paradigm. The inference stage outputs the anomaly score by directly calculating the uncertainty distribution of the sample distribution, avoiding the common suboptimal judgment method based on reconstruction differences, thereby effectively improving the accuracy of anomaly detection.

[0119] At the same time, the algorithm efficiency of the method of the present invention for executing one inference is also calculated. MACs (Multiply-Accumulate Operations) refers to the number of multiplication-accumulation operations, which measures the total number of multiplication and addition operations in a neural network. It is an important indicator of computational complexity, especially in evaluating the execution efficiency of the model on hardware. The execution efficiency of the traffic data anomaly detection method based on uncertainty measurement of the present invention and other comparative methods are shown in Table 2.

[0120] Table 2: Inference efficiency comparison

[0121] Model MMR GANomaly ARCADE MFAD UiD MACs(G) 12.7 0.98 0.82 0.99 0.25

[0122] It can be seen from Table 2 that the traffic data anomaly detection system provided by the present invention achieves the highest performance in terms of algorithm efficiency.

[0123] In summary, the present invention uses a trained autoencoder model to extract and refine the complex time series patterns and evidence of normal traffic data, providing an effective and efficient expression for subsequent semi-supervised modeling. At the same time, the evidence learning method introduced by the present invention directly places the prior in the likelihood function, and by training the neural network to output the hyperparameters of the high-order evidence distribution, it can efficiently learn the representation of cognitive and random uncertainty without sampling. This is an effective and efficient way to measure uncertainty. Finally, the calculation of the anomaly score is directly based on the uncertainty of the data, which not only achieves extremely high detection accuracy, but also has excellent computational efficiency.

Claims

1. A method for detecting anomalies in flow data based on uncertainty measurement, characterized in that: The following steps are involved: S1, build a traffic data anomaly detection system; The flow data anomaly detection system includes an automatic encoder, a reconstruction encoder, an output module and an anomaly recognition module; The automatic encoder is used to reconstruct the traffic data to obtain a reconstructed sample; The reconstruction encoder is used to embed the reconstructed samples into the uncertainty space to obtain the uncertainty distribution; The output module is used to obtain distribution parameters according to the uncertainty distribution; The anomaly identification module is used to calculate an anomaly score based on the distribution parameter, and identify the abnormal situation of the flow data based on the anomaly score; S2, training the traffic data anomaly detection system using a training set constructed from normal traffic data samples; S3, inputting the traffic data to be detected into the trained traffic data anomaly detection system to identify the anomalies of the traffic data to be detected.

2. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 1, characterized in that: The automatic encoder includes an encoder and a decoder; the encoder is used to extract hidden layer features as a high-dimensional representation of traffic data; and the decoder is used to obtain reconstructed samples based on the high-dimensional representation.

3. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 2 is characterized in that: The encoder uses a convolutional neural network, the decoder uses a transposed convolutional neural network, and the reconstruction encoder uses a convolutional neural network.

4. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 1, characterized in that: The output module uses a fully connected layer.

5. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 1, characterized in that: The anomaly identification module calculates the anomaly score according to the following formula: Among them, α, β, and v represent the distribution parameters obtained by the output module; Then the anomaly score is compared with the set threshold. When the anomaly score is greater than the set threshold, the traffic data to be detected is determined to be abnormal traffic data, otherwise it is normal traffic data.

6. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 1, characterized in that: Step S2 includes the following sub-steps: S21 inputs the normal traffic data samples in the training set into the automatic encoder to obtain the reconstructed samples, and extracts the output of the encoder as evidence; then obtains the reconstruction loss based on the normal traffic data samples and the reconstructed samples; S22 embeds the reconstructed sample into the uncertainty space through the reconstruction encoder to obtain the uncertainty distribution; and obtains the distribution parameters through the output module; and then combines the evidence obtained in step S21 to obtain the uncertainty learning optimization target; S23 determines the total loss according to the reconstruction loss and the uncertainty learning optimization target, and then optimizes the parameters of the flow data anomaly detection system according to the total loss; Repeat the above steps S21-S23 until the traffic data anomaly detection system converges.

7. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 6, characterized in that: In step S21, the reconstruction loss function is defined as: Among them, ||·||1 represents the L1 regular term, x represents normal traffic data, Represents the reconstructed sample.

8. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 6, characterized in that: In step S22, the uncertainty learning optimization objective is expressed as follows: ω=2β(1+v); Among them, u represents the uncertainty distribution obtained by the reconstruction encoder, and z represents the evidence obtained by the encoder.

9. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 6, characterized in that: In step S23, the total loss is expressed as: THE 总 =L sme +λL NLL ; Among them, L sme represents the reconstruction loss, L MLL represents the uncertainty learning optimization objective, and λ represents the weight factor.

10. The method for detecting anomalies in flow data based on uncertainty measurement according to claim 6, characterized in that: According to the total loss, the parameters of the traffic data anomaly detection system are optimized through the Adam optimization algorithm.

Images