Network traffic anomaly detection method, system and equipment based on deep learning

Abstract

The invention belongs to the technical field of network security detection, and provides a network traffic anomaly detection method, system and device based on deep learning. The method comprises the steps of original network traffic data acquisition and multi-source fusion, data preprocessing, network traffic feature engineering, small sample adaptive learning, network traffic anomaly detection, data distribution drift adaptation and incremental learning, and model iterative optimization and feedback. The system realizes rapid model adaptation and high-value sample screening in a small sample environment, greatly reduces dependence on annotated data, effectively deals with data distribution drift in combination with an incremental learning technology, improves long-term applicability of the model in a dynamic network environment, introduces causal reasoning into an anomaly detection process, and improves the accuracy of anomaly detection. Interpretable detection results and anti-fact analysis are provided, continuous self-iteration of system performance is achieved through closed-loop evaluation and a parameter optimization mechanism, and the reliability of the detection results and the operation and maintenance response efficiency are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of network security detection technology, and in particular to methods, systems and devices for detecting network traffic anomalies based on deep learning. Background Technology

[0002] Network traffic anomaly detection is one of the core technologies for ensuring network security. It analyzes the characteristics and patterns of traffic by collecting TCP and UDP packets and multi-source logs in network links, and identifies abnormal traffic that deviates from the normal pattern. This allows for early warning or blocking of malicious attacks, avoiding risks such as network paralysis and data leakage. As network scale expands and attack methods evolve, traffic data is characterized by massive volume, multiple sources, and strong time sequence, which places higher demands on the real-time performance, accuracy, and small sample adaptability of anomaly detection.

[0003] Traditional network traffic anomaly detection techniques typically follow a basic process of data acquisition, preprocessing, feature extraction, and detection: TCP and UDP packets are captured using conventional tools, the data is simply denoised and completed, statistical features are manually extracted, and then traditional machine learning models or basic deep learning models are used to determine anomalies.

[0004] However, existing technologies still have some shortcomings. For example, they have weak adaptability to small sample scenarios and lack anomaly attribution. Traditional models require a large amount of labeled data to achieve effective training, while new and unknown anomalies in the network often only have a small number of samples, making it impossible for models to quickly adapt to such anomalies. At the same time, labeling all samples requires a large amount of manual labor, which limits their practicality. Traditional models are mostly closed black box systems, and the detection results can only distinguish between normal and abnormal categories without exploring the root cause of the anomaly. The operation and maintenance team cannot quickly locate and deal with the problem based on the results, resulting in a lack of traceable root cause support for subsequent defense decisions and low response efficiency. Summary of the Invention

[0005] The purpose of this application is to propose a network traffic anomaly detection method, system, and device based on deep learning, so as to solve the technical problems of weak adaptability to small sample scenarios and lack of anomaly attribution mentioned in the background art.

[0006] To address the aforementioned technical problems, this application provides a deep learning-based method for detecting network traffic anomalies, employing the following technical solution: A deep learning-based method for detecting network traffic anomalies includes the following steps: Raw network traffic data is collected and combined with NetFlow logs and application layer logs. Multi-source data is fused based on session quintuples and timestamps to generate a session-level multi-source fused dataset. The fused dataset is preprocessed, including noise filtering based on traffic type weights, missing value imputation using an improved KNN algorithm, and SMOTE anomaly sample synthesis with time-series gradient enhancement, resulting in a balanced dataset. Based on the balanced dataset, statistical, temporal, and semantic features are extracted. High-discrimination and stable features are selected using improved mutual information entropy to form an optimized feature set. Meta-parameters are initialized using an improved MAML algorithm, and high-value samples are selected and labeled using an improved DQN algorithm to achieve small-sample adaptive learning. A classification model is trained using the optimized feature set to classify traffic as normal, known anomalies, and unknown anomalies, and anomaly attribution explanations are generated through causal reasoning. Traffic distribution drift is monitored, and model parameters are updated using an improved incremental SVM algorithm to adapt to changes in data distribution. A multi-dimensional comprehensive evaluation function is constructed, parameters are optimized based on performance bottlenecks, and an operation and maintenance feedback report is generated to achieve closed-loop optimization.

[0007] Preferably, in the raw network traffic data collection, the PFRING framework is used to capture raw TCP and UDP data packets in real time, and the multi-source data association and fusion includes session segmentation, log association, and invalid data removal.

[0008] Preferably, the noise filtering incorporates flow type weights to improve the 3σ principle; the missing value imputation uses a time decay factor to improve the KNN algorithm, with continuous fields filled with the mean and discrete fields filled with the mode; the abnormal sample synthesis preserves the abnormal temporal features through temporal gradients and controls the cosine similarity between the synthesized sample and the real abnormality.

[0009] Preferably, the statistical features are normalized by Z-score to form a 16-dimensional vector; the temporal features are extracted through a 3-layer LSTM network with 128 hidden units, outputting a 64-dimensional vector; and the semantic features are extracted based on a Word2Vec and protocol weight-enhanced multi-head attention mechanism.

[0010] Preferably, the MAML algorithm introduces anomaly type weights, with unknown anomalies having a greater weight than known anomalies, optimizing meta-parameters to fit small samples; the DQN uses mutual information, temporal stability, and prediction uncertainty as state inputs, and balances labeling cost and accuracy through a reward function.

[0011] Preferably, the classification model outputs three probabilities based on the Softmax classifier, sets an unknown anomaly detection threshold to determine unknown anomalies; the causal inference uses the DoWhy framework to calculate the total effect and individual treatment effect, and generates key causal factors and counterfactual explanations.

[0012] Preferably, the flow distribution drift monitoring uses a 24-hour sliding window to calculate the average KL divergence and sets a drift threshold to determine the drift status; the incremental learning introduces anomaly type priority and updates the SVM support vector coefficients; the comprehensive evaluation function includes accuracy, false alarm rate, annotation cost and drift fitness rate, and dynamically adjusts the model parameters according to the evaluation results.

[0013] To address the aforementioned technical issues, this application also provides a deep learning-based network traffic anomaly detection system, comprising: a data acquisition module for real-time capture of PCAP packets, NetFlow logs, and application layer proxy logs to obtain raw network data; a fusion module for generating a session-level multi-source fusion dataset from the raw network data using session quintuples and timestamp association rules; a preprocessing module for noise filtering, missing value imputation, and anomaly sample synthesis on the fusion dataset to output a balanced dataset; a feature engineering module for extracting statistical, temporal, and semantic features from the balanced dataset, and filtering high-discrimination features through improved mutual information entropy to form an optimized feature set; a few-shot learning module for initializing meta-parameters based on an improved MAML algorithm, and combining an improved DQN algorithm to filter high-value samples for labeling and optimization; a classification detection module for training a classification model using the optimized feature set and outputting classification results for normal, known anomalies, and unknown anomalies; an adaptive optimization module for monitoring traffic distribution drift and dynamically updating model parameters through incremental SVM; and an explanation and feedback module for generating anomaly attribution explanations based on causal reasoning, constructing a comprehensive evaluation function to iteratively optimize model parameters, and generating a feedback report.

[0014] Preferably, the data acquisition module uses the PFRING framework to capture raw data packets; NetFlow logs are parsed using the IPFIX protocol; application layer proxy logs are obtained through a RESTful API; the fusion module generates a unique session ID by segmenting the data using PCAP based on source IP + destination IP + source port + destination port + transport protocol + timestamp, and supplements the session duration and byte rate of NetFlow logs and the URL and DNS domain name of application layer logs by matching the session ID + timestamp, thus eliminating data that fails to be associated; the preprocessing module performs noise filtering based on traffic type weights; missing value imputation uses an improved KNN algorithm; outlier sample synthesis uses temporal gradient enhancement SMOTE; in the feature engineering module, statistical features are standardized by Z-score to form a 16-dimensional vector; temporal features are extracted using LSTM and output as a 64-dimensional temporal vector; semantic features are combined with the protocol. The weight matrix enhances the weights of high-risk protocols and outputs a 32-dimensional vector; the improved mutual information entropy filtering outputs a 48-dimensional optimized feature set; in the few-shot learning module, the improved MAML algorithm optimizes meta-parameters through gradient descent; the improved DQN algorithm filters samples based on the reward function to obtain labeled samples with high information value and unlabeled samples; the classification and detection module uses a Softmax classifier for classification, combines DoWhy to generate a causal explanation report and performs branch processing, and sets an unknown anomaly detection threshold to determine unknown anomalies; the adaptive optimization module monitors distribution drift through KL divergence; incremental SVM updates parameters with a sliding window, and the parameters are fed back to the feature engineering module and the classification and detection module; in the explanation feedback module, causal inference generates an attribution report based on counterfactual analysis; the comprehensive evaluation function integrates accuracy, false positive rate, and comprehensive evaluation value, and the optimized parameters are fed back to the few-shot learning module and the adaptive optimization module.

[0015] To address the aforementioned technical problems, this application also provides a computer device, including a memory, a processor, and a network interface. The memory, processor, and network interface are interconnected via a system bus. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the method described above.

[0016] The beneficial effects of this invention are as follows: This application provides a deep learning-based network traffic anomaly detection method. It achieves accurate correlation and fusion of multi-source data through session quintuples and timestamps, and introduces an improved mutual information entropy to select a high-discrimination and stable 48-dimensional optimized feature set from statistical, temporal, and semantic features, fundamentally improving the feature representation quality. By improving the MAML algorithm to initialize meta-parameters, the model can adapt to unknown anomalies with only a small number of labeled samples. Combined with an improved DQN algorithm to select high-value samples and reduce labeling costs, and finally generating anomaly attribution explanations through causal inference, this comprehensively solves the problems of weak small-sample adaptation, high labeling costs, and lack of attribution mechanisms, forming a complete optimization chain from data foundation and model adaptation to result interpretation. This method embeds a causal reasoning engine into the detection process, using the DoWhy framework to quantitatively calculate the total effect and individual treatment effect of features on abnormal results. It not only outputs what the anomaly is, but also generates why it is a key causal factor and counterfactual explanations for what would have happened if it hadn't occurred. This mechanism transforms the detection model from an incomprehensible black box into a traceable and verifiable white box, providing operations personnel with solid decision-making basis and greatly improving the accuracy and efficiency of anomaly handling.

[0017] This method uses KL divergence to monitor data distribution drift in real time and drives an improved incremental SVM algorithm to dynamically update model parameters, enabling the system to continuously self-optimize in dynamically changing network environments. Combined with a constructed multi-dimensional comprehensive evaluation function and closed-loop feedback mechanism, the system can automatically identify performance bottlenecks and iteratively optimize key parameters, thereby ensuring long-term stable detection accuracy and controlling performance degradation in drift scenarios to within 5%, significantly improving the system's long-term operational value and practical feasibility. Attached Figure Description

[0018] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of Embodiment 1 of this application; Figure 2 This is a system architecture diagram of Embodiment 2 of this application; Figure 3 This is a principle block diagram of Embodiment 3 of this application. Detailed Implementation

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings of this application are intended to cover non-exclusive inclusion.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0023] Example 1 like Figure 1 As shown, the deep learning-based network traffic anomaly detection method includes steps 1 to 7, wherein: Step 1: Raw network traffic data collection and multi-source fusion: Collect raw network traffic data, combine NetFlow logs and application layer logs, and perform multi-source data association and fusion based on session quintuples and timestamps to generate session-level multi-source fusion datasets.

[0024] Furthermore, in the raw network traffic data collection, the PFRING framework is used to capture raw TCP and UDP packets in real time, and multi-source data association and fusion includes session segmentation, log association, and invalid data removal.

[0025] Step 1.1: Raw packet capture.

[0026] PFRING (High-Performance Packet Capture Framework) is used to capture raw TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) packets in network links in real time. The sampling rate is 100% with no packet loss, the capture latency is ≤1ms, and the packets are stored in PCAP format. This results in ≥1 million PCAP format packets per hour with a packet loss rate of ≤0.1%. The packets cover stable TCP business traffic and real-time UDP interactive traffic.

[0027] Step 1.2: Multi-source data association and fusion.

[0028] The session 5-tuple includes source IP, destination IP, source port, destination port, and transport protocol, with a timestamp error ≤ 1 second. Using session 5-tuple + timestamp association rules, a three-step fusion is performed on PCAP packets, router and switch NetFlow logs, and application layer agent logs. The three-step fusion method includes: Session Segmentation: Segment PCAP data packets by source IP + destination IP + source port + destination port + transport protocol + timestamp to generate a unique session ID; Log Association: Match session ID + timestamp to supplement the corresponding session with the session duration and byte rate of NetFlow logs, and the URL and DNS domain name of application layer logs; Invalid Data Removal: Remove data that fails to associate.

[0029] The fusion process yields a session-level multi-source fusion dataset containing 32 original fields, such as packet length, transmission rate, and URL.

[0030] Step 2: Data preprocessing: The fused dataset is preprocessed, including noise filtering based on traffic type weights, missing value filling of improved KNN, and synthesis of SMOTE outlier samples with time-series gradient enhancement, to obtain a balanced dataset.

[0031] Furthermore, noise filtering incorporates traffic type weights to improve the 3σ principle, with TCP weighting at 0.8 and UDP weighting at 0.5; missing value imputation uses a time decay factor to improve the KNN algorithm, filling continuous fields with the mean and discrete fields with the mode; and anomaly sample synthesis preserves the anomaly temporal characteristics through temporal gradients, controlling the cosine similarity between the synthesized sample and the real anomaly.

[0032] Step 2.1: Noise Filtering: To address the non-strict normal distribution of network traffic, a traffic type weight w is introduced. t An improved 3σ principle algorithm is used to identify and filter noise in the fused dataset, outputting a denoised dataset.

[0033] Formula 1: |x-μ t |>3σ t ·w t Where x is the data packet field value; μ t σ represents the average value of a certain type of traffic field. t The standard deviation of this type of flow field; w t For traffic type weights, w t =0.8, for UDP, w t =0.5. Calculate the μ value for each field, categorized by TCP and UDP. t and σ t Substitute into Formula 1. If the formula is satisfied, the data is identified as noise, and the noise data is filtered out.

[0034] Step 2.2: Missing value imputation: Use the improved KNN (K-Nearest Neighbors) algorithm to impute missing values ​​in the denoised dataset.

[0035] Formula 2:

[0036] Where d(x) i ,x j ) represents the original KNN Euclidean distance. x i For missing samples; x j d'(x) represents candidate samples, m is the dimension of non-missing features, and α = 0.9 is the time decay factor. i ,x j () is a similarity metric used to quantify x i With x j The degree of similarity is indicated by the smaller the value of d'(xi,xj), the more similar the two samples are. j The more suitable it is as x i The nearest neighbors are used to fill in the missing values.

[0037] For missing sample x i Filter K=5 d'(x) i ,x j The smallest candidate sample. Continuous fields are filled with the mean of K candidate samples, and discrete fields are filled with the mode. The final result is a completed dataset with a missing data rate reduced from 8% to below 0.5% and a relative error of ≤3% for the filled fields.

[0038] 2.3: Class Imbalance Handling: An improved SMOTE (Synthetic Minority Oversampling) algorithm is used to synthesize anomalous samples in the completed dataset.

[0039] Traditional SMOTE synthesizes samples only in the feature space, which may lose temporal trends; the improved SMOTE utilizes temporal gradients. t x ensures that the synthesized samples retain the temporal characteristics of the anomalies, making them more similar to real anomalies.

[0040] Formula 3:

[0041] Where, x i '、x j ' represents two real anomaly samples of the same anomaly type; λ∈[0,1] represents random coefficients; γ seq =0.3 represents the time series weight; ▽ t x represents the rate of change of features between adjacent time steps, ensuring that the synthesized sample retains the temporal trend. ▽ t x=x t -x t-1 x t It is the feature of the abnormal sample at the current time step, xt-1 These are the features of the anomalous samples from the previous time step; x new This is a newly synthesized anomalous sample.

[0042] Anomaly types refer to traffic categories in the network that exhibit malicious intent or abnormal behavior. Anomaly types include DDoS attacks, SQL injection attacks, port scanning, and DNS hijacking. Samples are synthesized by grouping them according to anomaly type, with x randomly selected from each group. i 'and x j ';Put λ and ▽ t Substituting x into formula 3, we can synthesize x. new After synthesis, the proportion of anomalous samples in the total sample size is calculated. If the proportion is off, some synthesized samples are deleted or a corresponding number of samples are added. Low-quality synthesized samples with a cosine similarity of <95% to real anomalous samples are prioritized for deletion. The proportion of anomalous samples after synthesis is controlled to be 10% to balance realism and category balance, and the cosine similarity between synthesized samples and real anomalous samples is verified to be ≥95%. The final result is a balanced dataset with an anomalous sample proportion of 10%, a sample size of ≥1 million, and a feature dimension of 28. It should be noted that the cosine similarity is calculated based on the 28-dimensional completed dataset output in step 2.2.

[0043] Step 3: Network traffic feature engineering: Based on a balanced dataset, extract statistical features, temporal features, and semantic features. Improve mutual information entropy to filter features with high discriminative power and stability to form an optimized feature set.

[0044] Furthermore, eight core statistical features are extracted and standardized by Z-score to form a 16-dimensional vector; temporal features are extracted through a three-layer LSTM network with 128 hidden units, outputting a 64-dimensional vector; semantic features are extracted based on a Word2Vec and protocol weight-enhanced multi-head attention mechanism.

[0045] Step 3.1: Statistical Feature Extraction: Use session-level aggregation and standardization methods to extract and standardize statistical features from the balanced dataset.

[0046] Calculate 8 core statistical features: The methods for extracting statistical features are as follows: Stream duration: Calculates the time difference between the first and last data packets in a session: Original duration T = t end -t start , t start t is the session start time. end The session end time; derived metrics include the maximum duration T in the sample set. max .

[0047] Packet Count and Total Bytes: Counts the number of all data packets and the total number of bytes in a session. Packet Count N = ∑k=1 n 1, n is the total number of data packets in the session; total number of bytes B = ∑ k=1 n len(pk) is the length of the k-th data packet.

[0048] Average packet size and packet rate: Calculated based on the number of packets and duration, average packet size (-)=B / N; Packet rate R p =N / T p .

[0049] Byte Rate and Uplink / Downlink Ratio: Calculated based on total number of bytes and duration, and uplink / downlink byte count, byte rate R b =B / T; Uplink / Downlink Ratio: R up / down =B up / B down B up B represents the number of bytes in the uplink. down This represents the number of bytes in the downlink.

[0050] TCP Flag Percentage: The percentage of packets containing specific TCP flags in a session, such as the percentage of SYN packets (R). SYN =N SYN / N,N SYN The number of packets containing the SYN flag; the percentage of FIN packets (R). FIN =N FIN / N,N FIN This represents the number of data packets containing the SYN flag.

[0051] Mean and variance of packet intervals: Calculate the time interval Δt between adjacent packets within a session. k =t k -t k-1 Then calculate the mean and variance, and the interval mean μ. △t =∑ k=2 n △t k / (n-1); Interval variance σ △t 2 =∑ k=2 n (△t k -μ △t ) 2 / (n-2).

[0052] Source IP and Destination IP Packet Ratio: This represents the percentage of packets sent from the source IP to the destination IP in a session: Source IP Ratio src =N src / N,N src Number of packets sent to the source IP; percentage of packets sent to the destination IP (R) dst=N dst / N,N dst The number of data packets sent to the destination IP.

[0053] DNS query frequency and HTTP request method proportion: Statistics for application layer protocols: DNS query frequency f DNS =N DNS / T, N DNS Number of DNS lookup packets within a session; percentage of HTTP POST method (R) POST =N POST / N HTTP N POST N represents the number of POST requests. HTTP This represents the total number of HTTP requests.

[0054] The extracted statistical features were standardized using the Z-score standardization algorithm to eliminate the influence of dimensions. Eight core statistical features, each containing two indicators, were combined after standardization to form a 16-dimensional standardized statistical feature vector V. stat =[x 1norm ,x 2norm ,...,x 16norm ], where x 1norm ~x 16norm These are 16 indicators corresponding to 8 categories of features.

[0055] Step 3.2: Temporal feature extraction.

[0056] A 3-layer LSTM (Long Short-Term Memory) network was used to extract temporal features from a balanced dataset. The LSTM network consisted of an input layer, two hidden layers, and an output layer. The hidden layers had 128 hidden units, and the output layer consisted of a fully connected network. The LSTM had 50 input windows, each containing 10 data packets. Temporal feature extraction was achieved through forget gates, input gates, cell state gates, and output gates.

[0057] The forgetting gate is formula 4:

[0058] Among them, W f h is the forget gate weight matrix; t-1 The hidden state of the previous time step; x t Enter the current time step; b η =1.2 is the anomaly sensitivity factor, which enhances the weight of anomaly time-series features through training with historical anomaly data; b f For bias; σ is the sigmoid function; f t 'This is the output of the improved forget gate.'

[0059] Each session is divided into 50 windows based on time steps, with each window containing the statistical characteristics of 10 data packets, forming a 50×16 input sequence. After processing by an LSTM network, the hidden state h of the last time step is extracted. 50 Through compression using a fully connected layer, a 64-dimensional temporal feature vector V is obtained. seq =[s1,s2,...,s 64 ], where s1~s 64 These are the compressed time-series feature values. Each value represents the characteristics of the flow in different time-series dimensions. The cosine distance between abnormal and normal samples is ≥0.7.

[0060] Step 3.3: Semantic feature extraction.

[0061] We use Word2Vec word vectors + 8-head MultiHeadAttention (multi-head attention mechanism) and introduce a protocol weight matrix P to enhance the semantic weight of high-risk protocols and perform semantic feature extraction on the balanced dataset.

[0062] Formula 5:

[0063] Where Q is the query matrix; K is the key matrix; V is the value matrix; P is the protocol weight matrix, HTTP=1.5, DNS=1.2, others=1.0; d k =32 represents the key dimension, enhancing the semantic weight of high-risk protocols; Attention′(Q,K,V) is the result of multi-head attention, quantifying the similarity between the query vector Q and the key vector K, and weighting the value vector V through similarity weights to output features that focus on key semantics.

[0064] During computation, the application layer fields are first converted into Word2Vec word vectors, forming a semantic sequence of length 20. After parallel computation by 8 Attention heads, a 20×32 matrix is ​​concatenated and then compressed into 32-dimensional semantic features through a fully connected layer, resulting in the semantic feature vector V. sem =[m1,m2,...,m 32 ], where m1~m 32 These are the compressed semantic feature values. Each value represents the characteristics of the application layer text in different semantic dimensions. The cosine distance between malicious and normal URLs is ≥0.8.

[0065] Step 3.4: Feature Filtering. The improved mutual information entropy filtering method is used to perform feature filtering on the feature sets merged in steps 3.1 to 3.3.

[0066] Formula 6:

[0067] Where X is the feature; Y is the anomaly label, taking values ​​of 0 and 1, where 0 = normal and 1 = abnormal; P(x,y) is the joint probability; P(x) and P(y) are the marginal probabilities; MI(X,Y) is the mutual information between feature X and anomaly label Y, and the larger the MI, the stronger the feature discrimination.

[0068] Some features are greatly affected by network fluctuations; therefore, a feature stability factor δ is introduced. Formula 7: δ = 1 - σ MI / E[MI] Where, σ MI δ represents the standard deviation of mutual information across different time windows, and E[MI] is the mean of MI. The closer δ is to 1, the more stable the feature.

[0069] Set a screening threshold: MI(X,Y)·δ>0.3 to retain features with high discriminative power and stability.

[0070] Using a 24-hour sliding window with 100,000 samples per window, calculate MI(X,Y) and δ for 96-dimensional features; retain features that satisfy MI(X,Y)·δ>0.3 and remove redundant features; the filtered statistical features, temporal features, and semantic features each comprise an optimized feature set V. opt =[V stat 16 V seq 16 V sem 16 ], where: V stat 16 : 16 dimensions selected from 16 standardized statistical features, retaining the ones with the highest discriminative power; V seq 16 : 16 dimensions were selected from 64 time-series features, retaining those strongly correlated with anomalous time-series trends; V sem 16 : Sixteen dimensions were selected from the 32 semantic features, retaining those strongly correlated with malicious semantics. V opt The total dimensions are 48, including ≥50 labeled abnormal samples per class and ≥100,000 unlabeled samples.

[0071] Step 4: Small sample adaptive learning: Initialize meta-parameters based on the improved MAML algorithm, and combine the improved DQN algorithm to select high-value samples for labeling optimization to achieve small sample adaptive learning.

[0072] Furthermore, the MAML algorithm introduces anomaly type weights, with unknown anomalies having a greater weight than known anomalies, thereby improving its adaptability to unknown anomalies and optimizing the adaptation of meta-parameters to small samples. The DQN algorithm is improved by using mutual information, temporal stability, and prediction uncertainty as state inputs, and balancing annotation costs and accuracy through a reward function.

[0073] Step 4.1: Meta-learning initialization: Using the improved MAML (Model Independent Meta-learning) algorithm, train the initial meta-parameters of the basic detection model so that the model can quickly adapt to new anomalies with only a small number of labeled samples.

[0074] In network anomaly detection, there are very few labeled samples for novel and unknown anomalies. Traditional models require a large amount of labeled data to train, while meta-learning initialization, through optimization, allows the model to quickly learn to identify novel anomalies with only a small number of labeled samples. MAML builds a basic model for anomaly detection based on meta-learning initial parameters.

[0075] The basic detection model is a classification model that fuses temporal and semantic features. Specifically, it consists of an input layer, an LSTM temporal feature extraction layer, a MultiHeadAttention semantic feature extraction layer, and a Softmax classification layer. The classification layer outputs two probabilities: normal or abnormal. The MAML algorithm does not change the model structure, but only optimizes its initial parameters.

[0076] Unknown anomaly detection has higher priority, and anomaly type weight w is introduced. a Correcting the loss in MAML, unknown anomaly w a =1.5, known anomaly w a =1.0, which allows MAML to focus more on adaptability to unknown exceptions when optimizing initial parameters.

[0077] Formula 8: L Ti '=L Ti (f θ ki )·w a Among them, L Ti 'This represents the weighted task loss; T' i For a single task, including a support set and a query set; L Ti For a single task T i Cross-entropy loss; f θ ki For the model parameters θ via k i The updated task adaptation model; k i The inner loop updates the step count, taking steps 1-5; w a Weights for abnormal types.

[0078] Improved MAML objective function: Formula 9: θ' = argmin θ ∑ Ti∈T L Ti ' Where θ' is the initial parameter for meta-learning; T is the task set; the initial parameter is optimized to adapt to a small sample size.

[0079] The calculation steps are as follows: Task set construction: Optimize feature set V opt Divide into 100 tasks T=T1,...,T 100 Each T i Includes support set S i 10 labeled samples, query set Q i 100 test samples; all labeled samples from the support set and query set together constitute the training set; Inner loop: For each T i , using S i Calculate the loss L Ti (f θ Gradient update parameters: θ i =θ-α▽ θ L Ti (f θ ), where L Ti (f θ The model uses initial meta-parameters θ without inner loop updates in task T. i The cross-entropy loss is calculated on the support set, and α=0.01 is the inner loop learning rate; Outer loop: θ i Substitute task T i Q i L was calculated Ti Update the initial parameters: θ' = θ - β▽ θ ∑ Ti∈T L Ti ', where β=0.001 is the meta-learning rate. After 100 training rounds, the initial meta-learning parameters θ' are obtained. The accuracy of the model in small sample scenarios is ≥85%, and the speed of adapting to unknown anomalies is 20% faster than that of standard MAML.

[0080] Step 4.2: Reinforcement Learning Label Optimization: Using the improved DQN (Deep Q-Network) algorithm, high-information-value samples are selected and labeled for θ' and unlabeled samples.

[0081] Reinforcement learning modeling: The DQN model structure is: input layer, hidden layer 1, hidden layer 2, output layer, and the output layer outputs the action value Q.

[0082] The input to the DQN model is state s: s=[MI,δ t [,H], where MI is the mutual information between the sample and the known anomaly; the larger the MI, the more likely the sample is to be a known anomaly, δ tLet H represent the temporal stability of the sample, H represent the model prediction uncertainty, H = -∑P(y|x)logP(y|x), where P(y|x) is the prediction probability of the basic detection model that sample x belongs to category y, and log is the natural logarithm; Action a: 0 means no labeling, 1 means requesting labeling; Reward function r: r = 0.8·acc + 0.2·(1-cost), where acc is the accuracy improvement value after labeling, cost is the labeling cost coefficient, labeling = 1, no labeling = 0.

[0083] The output is the action value Q for each action. The larger the Q value, the higher the long-term cumulative reward that the action can bring. The model will prioritize actions with large Q values.

[0084] Improved Q function, Formula 10: Q(s,a)=r+γ dis max a' Q(s',a')·η exp Where Q(s,a) is the action value of performing action a in state s, and Q(s,a) is a quantitative indicator of long-term expected return; the larger the value, the better the action; s' is the next state after performing the action; r is the reward function; γ dis =0.9 is the discount factor; η exp =0.95 is the exploration rate decay factor to avoid overexploration.

[0085] Q(s,a) is used to guide the selection of labeled actions: by calculating the Q-values ​​of different actions, actions with larger Q-values ​​are selected. If Q(s,1) > Q(s,0), ​​meaning the long-term expected return of the labeled action is higher than the long-term expected return of the unlabeled action, then manual labeling of the sample is requested, and a=1; otherwise, no labeling is performed, and a=0. This is achieved through the reward function r and the discount factor γ. dis This ensures that the model prioritizes sample annotations that can significantly improve accuracy and have low cost, ultimately reducing annotation costs by 60% and balancing annotation benefits and costs.

[0086] The experience replay pool, Q-network, and epsilon greedy strategy are initialized. After 50 training cycles, two branch results are finally obtained.

[0087] The training loop includes: Sampling: 1000 samples are sampled from unlabeled samples in each round, and the state s is calculated; Action selection: Based on an epsilon greedy strategy, the DQN model predicts Q(s,0) and Q(s,1), and the action a with the larger Q value is selected; Reward calculation: Action a is executed. If labeled, manual labeling and accuracy calculation are performed. If not labeled, detection proceeds directly, and r is calculated; Experience storage: (s,a,r,s') is stored in the replay pool; Network update: 32 experience samples are sampled from the replay pool, and the DQN network is updated using MSE (mean squared error) loss with a learning rate of 0.0001 to minimize the difference between the predicted Q value and the target Q value. The trained DQN model can stably output the optimal labeled action, ultimately reducing the labeling cost from 10,000 to 4,000, while ensuring improved model generalization of labeled samples.

[0088] The branch results include: Branch 1, action a=1: 4000 high-information-value labeled samples, the basic detection model is the initial model, θ' is used as the initial parameter, 5 rounds of stochastic gradient descent optimization are performed using 4000 labeled samples, the learning rate is 0.001, the parameters are updated using cross-entropy loss in each round, and finally the fine-tuned parameters θ are obtained. fine Branch 2, action a=0: 96,000 low-information-value unlabeled samples are directly passed to step 5 for anomaly detection.

[0089] Step 5: Network traffic anomaly detection: Use the optimized feature set to train a classification model to classify traffic as normal, known anomaly, and unknown anomaly, and generate anomaly attribution explanations through causal reasoning.

[0090] Furthermore, the classification model outputs three probabilities based on the Softmax classifier, sets an unknown anomaly detection threshold to determine unknown anomalies; the causal inference uses the DoWhy framework to calculate the total effect and individual treatment effect, and generates key causal factors and counterfactual explanations.

[0091] Step 5.1: Anomaly Classification Model: Using a Softmax classifier, classify θ fine Unlabeled samples are classified into three traffic categories: normal, known abnormal, and unknown abnormal.

[0092] Probability calculation for the Softmax classifier, Equation 11:

[0093] Among them, z i The model outputs logits, corresponding to three types of traffic: normal, known anomaly, and unknown anomaly; P(y i |x) represents the category y to which sample x belongs. i The probability, ∑P(y) j |x)=1.

[0094] An unknown anomaly detection threshold τ=0.6 is introduced, which is determined by 5-fold cross-validation to balance the recognition rate and false alarm rate. The classification rules are set as follows: if P(normal|x)>τ, it is normal traffic; if P(known anomaly|x)>τ, it is known anomaly, and the attack type is output based on the label matching of known anomaly samples; if max(P(y|x))<τ, it is unknown anomaly, and the model has a low probability for all three types, so it is judged as a novel anomaly.

[0095] During computation, the 48-dimensional optimized feature set is input into the basic detection model θ. fine Output three types of probabilities and classify them according to the rules. Output the classification label of each sample to obtain three types of traffic labels.

[0096] Step 5.2: Causal reasoning analysis.

[0097] Using the DoWhy (causal reasoning framework), we perform anomaly attribution and counterfactual explanation processing on the output known and unknown anomaly samples, their corresponding 48-dimensional features, and classification labels. A causal graph G is constructed using a Bayesian network, with the 48-dimensional optimized feature set as the causal variable and the anomaly labels as the outcome variable.

[0098] Calculate the total effect TE using Formula 12: TE = E[Y|do(X=1)] - E[Y|do(X=0)] Where X is one of the 48-dimensional optimized features; Y is the anomaly label, 1 = abnormal, 0 = normal; E[·] is the mathematical expectation; do(X=1) indicates that the intervention feature X is an anomaly; do(X=0) indicates that the intervention feature X is a normal value; TE>0.3 indicates that X is a key causal factor.

[0099] Calculate the individual treatment effect (ITE) using Formula 13: ITE u =Y u (X=1)-Y u (X=0) Among them, ITE u To represent the individual treatment effect of sample u, we quantify the contribution of feature X to anomalies in a single sample u; u is a single unknown anomaly sample; Y u (X=1) represents the anomaly label of sample u when X=1, where 1 = anomaly; Y u (X=0) indicates that the sample u is an abnormal label when X=0, and 0=normal.

[0100] The calculation first constructs a causal graph, with features as causal variables and anomaly labels as outcome variables. After controlling for confounding variables, TE and ITE are calculated to generate counterfactual explanations. For example, if the URL does not contain malicious keywords, the anomaly probability drops from 85% to 23%. Finally, an anomaly sample causal explanation report is obtained, which includes key factors, causal path graphs, and counterfactual statements.

[0101] Counterfactual prediction, Formula 14: Y x (u)=f(x,u)+ε Among them, Y x (u) is the predicted label of sample u after intervention X=x; f(x,u) is the causal prediction model; x is the value of X after intervention; ε is the error.

[0102] Explanation: If the URL of sample u does not contain the malicious keyword A, the anomaly probability drops from 85% to 23%.

[0103] The calculation steps are as follows: Data preprocessing: Discretize the features of abnormal samples, such as packet rate being divided into low, medium and high, and construct a causal analysis dataset; Causal identification: Control confounding variables, such as network bandwidth, through backdoor criteria, and calculate TE and ITE; Explanation generation: Output key causal factors, causal path diagrams and counterfactual statements for unknown abnormal samples, such as DNS query frequency → packet rate → anomaly; Output attack attributions for known abnormal samples, such as the key factor of DDoS being a sudden increase in packet rate.

[0104] The above processing outputs an abnormal sample causal explanation report, including a list of key causal factors, a causal path visualization diagram, and counterfactual explanation statements.

[0105] Step 5.3: Classification result branch processing: The classification labels and causal explanation reports are processed in three branches.

[0106] Branch 1, Normal Traffic, ≈89%: Store normal traffic samples in the normal traffic database, update the normal traffic characteristic distribution parameters using a 24-hour sliding window, and obtain the updated normal traffic database, including the real-time mean μ. norm Standard deviation σ norm This result is used to provide a historical distribution benchmark for drift monitoring in step 6.

[0107] Branch 2, known anomalies, accounting for approximately 1%: Using the alarm output + defense linkage method, known anomaly samples are processed, and alarm information is output. The alarm information includes attack type, source IP, attack time, session 5-tuple, and causal attribution. A blocking command is sent to the firewall to block the abnormal source IP. This result is used to add the abnormal samples to the known anomaly database, providing data for incremental learning in step 6.

[0108] Branch 3, Unknown anomalies, accounting for approximately 10%: Mark the unknown anomaly samples as high-priority samples to be learned and store them in the unknown anomaly cache pool. The cached content includes sample features, classification probabilities, and counterfactual explanations; thus, the unknown anomaly cache pool is obtained; this result is used to pass into step 6 for incremental learning.

[0109] Step 6: Data Distribution Drift Adaptation and Incremental Learning: Monitor traffic distribution drift, update classification model parameters based on the improved incremental SVM algorithm to adapt to changes in data distribution. The improved incremental SVM algorithm introduces anomaly type priority based on the traditional SVM algorithm to update support vector coefficients.

[0110] Furthermore, the flow distribution drift monitoring uses a 24-hour sliding window to calculate the average KL divergence and sets a drift threshold to determine the drift status; incremental learning introduces anomaly type priority and updates the SVM support vector coefficients.

[0111] Step 6.1: Distribution drift monitoring: Using the improved KL divergence monitoring method, the normal flow database and training set are processed to determine the difference in flow distribution and whether there is drift.

[0112] Formula 15: D KL (P||Q)=∑ x P(x)log[P(x) / Q(x)] Among them, D KL (P t ||Q) is the original KL divergence, which measures the difference in distribution between P and Q, D KL ≥0, the larger the value, the greater the difference; x is a specific value in the 48-dimensional optimized feature set; P is the real-time traffic feature distribution, which comes from the normal traffic database; P(x) is the probability of feature x in P; Q is the historical training data distribution, which comes from the labeled samples of the training set; Q(x) is the probability of feature x in Q.

[0113] Network traffic exhibits daily periodicity. Introducing a 24-hour sliding window T=24h, the average KL divergence within the window is calculated: Formula 16: D avg =(1 / T)∑ t=1 T D KL (P t ||Q) Among them, D avg The average KL divergence is the value of the 24-hour sliding window; T=24h is the duration of the sliding window; P t Let P be the real-time traffic characteristic distribution for hour t. Every hour, extract all normal samples from the normal traffic database for that hour, and statistically analyze the frequency of each value in the 48-dimensional optimized feature set, thus obtaining P. t .

[0114] Set a drift threshold, based on historical data statistics, D avg When the value is greater than 0.3, the model accuracy decreases by ≥10%, therefore D is set to... th =0.3. The 48-dimensional characteristic distribution P of the hourly normal flow database. t ; Calculate D KL (Pt ||Q), Update D avg Determine the drift state: D avg ≤0.3, no drift, 0.3 < D avg ≤0.5, mild, D avg >0.5, severe. No drift means θ is maintained. fine If there is drift, step 6.2 incremental learning is triggered, and a drift status report is output in real time.

[0115] Step 6.2: Incremental Learning: Traditional SVM requires full retraining, resulting in long training times and high resource consumption. This embodiment uses an improved incremental SVM (Support Vector Machine) algorithm to update the anomaly classification model parameters based on the known anomaly database, the unknown anomaly cache pool, and drift state reports. The anomaly classification model is based on θ. fine The SVM classifier.

[0116] In incremental SVM, the Lagrange multipliers α' of the support vectors are used to measure the influence of a particular support vector on the SVM classification hyperplane. Learning unknown anomalies has a higher priority; therefore, the improved incremental SVM algorithm introduces anomaly type priority p. a Unknown anomaly p a =1.2, known anomaly p a =1.0, correct α'.

[0117] Formula 17: α new '=α old '+η lr ·p a ·▽L Where, α new ' represents the updated support vector coefficients; α old ' represents the support vector coefficients before the update; η lr =0.005 is the learning rate; p a ▽L represents the priority of the anomaly type; ▽L represents the hinge loss gradient.

[0118] The calculation steps are as follows: Incremental datasets are constructed according to drift status: No drift: 10,000 normal samples + 960 known anomalies + 9,600 unknown anomalies; With drift: 20,000 normal samples + 960 known anomalies + 9,600 unknown anomalies; The sample size is calculated based on the anomaly rate of 10% in the balanced dataset in step 2.3 and the enterprise's average daily traffic of 100,000 records. Parameter update: Fix the parameters of the underlying feature extraction layers, such as the LSTM temporal feature extraction layer and the multi-head attention semantic feature extraction layer, and only update the α' of the SVM; Model validation: Use 10% incremental data as the validation set. If the accuracy improves by ≥5%, obtain the incremental model parameters θ. incrSave; otherwise, roll back to θ. fine .

[0119] After the above processing, θ incr Parameter update rate ≈ 15%, training time ≤ 1 hour; accuracy decay from 20% to below 5% in distribution drift scenarios, θ incr As new detection model parameters, they are passed into step 7 for iterative optimization.

[0120] Step 7: Model Iteration Optimization and Feedback: Construct a multi-dimensional comprehensive evaluation function, optimize parameters based on performance shortcomings, and generate an operation and maintenance feedback report to achieve closed-loop optimization.

[0121] Furthermore, the comprehensive evaluation function includes accuracy, false alarm rate, annotation cost, and drift adaptation rate, and the model parameters are dynamically adjusted based on the evaluation results.

[0122] Step 7.1: Construction of comprehensive evaluation function: Use a multi-dimensional comprehensive evaluation function to quantify the model performance of the feedback data throughout the process. The feedback data throughout the process includes the accuracy and false alarm rate in step 5, the annotation cost in step 4, the number of drifts in step 6, and manually misjudged samples.

[0123] Formula 18: F = 0.4·acc - 0.3·fr - 0.2·cost + 0.1·dr Where F is the comprehensive evaluation value, with a target F≥0.85; acc is the detection accuracy, with a weight of 0.4; fr is the false alarm rate, with a weight of 0.3; cost is the annotation cost coefficient, with a weight of 0.2; and dr is the drift adaptation rate, with a weight of 0.1. The weights are determined using the analytic hierarchy process (AHP) combined with the enterprise's operational needs, as follows: Target layer: Improve detection effectiveness and operational efficiency; Criterion layer: Accuracy: acc weight 0.4, loss due to missed detections > false positives; False positive rate: fr weight 0.3, false positives increase operational costs; Labeling cost: cost weight 0.2, manual labeling costs are high; Drift adaptation rate: dr weight 0.1, drift occurs at a low frequency; Consistency test: CR = 0.08 < 0.1, weight allocation is reasonable.

[0124] During the calculation, weekly feedback data is collected, and acc, fr, cost, and dr are calculated and substituted into the formula to obtain the comprehensive evaluation value F and performance weakness analysis. If F < 0.85, it is determined whether the acc is low or the cost is high.

[0125] Step 7.2: Parameter Iteration Optimization: Use targeted parameter adjustment strategies to optimize the key parameters of the model for F-value and bottleneck analysis.

[0126] Optimization rules: If acc < 98%: adjust the number of MAML inner loop steps from 5 to 8, and the number of LSTM hidden units from 128 to 256; If fr > 1.2%: adjust the classification threshold τ from 0.6 to 0.65, and the weight of acc in the DQN reward function from 0.8 to 0.9; If cost > 4000 items: adjust the DQN exploration rate decay factor η from 0.95 to 0.98; If dr < 90%: reduce the KL divergence threshold D. th : 0.3→0.25, increase the incremental learning sample size from 20,000 to 30,000.

[0127] Finally, the key parameters of the optimized model were obtained.

[0128] Step 7.3: Feedback Report Generation: Using the report generation template, the evaluation results of Step 7.1, the parameter adjustment content of Step 7.2, and the abnormal statistical data of Steps 5 and 6 are processed to generate operation and maintenance reports, resulting in weekly and monthly inspection reports. These reports are fed back to the operation and maintenance team, and the optimized parameters are used as the initial parameters for the next round of processes, forming a closed loop.

[0129] The report includes: core metrics: accuracy, false positive rate, unknown anomaly identification rate, annotation cost, and drift count; anomaly statistics: percentage of known anomaly types, number of unknown anomalies discovered, and top 5 anomaly source IPs; and model updates: parameter adjustments, incremental learning iterations, and performance improvement.

[0130] As one implementation method, the following scenario is used as an example: Network environment: Enterprise campus network, 1000 terminals, outbound bandwidth 100Mbps; Time range: 2024-05-20 09:00-10:00, peak working hours, high traffic density; Core task: Detect DDoS attacks and new SQL injections during this period, i.e., known anomalies and unknown anomalies.

[0131] The testing process is as follows: 1. Data acquisition and fusion.

[0132] Raw packet capture: 1.2 million PCAP packets were captured using PFRING, with a packet loss rate of 0.08% ≤ 0.1% and a latency of 0.8ms ≤ 1ms, including 900,000 TCP packets and 300,000 UDP packets.

[0133] Multi-source fusion: Session segmentation: Segment the IP address (source IP=192.168.1.101, destination IP=203.0.113.5, source port=54321, destination port=80, protocol=TCP, timestamp=09:05:30.2) to generate a session. ID=TCP_192.168.1.101_203.0.113.5_54321_80_0905302; Log association: Matching NetFlow logs and application layer logs, NetFlow log session duration 120s, byte rate 1024KB / s; Invalid data removal: Removing 20,000 missing PCAP logs, resulting in 1.18 million session-level fused data entries.

[0134] 2. Data preprocessing.

[0135] Noise filtering, taking the TCP traffic packet length field as an example: TCP traffic statistics: mean packet length μ t =1450 bytes, standard deviation σ t =200 bytes, weight w t =0.8; A data packet length x = 2200 bytes, substituting into Formula 1: |2200-1450| = 750, 3σt·wt = 3 × 200 × 0.8 = 480; Since 750 > 480, it is judged as noise, a total of 59,000 noise entries are filtered, resulting in 1,121,000 denoised data entries. Missing value imputation, taking the DNS query frequency field as an example: missing sample x i : Session ID=TCP_192.168.1.202_203.0.113.6_65432_443_0910251, DNS query frequency is missing; Candidate sample selection: K=5, original Euclidean distance d(x) i ,x j =2.5, time decay factor α = 0.9, x j1 With x i If the time interval is 5 seconds, then d'(x) i ,x j1 ) = 2.5 × 0.9 = 2.25; Imputation: The DNS query frequency of the 5 candidate samples is [0.5, 0.6, 0.5, 0.7, 0.5] times / s, with a mean of 0.56 times / s. Imput x using the mean. i The final missing rate was 0.4% < 0.5%.

[0136] Class Imbalance Handling: Real DDoS Samples x i Packet rate = 1000 packets / s, x j Packet rate = 1200 packets / s; Timing gradient ▽ t x=x t -x t-1 =1000-900=100 packets / s, x t x represents the current DDoS sample packet rate. t-1Let be the packet rate of the previous time step; substituting into Formula 3, λ = 0.6, γ seq =0.3:x new =1000+0.6×(1200-1000)+0.3×100=1150 packets / s; synthesize 99,890 abnormal samples, delete 389 samples with cosine similarity = 93%<95%, and finally have 100,000 abnormal samples, accounting for 10%, with a total of 1.1 million samples and 28-dimensional features.

[0137] 3. Feature engineering.

[0138] Statistical feature extraction, taking session ID=TCP_192.168.1.101_... as an example: t end =09:07:30.2, t start =09:05:30.2, flow duration T=120s, T max =300s; Number of packets N=1200, Total number of bytes B=1200×1500=1.8e6 bytes; Z-score normalization: Packet rate Rp=10 packets / s, μ=5 packets / s, σ=3 packets / s, x norm =(10-5) / 3≈1.67; forming a 16-dimensional statistical characteristic V stat =[1.2,0.8,...,1.67,...].

[0139] Temporal feature extraction: Session segmentation: 50 windows, 10 data packets per window, input sequence 50×16; LSTM forget gate calculation: W f ·[h t-1 ,x t ]=0.5, b η =1.2, b f =0.1, then f t =σ(0.5×1.2+0.1)=σ(0.7)≈0.668; Take the last time step h 50 Compressed into 64-dimensional time-series features V seq =[0.321,0.45,...].

[0140] Semantic feature extraction, URL=http: / / example.com / login: Word2Vec word vectors: split into ["http","example.com","login"], forming a 20-dimensional sequence; Multi-head attention calculation: P=1.5, d k =32, calculate Attention'(Q,K,V) to focus on the login feature; compress it into a 32-dimensional semantic feature V. sem =[0.6,0.4,...].

[0141] Feature selection: Calculate MI and δ for 96-dimensional features: Packet rate feature MI=0.45, σ MI =0.05, E[MI]=0.5, δ=1-0.05 / 0.5=0.9; MI·δ=0.45×0.9=0.405>0.3, retain; finally, select 48-dimensional optimized features V opt =[V stat16 V seq16 V sem16 ].

[0142] 4. Small sample adaptive learning.

[0143] Meta-learning initialization: Task set: 100 tasks, S per task i =10 labeled samples, Q i =100 samples; Inner loop: α=0.01, L Ti =0.3, θ1=θ-0.01×▽θ×0.3; Outer loop: Unknown anomaly w a =1.5, L Ti =0.3×1.5=0.45, β=0.001, θ' is obtained after 100 training rounds, with an accuracy of 88%.

[0144] Reinforcement learning annotation optimization: DQN state s=[MI=0.5,δt=0.85,H=0.6]; H=-[0.4log0.4+0.3log0.3+0.3log0.3]≈0.6; calculate Q(s,1)=0.12+0.9×0.8×0.95≈0.804, Q(s,0)=0.6485, select annotation a=1; train for 50 rounds, obtain 4000 labeled samples and 96,000 unlabeled samples, fine-tune θ' to obtain θ fine .

[0145] 5. Anomaly detection.

[0146] Anomaly classification, for a certain HTTP session sample: output logitsz=[-1.2 (normal), 2.5 (DDoS), 0.8 (unknown)]; Formula 11 calculates the probability: P(normal)=0.02, P(DDoS)=0.828, P(unknown)=0.151; τ=0.6. Since P(DDoS)=0.828>0.6, it is judged as a known abnormal DDoS.

[0147] Causal reasoning: The total effect TE = E[Y|do(packet rate=1000)] - E[Y|do(packet rate=10)] = 0.9 - 0.1 = 0.8 > 0.3, indicating that packet rate is the key factor. Counterfactual explanation: If the packet rate decreases from 1000 packets / s to 10 packets / s, the anomaly probability decreases from 82.8% to 15%. Branch processing: Output an alarm, attack type = DDoS, source IP = 192.168.1.101, send a blocking command to the firewall, and add the sample to the known anomaly database.

[0148] 6. Drift adaptation and incremental learning.

[0149] Drift monitoring: 24-hour Davg = (0.25 + 0.28 + ... + 0.32) / 24 ≈ 0.29 ≤ 0.3, no drift, maintain θ fine Incremental learning, simulating mild drift: Incremental dataset: 20,000 normal samples + 960 known anomalies + 9,600 unknown anomalies; Support vector update: α old =0.4, η lr =0.005, p a =1.2, ▽L=0.3, α new =0.4 + 0.005 × 1.2 × 0.3 = 0.4018; The verification accuracy improved by 7%, yielding θ. incr .

[0150] 7. Iterative optimization and feedback.

[0151] Comprehensive assessment, week 3 of May: Indicators: acc=0.98, fr=0.01, cost=0.4, dr=0.9; F=0.4×0.98+0.3×(1-0.01)+0.2×(1-0.4)+0.1×0.9=0.392+0.297+0.12+0.09=0.901≥0.85. Parameter optimization: No weaknesses, parameters maintained; Feedback report: Core indicators: Accuracy 98%, False positive rate 1%, Anomaly statistics: DDoS rate 60%, Model update: No parameter adjustments.

[0152] Final detection results: Normal traffic: 979,000 entries, accounting for 89%, stored in the normal traffic database; Known anomalies: 11,000 entries, accounting for 1%, all of which were DDoS attacks, and 10 abnormal source IPs were blocked; Unknown anomalies: 110,000 entries, accounting for 10%, marked as new types of SQL injection, and stored in the unknown anomaly cache pool.

[0153] This application significantly improves data quality and representation capabilities through multi-source data fusion and intelligent preprocessing, providing high-quality input for subsequent deep learning models. By introducing meta-learning and reinforcement learning mechanisms, it achieves rapid model adaptation and high-value sample selection in small-sample environments, greatly reducing annotation costs and improving the model's ability to detect unknown anomalies. Combining causal inference and incremental learning mechanisms enhances the model's interpretability and dynamic adaptability, enabling it to maintain high accuracy and low false alarm rate in constantly changing network environments, demonstrating strong practicality and promotional value.

[0154] Example 2 Based on the same inventive concept as the deep learning-based network traffic anomaly detection method provided in the embodiments of this application, the embodiments of this application also provide a deep learning-based network traffic anomaly detection system. If there is anything unclear about the content in the system embodiments, please refer to the corresponding content in the method embodiments.

[0155] like Figure 2 As shown, a deep learning-based network traffic anomaly detection system includes: Data acquisition module: used to capture PCAP packets, NetFlow logs and application layer agent logs in real time to obtain raw network data; Fusion module: Used to generate session-level multi-source fusion datasets from raw network data by associating session 5-tuples with timestamps; Preprocessing module: used to filter noise, impute missing values, and synthesize outlier samples on the fused dataset, outputting a balanced dataset; Feature engineering module: used to extract statistical, temporal and semantic features from balanced datasets, and to filter high-discrimination features by improving mutual information entropy to form an optimized feature set; The few-shot learning module is used to initialize meta-parameters based on the improved MAML algorithm and combine the improved DQN algorithm to select high-value samples for labeling and optimization. Classification and detection module: Used to train a classification model using an optimized feature set, and output classification results for normal, known abnormal and unknown abnormal. Adaptive optimization module: used to monitor traffic distribution drift and dynamically update model parameters through incremental SVM; Explanation and Feedback Module: Used to generate anomaly attribution explanations based on causal reasoning, construct a comprehensive evaluation function to iteratively optimize model parameters, and generate feedback reports.

[0156] Furthermore, the data acquisition module uses the PFRING framework to capture raw data packets; NetFlow logs are parsed using the IPFIX protocol; and application layer proxy logs are obtained through a RESTful API. The fusion module segments the data using PCAP to generate a unique session ID, based on source IP, destination IP, source port, destination port, transport protocol, and timestamp. It then uses this session ID and timestamp to supplement the session duration and byte rate of NetFlow logs, as well as the URL and DNS domain name of application layer logs, removing data from failed associations. The session-level dataset is then output to the preprocessing module. The preprocessing module performs noise filtering based on traffic type weights; missing value imputation uses an improved KNN algorithm; and outlier synthesis is achieved through temporal gradient enhancement SMOTE. In the feature engineering module, statistical features are standardized by Z-score to form a 16-dimensional vector; temporal features are extracted by LSTM and output as a 64-dimensional temporal vector; semantic features are combined with the protocol weight matrix to enhance the weights of high-risk protocols and output as a 32-dimensional vector; and improved mutual information entropy filtering outputs a 48-dimensional optimized feature set. In the few-shot learning module, the improved MAML algorithm optimizes meta-parameters through gradient descent; the improved DQN algorithm filters samples based on the reward function to obtain labeled samples with high information value and unlabeled samples. The classification and detection module uses the Softmax classifier for classification, combines DoWhy to generate a causal explanation report and branch processing, and sets an unknown anomaly detection threshold to determine unknown anomalies; The adaptive optimization module monitors distribution drift using KL divergence; the incremental SVM updates parameters with a sliding window, and the model is updated once a day, with the parameters fed back to the feature engineering module and the classification and detection module. In the explanation and feedback module, causal reasoning generates attribution reports based on counterfactual analysis; the comprehensive evaluation function integrates accuracy, false alarm rate, and comprehensive evaluation value, and optimizes parameters to feed back to the few-sample learning module and the adaptive optimization module.

[0157] The system's workflow is as follows: The data acquisition module captures PCAP packets, NetFlow logs, and application layer agent logs in real time to generate raw network data; the fusion module constructs a session-level multi-source fusion dataset using session quintuples and timestamp association rules; the preprocessing module performs noise filtering, improved KNN missing value filling, and time-series gradient enhancement SMOTE anomalous sample synthesis on the fusion dataset, outputting a balanced dataset; the feature engineering module extracts statistical, temporal, and semantic features, and forms an optimized feature set through improved mutual information entropy filtering; the few-shot learning module initializes meta-parameters based on an improved MAML algorithm, and combines an improved DQN algorithm to select high-value samples for labeling and optimization; the classification and detection module trains a classification model using the optimized feature set, outputting normal, known anomalous, and unknown anomalous results; the adaptive optimization module monitors traffic distribution drift through KL divergence and dynamically updates model parameters using incremental SVM; the explanation and feedback module generates anomaly attribution reports based on counterfactual analysis, iteratively optimizes parameters through a comprehensive evaluation function, and feeds back to the few-shot learning and adaptive optimization modules, forming a detection-attribution-optimization closed loop.

[0158] Example 3 Based on the same inventive concept as the deep learning-based network traffic anomaly detection method provided in the embodiments of this application, the embodiments of this application also provide a computer device. If there is anything unclear about the content in the device embodiments, please refer to the corresponding content in the method embodiments.

[0159] like Figure 3 As shown, a computer device 100 includes a memory 101, a processor 102, and a network interface 103. The memory, processor, and network interface are interconnected via a system bus. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the method described above.

[0160] It should be noted that only the computer device 100 with components 101-103 is shown in the figure. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices.

[0161] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0162] The memory 101 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory, random access memory, static random access memory, read-only memory, electrically erasable programmable read-only memory, programmable read-only memory, magnetic memory, and magnetism. In some embodiments, the memory 101 may be an internal storage unit of the computer device 100, such as the hard disk or memory of the computer device 100. In other embodiments, the memory 101 may also be an external storage device of the computer device 100, such as a plug-in hard disk, smart memory card, secure digital card, and flash memory card equipped on the computer device 100. Of course, the memory 101 may include both the internal storage unit and the external storage device of the computer device 100. In the embodiments of this application, the memory 101 is generally used to store the operating system and various application software installed on the computer device 100, such as computer-readable instructions of methods. In addition, the memory 101 may also be used to temporarily store various types of data that have been output or will be output.

[0163] In some embodiments, processor 102 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. Processor 102 is typically used to control the overall operation of computer device 100. In embodiments of this application, processor 102 is used to execute computer-readable instructions stored in memory 101 or process data, such as executing computer-readable instructions of the methods provided in embodiments of this application.

[0164] The network interface 103 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 100 and other electronic devices.

[0165] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A network traffic anomaly detection method based on deep learning, characterized in that, Includes the following steps: Raw network traffic data is collected, and combined with NetFlow logs and application layer logs. Multi-source data is correlated and fused based on session quintuples and timestamps to generate a session-level multi-source fused dataset. The fused dataset is preprocessed, including noise filtering based on traffic type weights, missing value imputation, and outlier synthesis, to obtain a balanced dataset. Based on a balanced dataset, statistical features, temporal features, and semantic features are extracted. High-discrimination and stable features are selected by improving mutual information entropy. The mutual information between features and anomaly labels and the feature stability factor are calculated. Features with mutual information × feature stability factor > threshold are selected and retained to form an optimized feature set. Based on the MAML algorithm, anomaly type weights are introduced, with unknown anomalies having a greater weight than known anomalies. Meta parameters are initialized, and high-value samples are selected using the DQN algorithm for labeling and optimization, thus achieving small-sample adaptive learning. A classification model is trained using an optimized feature set to classify traffic as normal, known abnormal, and unknown abnormal, and an explanation for the cause of the abnormality is generated through causal reasoning. To monitor traffic distribution drift, an improved incremental SVM algorithm is used to update the classification model parameters and adapt to changes in data distribution. The improved incremental SVM algorithm introduces anomaly type priority based on the traditional SVM algorithm to update the support vector coefficients. Construct a multi-dimensional comprehensive evaluation function, optimize parameters based on performance shortcomings, and generate operation and maintenance feedback reports to achieve closed-loop optimization.

2. The network traffic anomaly detection method based on deep learning according to claim 1, characterized in that, The collection of raw network traffic data includes real-time capture of raw TCP and UDP packets using the PFRING framework, and the multi-source data association and fusion includes session segmentation, log association, and invalid data removal.

3. The network traffic anomaly detection method based on deep learning according to claim 2, characterized in that, The noise filtering incorporates flow type weighting to improve the 3σ principle; The missing value imputation uses a time decay factor improved KNN algorithm to calculate the similarity between samples. Continuous fields are imputed with the mean, and discrete fields are imputed with the mode. The anomalous sample synthesis uses the SMOTE algorithm with temporal gradient enhancement, which preserves the temporal features of the anomalous sample through temporal gradient and controls the cosine similarity between the synthesized sample and the real anomalous sample.

4. The network traffic anomaly detection method based on deep learning according to claim 1, characterized in that, The statistical features are standardized using Z-scores to form a 16-dimensional vector. The temporal features are extracted through a 3-layer LSTM network with 128 hidden units, and the output is a 64-dimensional vector. The semantic features are extracted based on Word2Vec and a protocol weight-enhanced multi-head attention mechanism.

5. The deep learning-based network traffic anomaly detection method according to claim 4, characterized in that, The steps of the DQN algorithm to select high-value samples include: constructing a deep Q-network model with mutual information, temporal stability, and prediction uncertainty as state inputs; the action space of the deep Q-network model includes two actions: requesting manual labeling and not labeling; balancing labeling cost and accuracy through a reward function; calculating and comparing the long-term value of the two actions; when the deep Q-network model evaluates that the long-term expected benefit of performing the labeling action on the current network traffic sample is higher than the long-term expected benefit of not performing the labeling action, the sample is determined to be a high-value sample, and the manual labeling process is initiated.

6. The network traffic anomaly detection method based on deep learning according to claim 1, characterized in that, The classification model is based on the output of three probabilities by the Softmax classifier, and sets an unknown anomaly detection threshold to determine unknown anomalies. The causal reasoning uses the DoWhy framework to calculate the total effect and individual treatment effects, generating key causal factors and counterfactual explanations.

7. The network traffic anomaly detection method based on deep learning according to claim 6, characterized in that, The flow distribution drift monitoring uses a 24-hour sliding window to calculate the average KL divergence and sets a drift threshold to determine the drift status; the incremental learning introduces anomaly type priority and updates the SVM support vector coefficients; the comprehensive evaluation function includes accuracy, false alarm rate, annotation cost and drift fitness rate, and dynamically adjusts the model parameters based on the evaluation results.

8. A deep learning-based network traffic anomaly detection system, used in any one of the deep learning-based network traffic anomaly detection methods according to claims 1 to 6, characterized in that, include: Data acquisition module: used to capture PCAP packets, NetFlow logs and application layer agent logs in real time to obtain raw network data; Fusion module: Used to generate session-level multi-source fusion datasets from raw network data by associating session 5-tuples with timestamps; Preprocessing module: used to filter noise, impute missing values, and synthesize outlier samples on the fused dataset, outputting a balanced dataset; Feature engineering module: used to extract statistical, temporal and semantic features from balanced datasets, and to filter high-discrimination features by improving mutual information entropy to form an optimized feature set; The few-shot learning module is used to initialize meta-parameters based on the improved MAML algorithm and combine the improved DQN algorithm to select high-value samples for labeling and optimization. Classification and detection module: Used to train a classification model using an optimized feature set, and output classification results for normal, known abnormal and unknown abnormal. Adaptive optimization module: used to monitor traffic distribution drift and dynamically update model parameters through incremental SVM; Explanation and Feedback Module: Used to generate anomaly attribution explanations based on causal reasoning, construct a comprehensive evaluation function to iteratively optimize model parameters, and generate feedback reports.

9. The deep learning-based network traffic anomaly detection system according to claim 8, characterized in that, The data acquisition module uses the PFRING framework to capture raw data packets; NetFlow logs are parsed via the IPFIX protocol; and application layer proxy logs are obtained via a RESTful API. The fusion module generates a unique session ID by cutting PCAP based on the source IP, destination IP, source port, destination port, transmission protocol, and timestamp. It then uses the session ID and timestamp to match and supplement the session duration and byte rate of the NetFlow logs, as well as the URL and DNS domain name of the application layer logs, and removes data that fails to associate. The preprocessing module performs noise filtering based on traffic type weights; missing value imputation uses an improved KNN algorithm; and outlier sample synthesis is achieved through temporal gradient enhancement SMOTE. In the feature engineering module, statistical features are standardized using Z-score to form a 16-dimensional vector. Temporal features are extracted using LSTM and output as a 64-dimensional temporal vector. The semantic feature and protocol weight matrix are combined to enhance the weight of high-risk protocols and output a 32-dimensional vector; after improving mutual information entropy filtering, a 48-dimensional optimized feature set is output. In the few-shot learning module, the improved MAML algorithm optimizes the meta-parameters through gradient descent; The improved DQN algorithm filters samples based on a reward function to obtain labeled samples and unlabeled samples with high information value; The classification and detection module uses the Softmax classifier for classification, combines DoWhy to generate a causal explanation report and performs branch processing, and sets an unknown anomaly detection threshold to determine unknown anomalies. The adaptive optimization module monitors distribution drift through KL divergence; the incremental SVM updates parameters with a sliding window, and the parameters are fed back to the feature engineering module and the classification and detection module. In the explanation feedback module, causal reasoning generates an attribution report based on counterfactual analysis; the comprehensive evaluation function integrates accuracy, false alarm rate and comprehensive evaluation value, and optimizes parameters to feed back to the small sample learning module and the adaptive optimization module.

10. A computer device, comprising a memory, a processor, and a network interface, characterized in that, The memory, processor, and network interface are interconnected via a system bus. The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps in the deep learning-based network traffic anomaly detection method as described in any one of claims 1 to 7.

Images