Network abnormal traffic detection method and system based on federated learning and drift detection

Abstract

The application discloses a network abnormal traffic detection method and system based on federated learning and drift detection. Through data preprocessing and spatio-temporal feature extraction on multiple edge nodes, feature data is generated. Each edge node independently detects whether the traffic has concept drift. If drift is detected, incremental learning is started and only the changed data is updated. The updated local model parameters are uploaded to the central server, aggregated and a global model is generated. After evaluation, it is distributed to the edge nodes for real-time traffic detection. This method combines federated learning and drift detection technology, not only protects data privacy, but also improves the accuracy and real-time performance of network traffic detection, suitable for traffic monitoring and anomaly detection in large-scale distributed networks. Graph neural networks and bidirectional long short-term memory networks are used to improve feature extraction and model training accuracy, ensuring stability and reliability in complex network environments.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method and system for detecting abnormal network traffic based on federated learning and drift detection. Background Art

[0002] Anomaly detection is crucial in many fields, including network security. Its purpose is to accurately identify abnormal data from a large amount of normal data, providing effective early warning of potential risks. In the context of network traffic analysis, its primary task is to monitor and analyze data packets in the network in real time to identify potential security threats. With the rapid development of information technology, network traffic has increased dramatically, and the network environment has become increasingly complex. Traditional anomaly detection methods have exposed many problems in responding to this change:

[0003] 1) Computing Resources and Privacy: When processing large-scale distributed data, traditional methods often face the dilemma of insufficient computing resources and the risk of privacy leakage. This is because traditional methods may require centralized processing of large amounts of raw data, which not only places extremely high demands on computing equipment but also makes it difficult to ensure data privacy and security.

[0004] 2) Concept Drift Challenge: Normal patterns of network traffic change over time, a phenomenon known as "concept drift." This makes existing models unable to effectively adapt to new traffic characteristics, resulting in reduced detection performance and difficulty in accurately identifying newly emerging abnormal traffic.

[0005] In response to the shortcomings of the above traditional methods, relevant patents have been used to explore improvements:

[0006] Patent CN116016110A addresses the security of power networks and the shortcomings of existing federated learning methods in power systems, proposing a detection method based on hierarchical federated learning. This method involves preprocessing data on the terminal before participating in hierarchical federated learning. This involves updating and summarizing model weights between the cloud and the terminal, and between the edge and the terminal. After feature extraction, a softmax classifier is used to identify anomalies. This enables model training using multi-terminal data while protecting privacy. The hierarchical model adapts to the massive access of terminals and the power system, reducing communication costs.

[0007] Patent CN118764439A addresses the issue of data drift in network traffic classification, leading to model performance degradation, by proposing an online drift detection and adaptive classification method. This method first constructs the original model and performs data and feature engineering, memory partitioning, and training. Drift detection is then performed. Once drift is detected, new data is collected, sampled from the old data, and merged with the new data. This data is then used to update the model. Fine-tuning and improvement techniques are employed to achieve real-time monitoring and adaptation to new data distributions, optimizing data processing and storage.

[0008] However, existing technologies still have room for improvement. This invention aims to effectively address the computational and privacy bottlenecks of traditional methods when processing large-scale distributed data by combining federated learning and real-time drift detection. It also addresses concept drift in network traffic, ensuring that anomaly detection systems can continuously adapt and maintain efficient operation in dynamic network environments. Summary of the Invention

[0009] The present invention provides a method for detecting abnormal network traffic based on federated learning and drift detection, which solves the computational and privacy bottlenecks of traditional methods when processing large-scale distributed data, and can cope with concept drift in network traffic.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a method for detecting abnormal network traffic based on federated learning and drift detection, comprising the following steps:

[0011] Step 1: Preprocess traffic data on multiple edge nodes, extract spatiotemporal features, and generate feature data;

[0012] Step 2: Each edge node trains an anomaly detection model based on feature data and updates the parameters of the local model;

[0013] Step 3: Detect network traffic based on the local model to determine whether concept drift occurs;

[0014] Step 4: When concept drift occurs, start incremental learning and only update the changed data;

[0015] Step 5: Upload the local model parameters to the central server, aggregate the model parameters of each edge node, and generate a global model;

[0016] Step 6: Evaluate the updated global model and send it to each edge node for real-time anomaly detection.

[0017] In step 1, the specific operations of pre-processing the traffic data include:

[0018] Get the original network traffic dataset, which contains the following features: i =[f1,f2,…,f k ], each feature includes the source IP address IP src , target IP address dst , protocol type Proto, traffic size Size, etc.;

[0019] Clean the data and remove outliers x outlier And fill the missing values ​​x miss , the processing method is:

[0020] a) Use triple standard deviation method to eliminate outliers:

[0021] x outlier >μ+3σ or x outlier <μ-3σ

[0022] b) Perform mean interpolation on missing values:

[0023]

[0024] Normalize the data to the [0,1] interval using the formula:

[0025]

[0026] Divide the cleaned data into training set, validation set and test set according to the proportion:

[0027] μ={D train ,D val ,D test}

[0028] Among them, the training set accounts for 70%, the validation set accounts for 15%, and the test set accounts for 15%.

[0029] In step 1, the specific steps of extracting spatiotemporal features and generating feature data are as follows:

[0030] Construct a graph structure of network traffic, which is defined as follows:

[0031] a) Node set V = {v1, v2, ..., v n}, each node represents a flow record;

[0032] b) Edge set E = {e ij}, each edge e ij Represents node v i and v j The correlation between them, such as the time dependence or spatial dependence of traffic;

[0033] c) Adjacency Matrix

[0034]

[0035] Initialize node feature matrix Each feature dimension x i Including timestamp, traffic size, etc.

[0036] The feature aggregation function of the graph neural network (GNN) is defined as follows:

[0037]

[0038] in, represents the features of node i in layer l, and N(i) represents the set of neighbor nodes of node i.

[0039] The feature aggregation function uses the maximum aggregation, which is defined as follows:

[0040]

[0041] Graph convolution update formula:

[0042] H (l+1) =σ(AH (l) W (l) )

[0043] Among them, H (l) Represents the node feature matrix of the lth layer. Initially, H (0) =X nodes , W (l) is the weight matrix of the lth layer, W (0) It is obtained by random initialization at the beginning of training, A is the adjacency matrix, and σ is the activation function.

[0044] After multiple layers of graph convolution operations, the final low-dimensional feature representation is generated:

[0045]

[0046] In step 2, BiLSTM is used to train an anomaly detection model based on feature data. The specific steps are as follows:

[0047] Input format: H output Change to the input format X that is compatible with the BiLSTM model input , this conversion process usually involves mapping the features into a format compatible with the LSTM model input, which is the linear layer in the code:

[0048] X input =Linear(H output )

[0049] Bidirectional LSTM: Use bidirectional LSTM to input By processing, the bidirectional LSTM can capture the contextual information before and after each time step, thereby better learning the temporal features in the sequence. The calculation formula is:

[0050] LSTM out (h n ,c n )=BiLSTM(X input )

[0051] Attention mechanism: To further strengthen the model's focus on important features, the attention mechanism weights the features of the LSTM output by calculating the attention weights at each time step, ultimately generating a weighted sum representation. Specifically, the output of the attention layer is calculated using the following formula:

[0052]

[0053] Where T is the sequence length, is the LSTM output at the i-th time step, α i is the calculated attention weight.

[0054] In step 3, the network traffic is detected based on the local model to determine whether concept drift occurs, which is achieved by the following steps:

[0055] Model accuracy evaluation: Before performing concept drift detection, it is necessary to evaluate the accuracy of the current model on historical traffic data. The formula is as follows:

[0056]

[0057] Among them, y i is the model's predicted output for the i-th sample at time t, y i is the true label of the i-th sample, n t is the total number of samples at time t, is an indicator function, which takes the value 1 when the prediction is correct and 0 otherwise.

[0058] Concept drift detection: The CUSUM (Cumulative Sum Control Chart) method detects concept drift by monitoring the cumulative change in model accuracy. The steps are as follows:

[0059] a) Calculate the mean of historical accuracy: Where T is the total number of historical samples, Accuracy t is the accuracy of the t-th evaluation.

[0060] b) Calculate the deviation at the current moment: δ t =Accuracy t -μ.

[0061] c) Calculate the cumulative amount: S t =max(0,S t-1 +δ t ), the cumulative amount is updated using the "maximum value" operation. When the resulting deviation is negative (the model accuracy is not improved), the cumulative amount is reset to zero and remains stable.

[0062] d) Drift detection and judgment: If the current cumulative amount S tIf the threshold θ is exceeded, it is determined that concept drift has occurred. Otherwise, continue to monitor and update the cumulative amount. The judgment formula is as follows:

[0063]

[0064] In step 4, when concept drift occurs, incremental learning is started using the Progressive Neural Network (PNN) algorithm to update only the changed data. The specific operations are as follows:

[0065] The Embedding layer generates an embedding vector for the input data of the new task:

[0066] e i =x i W E

[0067] Product layer, calculates the interaction term between new task features and old task features:

[0068] p ij =e i ⊙e j

[0069] Sum layer aggregates interactive features and generates new features:

[0070]

[0071] Finally, the PNN output layer combines the new and old task features to perform classification prediction.

[0072] In step 5, the model parameters of each edge node are aggregated to generate a global model. The specific operations include:

[0073] In each edge node k, use the local dataset D k ={x k,1 ,…,x k,n}, train the model.

[0074] a) Each node is based on the local loss function L k (w k ) Calculate the gradient to update the model parameters w k

[0075]

[0076] Where η is the learning rate, is the gradient of the loss function.

[0077] b) After training is completed, the local model parameters w k And the corresponding data volume n k Upload to the central server. The uploaded parameter set is:

[0078] P k ={w k ,n k}

[0079] The model parameters uploaded by each node are calculated according to the data volume n k Weighted aggregation, calculate the global model parameter w g , the formula is as follows:

[0080]

[0081] Where K represents the number of edge nodes, w g is the global model parameter, w k is the local model parameter of the kth edge node, n k is the data volume of the kth edge node, It is the sum of the data volume of all edge nodes.

[0082] In step 6, the global model is evaluated and sent to edge nodes to achieve real-time anomaly detection. The specific steps are as follows:

[0083] In the model evaluation phase, the test dataset D test Evaluate the global model w g The performance of the model is calculated, including accuracy, precision, recall, F1 score, and AUC value, and compared with the previous model w g-1 Compare.

[0084] A / B test was performed using t-test to compare the performance of the two models. g and w g-1 The test results of the two groups of performance indicators are calculated, and the t value and p value are calculated to determine the significance of the performance improvement. If the t test results show that the performance improvement is statistically significant, the current global model w is confirmed. g It can be sent; otherwise, the current global model will not be updated.

[0085] In the model distribution phase, the global model w' is sent using an encrypted communication protocol. g It is sent to each edge node to ensure that the data is not eavesdropped or tampered with during transmission.

[0086] Each edge node k receives the updated global model parameter w' g After that, load it into this model to obtain the latest detection model:

[0087] w g =w′ g .

[0088] The present invention also provides a network abnormal traffic detection system based on federated learning and drift detection, comprising multiple edge nodes and a central server, and the system includes the following modules:

[0089] Data preprocessing module: This module is configured at each edge node and is responsible for data preprocessing. It extracts spatiotemporal features through graph neural networks and generates effective features for anomaly detection.

[0090] Local training module: This module is configured on each edge node and uses the BiLSTM model to train the extracted feature data to generate a local anomaly detection model.

[0091] Drift detection module: This module is deployed on edge nodes and uses CUSUM to monitor the performance of the global model, triggering concept drift detection and ensuring that the model adapts to new attack patterns.

[0092] Incremental learning module: Configured on edge nodes, it initiates incremental learning when concept drift occurs, uses PNN to train the changed data, and avoids retraining the BiLSTM.

[0093] Federated learning aggregation module: configured on the central server, it uses the FedAvg method to receive and aggregate local model parameters uploaded by each edge node;

[0094] Model evaluation and deployment module: Configured on the central server, it uses t-tests to perform A / B testing, evaluates the global model, and decides whether to update and send it to each edge node for real-time detection.

[0095] Compared with existing technologies, this invention offers significant advantages: By combining federated learning and drift detection techniques, it balances privacy protection and computational efficiency analysis in distributed data processing, resolving the challenges of limited computing resources and privacy leakage faced by traditional methods in large-scale distributed scenarios. Furthermore, by introducing a real-time drift detection mechanism, this invention can promptly capture dynamic changes in network traffic patterns and rapidly update the model through incremental learning, thereby ensuring the continued adaptability and efficiency of the anomaly detection system in complex network environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0097] In the attached figure:

[0098] Figure 1 It is a schematic flow chart of the present invention;

[0099] Figure 2 It is a timing diagram of the present invention;

[0100] Figure 3This is the architecture diagram of the main detection model BiLSTM model. DETAILED DESCRIPTION

[0101] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0102] Example: Figure 1-Figure 3 As shown, a network abnormal traffic detection method based on federated learning and drift detection includes the following steps:

[0103] Step 1: Traffic capture and data preprocessing, extracting spatiotemporal features, and generating feature data.

[0104] In the specific implementation of the present invention, the network traffic data is captured using the Wireshark tool. Wireshark is a commonly used network protocol analysis tool that can capture all data packets in a network interface and store them as pacp files.

[0105] The raw traffic data before capture is directly transmitted by the network interface without any processing; the captured traffic data is formatted and stored in a pacp file, which contains detailed information about the data packet, such as timestamp, source IP address, destination IP address, etc. The following are the formats of the traffic data before and after capture:

[0106] Table 1 Traffic data before capture

[0107] Field Name Sample Data describe Raw bitstream 1010011101001010… Bitstream received directly by the network interface Packet size 1500 The size of the packet in bytes Transport Protocol 6 Transport protocol identifier, for example, 6 for TCP protocol

[0108] Table 2 Traffic data after capture

[0109] Field Name Sample Data describe Timestamp 45292.50001 The time of packet capture, accurate to milliseconds Source IP address 192.168.0.1 The IP address of the sender of the data packet Destination IP address 192.168.0.2 The IP address of the receiving end of the data packet Protocol Type TCP Transport protocol type Data size 1200 The size of the packet in bytes Port number 80 The port number used for data packet transmission

[0110] The captured pacp file is processed by Wireshark, and the data packets are parsed and exported as csv files for further data preprocessing and model input.

[0111] Data preprocessing mainly includes missing value interpolation, outlier removal and normalization. The specific steps are as follows:

[0112] Interpolate missing values ​​with their mean:

[0113]

[0114] For the flow size field, the triple standard deviation method is used to eliminate outliers:

[0115] x outlier >μ+3σ or x outlier <μ-3σ

[0116] Normalize the data to the [0,1] interval using the formula:

[0117]

[0118] Divide the cleaned data into training set, validation set and test set according to the proportion:

[0119] D={D train ,D val ,D test}

[0120] Among them, the training set accounts for 70%, the validation set accounts for 15%, and the test set accounts for 15%.

[0121] Network traffic data is represented in the form of a graph, where nodes represent each traffic record and edges represent the temporal dependencies between nodes. The specific steps are as follows:

[0122] Node features: Each node feature vector contains the following five elements: protocol type (3-dimensional vector), traffic size (1-dimensional scalar), timestamp (1-dimensional scalar), with a total dimension of 5.

[0123] Adjacency matrix generation: Construct the adjacency matrix A according to the following rules:

[0124]

[0125] The graph convolutional network (GCN) model from the graph neural network (GNN) family is used to extract the spatiotemporal features of network traffic. The graph convolution update formula is:

[0126] H (l+1) =σ(AH (l) W (l) )

[0127] Among them, H (l) Represents the node feature matrix of the lth layer. Initially, H (0) =X nodes , W (l) is the weight matrix of the lth layer, W (0) It is obtained by random initialization at the beginning of training, A is the adjacency matrix, and σ is the activation function.

[0128] After multiple layers of graph convolution operations, the final low-dimensional feature representation is generated:

[0129]

[0130] The model structure consists of three layers of graph convolution, and each layer is designed as follows:

[0131] The first layer of graph convolution: the input dimension is N*5, the output dimension is N*64, the weight matrix dimension is 5*64, and the activation function is ReLU, which is used to extract the basic spatiotemporal features of the nodes.

[0132] The second layer of graph convolution: the input dimension is N*64, the output dimension is N*128, the weight matrix dimension is 64*128, and the activation function is ReLU, which is used to capture more complex associations between nodes.

[0133] The third graph convolution: the input dimension is N*128, the dimension is N*128, the weight matrix dimension is 64*128, and the activation function is ReLU, which reduces the high-dimensional features to 64 dimensions for subsequent input.

[0134] Step 2: Train the network abnormal traffic detection model.

[0135] The feature matrix N*64 output by the GNN needs to be converted to the input format N*128 supported by the BiLSTM model. The mapping process is as follows:

[0136] Linear Mapping: The linear mapping layer converts the input feature tensor from N*64 to N*128, mapping it through the weight matrix (shape 64*128) and the bias vector (shape 128). The output tensor shape is N*128.

[0137] Bidirectional LSTM layer: The bidirectional LSTM layer receives a shape of N*seq len *128 input tensor, and output shape is N*seq len *512 bidirectional LSTM output tensor. The output dimension of bidirectional LSTM is hidden dim *2, where hidden dim =256, that is, the output of each time step is 512.

[0138] Classification layer: The classification layer receives the attention-weighted output of shape N*512 and maps it to an output tensor of shape N*2 through a linear layer (weight matrix shape 512*2), representing the binary classification result of each sample. The output tensor has a shape of N*2, where each row contains the predicted scores of two categories.

[0139] The final output of the model is a tensor of shape N*2, where N is the number of samples and 2 represents the predicted probability distribution of the two classes (normal / abnormal).

[0140] Step 3: Detect network traffic based on the local model to determine whether concept drift occurs.

[0141] The distribution of network traffic may change over time, leading to a decrease in the model's predictive performance due to new attack patterns or changes in user behavior patterns. The goal of concept drift detection is to monitor model performance and determine whether incremental learning is needed.

[0142] The performance of the global model is evaluated on the test dataset, and the following metrics are calculated:

[0143]

[0144] Among them, y i is the model's predicted output for the i-th sample at time t, y i is the true label of the i-th sample, n t is the total number of samples at time t, is an indicator function, which is 1 when the prediction is correct and 0 otherwise.

[0145] Drift detection algorithm, using the CUSUM (Cumulative Sum Control Chart) method to monitor changes in model performance in real time:

[0146] a) Calculate the mean of historical accuracy: Where T is the total number of historical samples, Accuracy t is the accuracy of the t-th evaluation.

[0147] b) Calculate the deviation at the current moment: δ t =Accuracy t -μ.

[0148] c) Calculate the cumulative amount: S t =max(0,S t-1 +δ t ), the cumulative amount is updated using the "maximum value" operation. When the resulting deviation is negative (the model accuracy is not improved), the cumulative amount is reset to zero and remains stable.

[0149] d) Drift detection and judgment: If the current cumulative amount S t If the threshold θ is exceeded, it is determined that concept drift has occurred. Otherwise, continue to monitor and update the cumulative amount. The judgment formula is as follows:

[0150]

[0151] The result of drift detection directly determines whether it is necessary to start the incremental learning process in step five.

[0152] Step 4: Start incremental learning and update only the changed data.

[0153] When concept drift is detected, a progressive neural network (PNN) is used to update the model. The design of PNN allows the model to quickly learn the characteristics of new data while retaining existing knowledge. The model structure is as follows:

[0154] Input layer: the output tensor of the BiLSTM classification layer, with a dimension of N*2;

[0155] Hidden layer 1: input dimension is N*2, output dimension is N*128, activation function is ReLU, and weight matrix dimension is 6*128.

[0156] Hidden layer 2: The input dimension is N*128, the output dimension is N*64, the activation function is ReLU, and the weight matrix dimension is 128*64.

[0157] Output layer: input dimension is N*64, output dimension is N*2, and activation function is Softmax.

[0158] Training process: Lock the original parameters of the BiLSTM model and train only the newly added PNN module. The optimizer is Adam, the learning rate is 0.001, and the loss function is cross entropy loss.

[0159] Step 5: Aggregate the model parameters of each edge node to generate a global model.

[0160] In the federated learning framework of this invention, each edge node independently trains a local BiLSTM model and, based on the training configuration in step 2, incrementally learns real-time traffic data to update the parameters of the PNN module. After completing training, the edge node uploads the BiLSTM model parameters and the PNN incremental learning parameters to the central server. The local model contains the following parameters:

[0161] BiLSTM model parameters:

[0162] a) Bidirectional LSTM layer: The feature matrix dimension is 256*1*28, the hidden layer dimension is 256, and the output dimension is 256*1*512. There are two LSTM layers, and the parameters of each layer are uploaded independently.

[0163] b) Attention mechanism layer: The input dimension is 256*1*512, the weight matrix dimension is 512*1, and the weighted sum dimension is 256*512

[0164] c) Classification layer: The input dimension is 256*512, the dimension of the weight matrix is ​​512*2, and the output dimension is 256*2.

[0165] PNN model parameters:

[0166] a) Input layer: The weight matrix dimension is 2*128, and the bias vector dimension is 128.

[0167] b) Hidden layer 1: The weight matrix dimension is 127*128, and the bias vector dimension is 128.

[0168] c) Hidden layer 2: The weight matrix dimension is 128*64, and the bias vector dimension is 64.

[0169] d) Output layer: The weight matrix dimension is 64*2, and the bias vector dimension is 2.

[0170] The central server performs weighted aggregation on the model parameters uploaded by the edge nodes using the FedAvg (federal average) algorithm. The specific formula is as follows:

[0171]

[0172] Where K represents the number of edge nodes, w g is the global model parameter, w k is the local model parameter of the kth edge node, n k is the data volume of the kth edge node, It is the sum of the data volume of all nodes.

[0173] Step 6: The updated global model is evaluated and sent to each edge node for real-time anomaly detection.

[0174] Before the updated global model is officially distributed to each edge node, it needs to undergo rigorous performance evaluation to ensure that it can maintain efficient operation in a dynamic network environment.

[0175] The evaluation focuses on the following performance indicators:

[0176] a) Accuracy: measures the overall prediction accuracy of the model.

[0177] b) Precision: evaluates the model's ability to identify positive samples.

[0178] c) Recall: Evaluates the model's ability to cover positive samples.

[0179] d) F1-Score: A comprehensive measure of precision and recall.

[0180] e)AUC (Area Under the Curve): measures the model's ability to distinguish between positive and negative samples.

[0181] A / B testing is an effective method to compare the performance of a new model with that of an old model. In this invention, A / B testing is performed by statistically analyzing the prediction results of the same test data set to determine whether the new model is significantly better than the old model in terms of performance.

[0182] Hypothesis Testing:

[0183] a) Null hypothesis (H0): There is no significant difference in performance indicators between Group A and Group B;

[0184] b) Alternative hypothesis (H1): The performance indicators of group B are significantly better than those of group A.

[0185] Use the t-test to perform A / B testing. The formula is as follows:

[0186] Assume that the means of sample A and sample B are and The standard deviations are s A and s B , the sample sizes are n A and n B , the calculation formula of the t-test statistic is:

[0187]

[0188] Degrees of freedom calculation formula:

[0189]

[0190] Calculate t-value and p-value:

[0191] Substitute the t-value and degrees of freedom (df) calculated from the above formula into the t-distribution table or use statistical software to calculate the corresponding p-value. If p < 0.05 (significance level of 5%), the null hypothesis H0 can be rejected, indicating that the performance of the new model has been significantly improved.

[0192] Update the performance-enhanced model to the edge nodes. After receiving the model, each edge node loads it into the local detection system to detect abnormal network traffic. The specific steps are as follows:

[0193] Data preprocessing: Clean, encode, and normalize the real-time traffic data according to the method in step 1.

[0194] Feature extraction: The processed features are input into the GNN model to generate a node feature matrix.

[0195] Classification detection: The feature matrix is ​​passed to the BiLSTM model and the PNN model in sequence, and the classification results (normal / abnormal) are output.

[0196] After completing real-time detection, edge nodes locally determine whether to initiate incremental learning to quickly adapt to the new data distribution. Once incremental learning is complete, the edge nodes upload the updated model parameters to the central server. After receiving the parameters uploaded by each edge node, the central server generates a global model using an aggregation algorithm to ensure that the model can adapt to the dynamic network environment of the entire system.

[0197] Step 5 distributes the global model from the central server to the edge nodes, while step 6 completes the upload of incremental learning parameters and global model aggregation from the edge nodes to the central server. Through this two-way interactive mechanism, the central server and edge nodes form a dynamic update loop from global to local and then from local to global. This closed-loop mechanism continuously optimizes system performance, enabling the model to rapidly respond to local environmental changes while maintaining global collaborative adaptability, thereby achieving efficient collaboration and global optimization in a dynamic network environment.

[0198] The present invention also provides a network abnormal traffic detection system based on federated learning and drift detection, comprising multiple edge nodes and a central server, and the system includes the following modules:

[0199] Data preprocessing module: This module is configured at each edge node and is responsible for data preprocessing. It extracts spatiotemporal features through graph neural networks and generates effective features for anomaly detection.

[0200] Local training module: This module is configured on each edge node and uses the BiLSTM model to train the extracted feature data to generate a local anomaly detection model.

[0201] Drift detection module: This module is deployed on edge nodes and uses CUSUM to monitor the performance of the global model, triggering concept drift detection and ensuring that the model adapts to new attack patterns.

[0202] Incremental learning module: Configured on edge nodes, it initiates incremental learning when concept drift occurs, uses PNN to train the changed data, and avoids retraining the BiLSTM.

[0203] Federated learning aggregation module: configured on the central server, it uses the FedAvg method to receive and aggregate local model parameters uploaded by each edge node;

[0204] Model evaluation and deployment module: Configured on the central server, it uses t-tests to perform A / B testing, evaluates the global model, and decides whether to update and send it to each edge node for real-time detection.

[0205] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A network abnormal traffic detection method based on federated learning and drift detection, characterized in that: The following steps are involved: S1. Preprocess traffic data on multiple edge nodes, extract spatiotemporal features, and generate feature data. S2. Each edge node trains an anomaly detection model based on feature data and updates the parameters of the local model; S3, detect network traffic based on the local model to determine whether concept drift occurs; S4. When concept drift occurs, start incremental learning and only update the changed data; S5. Upload the local model parameters to the central server, aggregate the model parameters of each edge node, and generate a global model; S6. Evaluate the updated global model and send it to each edge node for real-time anomaly detection. The graph neural network in step S1 is used to extract spatiotemporal features. The specific operations include: Construction of GNN: Each network traffic record consists of a set of features Description, each feature describes a different dimension of traffic data; the nodes of the entire network constitute a feature matrix : ； Extracting spatiotemporal features: The core of GNN is the graph convolution layer, which uses the adjacency matrix A and the node feature matrix To extract spatiotemporal features; specifically, the update formula of the graph convolution layer is: ； in, Indicates the The node feature matrix of the layer, initially = , It is The weight matrix of the layer, It is obtained by random initialization at the beginning of training, A is the adjacency matrix, is the activation function; High-dimensional feature mapping is low-dimensional vector: After multi-layer graph convolution operation, the node features will be mapped to low-dimensional space, and the obtained node features is the low-dimensional feature representation of each traffic record: ; Among them, each It is a traffic record The low-dimensional feature vector captures the spatiotemporal information of the traffic record; The anomaly detection model in step S2 uses BiLSTM to train the extracted features. The specific operations are as follows: Input format: Convert to an input format compatible with the BiLSTM model , that is, mapping the features to the linear layer: ； Bidirectional LSTM: Use bidirectional LSTM to For processing, the bidirectional LSTM can capture the contextual information before and after each time step, thereby better learning the temporal features in the sequence; the calculation formula is: ； Attention mechanism: The attention mechanism weights the features of the LSTM output by calculating the attention weights at each time step, and finally generates a weighted sum representation; specifically, the output of the attention layer is calculated by the following formula: ； Where T is the sequence length, is the LSTM output at the i-th time step, is the calculated attention weight; Model training: Finally, classification is performed through a linear classifier; the classification layer receives the feature representation weighted by the attention mechanism and generates the final prediction result; the calculation formula is: ； in, Is the final output of the model, indicating the probability that each sample belongs to each category; The drift detection method in step S3 is implemented by the following steps: Model accuracy evaluation: Before performing concept drift detection, it is necessary to evaluate the accuracy of the current model on historical traffic data. The formula is as follows: ； in, The model at time For the first The predicted output of the sample, It is The true labels of samples, It's time The total number of samples, is an indicator function, which is 1 when the prediction is correct and 0 otherwise; Concept drift detection: The CUSUM method detects concept drift by monitoring the cumulative changes in model accuracy. The steps are as follows: Compute the mean of historical accuracy: , where T is the total number of historical samples, is the accuracy of the t-th evaluation; Calculate the deviation at the current moment: ; Calculate the cumulative amount: , the cumulative amount is updated using the "maximum value" operation. When the result deviation is negative, the cumulative amount is reset to zero to maintain stability; Drift detection and judgment: The judgment formula is as follows: ； If the cumulative amount at the current moment Exceeding the set threshold , it is determined that concept drift occurs; otherwise, continue to monitor and update the accumulation.

2. The method for detecting abnormal network traffic based on federated learning and drift detection according to claim 1, characterized in that: The data preprocessing in step S1 includes the following operations: Get the raw network traffic dataset , original network traffic dataset Contains information such as network traffic characteristics and type tags for marking network traffic; right Perform data cleaning, fill missing values, remove outliers and noise, and generate a cleaned data set ; Will The datasets are divided into 70% training set, 15% validation set and 15% test set, and the minimum-maximum normalization is performed, and they are named 、 and .

3. The method for detecting abnormal network traffic based on federated learning and drift detection according to claim 1, characterized in that: The incremental learning method in step S4 is a progressive neural network algorithm, and the specific formula is as follows: ； in, It's a new task The corresponding network parameters, It's an old task The network parameters, It is a module added for new tasks. Represents the expansion of the network structure.

4. The method for detecting abnormal network traffic based on federated learning and drift detection according to claim 1, characterized in that: The model aggregation method in step S5 is the FedAvg method, and the specific formula is as follows: ； in, represents the number of edge nodes, are global model parameters, It is The local model parameters of the edge nodes, It is The amount of data per edge node, It is the sum of the data volume of all nodes.

5. The method for detecting abnormal network traffic based on federated learning and drift detection according to claim 1, characterized in that: The real-time anomaly detection method in step S6 is as follows: The updated global model is evaluated through A / B testing to compare the performance of the new and old models in terms of accuracy, precision, recall, and AUC values. The test results are analyzed using t-tests to ensure that the new model is superior to the old model. The evaluated model is sent to edge nodes for real-time traffic anomaly detection, and the edge nodes use the global model for anomaly detection. The entire process continuously optimizes the model through incremental learning and feedback mechanisms.

6. A network abnormal traffic detection system based on federated learning and drift detection, applied to the method of claim 1, characterized in that: The system includes multiple edge nodes and a central server, and the following modules: Data preprocessing module: This module is configured at each edge node and is responsible for data preprocessing. It extracts spatiotemporal features through graph neural networks and generates effective features for anomaly detection. Local training module: This module is configured on each edge node and uses the BiLSTM model to train the extracted feature data to generate a local anomaly detection model. Drift detection module: This module is deployed on edge nodes and uses CUSUM to monitor the performance of the global model, triggering concept drift detection and ensuring that the model adapts to new attack patterns. Incremental learning module: Configured on edge nodes, it initiates incremental learning when concept drift occurs, uses PNN to train the changed data, and avoids retraining the BiLSTM. Federated learning aggregation module: configured on the central server, it uses the FedAvg method to receive and aggregate local model parameters uploaded by each edge node; Model evaluation and deployment module: Configured on the central server, it uses t-tests to perform A / B testing, evaluates the global model, and decides whether to update and send it to each edge node for real-time detection.

Images