Malicious traffic detection model robustness enhancement method and system based on reinforcement learning and incremental learning

Abstract

The invention belongs to the technical field of malicious traffic detection and network security, and provides a malicious traffic detection model robustness enhancement method and system based on reinforcement learning and incremental learning. The method mainly solves the problems that an existing malicious traffic detection model is insufficient in robustness, poor in confrontation sample compliance, prone to catastrophic forgetting in a dynamic threat environment, weak in adaptability and the like. According to the main scheme, the method comprises the following steps: constructing state action space joint modeling, a three-order composite reward function and a compliance action mask mechanism through reinforcement learning in combination with a Transform strategy network, and generating an adversarial sample with high concealment and high compliance; an MMD concept drift detection mechanism is used for sensing attack mode evolution, a hierarchical EWC parameter protection strategy is combined, drift sensing-parameter protection collaborative incremental learning is achieved, and self-adaptive updating of a detection model is completed; an intelligent malicious traffic defense framework with high adversarial robustness and continuous environmental adaptability is constructed through the process, and meanwhile, the model performance can be verified through a related mechanism, so that the defense effect is ensured.

Description

Technical Field

[0001] This invention belongs to the field of malicious traffic detection and network security technology, specifically relating to a method and system for enhancing the robustness of a malicious traffic detection model based on reinforcement learning and incremental learning. Background Technology

[0002] With the acceleration of global digitalization, the scale and complexity of cyberspace are growing exponentially, making malicious traffic detection systems a core line of defense for protecting critical information infrastructure and data assets. However, existing malicious traffic detection systems are showing increasing limitations in dealing with increasingly intelligent and diverse advanced persistent threats (APTs), especially when facing adaptive attacks driven by cutting-edge technologies such as generative adversarial networks (GANs), where their defensive capabilities are particularly inadequate.

[0003] Attackers successfully carried out an "AI bypass" attack by simulating the encrypted traffic characteristics of legitimate collaborative software, which further confirms that the robustness of traditional detection models faces a severe test when faced with highly obfuscated malicious traffic.

[0004] Currently, research in the field of malicious traffic detection is mainly divided into two categories:

[0005] One type is traditional machine learning and deep learning methods that improve the performance of the detection model itself, such as SVM, KNN, CNN, LSTM, etc. These methods are essentially static defense models, with parameters fixed after training. When faced with continuously evolving and mutating malicious attacks, they lack generalization ability and robustness, and are easily affected by feature space perturbation attacks and fail.

[0006] Another type is adversarial learning methods that explore dynamic game theory between attack and defense, represented by GANs. However, these methods often suffer from pattern collapse and lack of compliance when generating adversarial examples. The generated examples may violate network protocol specifications and cannot be effectively transmitted and attacked in real network environments, thus limiting their practicality.

[0007] To address the "catastrophic forgetting" problem in updating defense models under dynamic threat environments, incremental learning techniques are increasingly being applied to the field of malicious traffic detection. Existing incremental learning methods mainly include basic incremental learning, replay-based methods, regularization-based methods, and knowledge distillation-based methods. However, these methods still have significant shortcomings when applied to malicious traffic detection scenarios: they fail to resolve the contradiction between "preserving historical knowledge and rapidly adapting to new threats," either by over-protecting historical parameters leading to a lag in adaptation to new threats, or by over-updating leading to forgetting; drift detection is disconnected from incremental updates, making it unable to dynamically respond to the evolution of attack patterns; and they do not incorporate the protocol characteristics and attack patterns of malicious traffic, resulting in limited adaptability and generalization.

[0008] In the paper "IoT-Based Android Malware Detection Using Graph Neural Network With Adversarial Defense," the research utilizes graph neural networks to generate API graph embeddings based on centrality metrics, integrating permissions and intent for detection. However, this method does not address adversarial example generation and incremental learning for malicious traffic, failing to solve the model adaptability problem in dynamic environments. In the paper "Android Malware Detection Based on Interpretability," researchers utilize multilayer perceptrons and attention mechanisms to achieve interpretability detection. However, this method only targets malware, not malicious traffic detection, and cannot mitigate "catastrophic forgetting," resulting in insufficient robustness.

[0009] Furthermore, some studies have attempted to apply reinforcement learning to the generation of malicious traffic adversarial attacks, but most rely on auxiliary models such as GANs, failing to completely eliminate the risk of pattern collapse. Most methods generally lack protocol compliance constraints on generated samples, limiting their practicality. Most importantly, existing research focuses on the success rate of a single attack, ignoring the fact that in dynamic threat environments, the detection model itself is constantly learning and evolving. Therefore, it fails to address the "catastrophic forgetting" problem that may occur when defense models continuously learn new threats, i.e., the model forgets how to identify old attacks while learning new knowledge.

[0010] In summary, existing malicious traffic detection models suffer from insufficient robustness, poor compliance with adversarial examples, weak adaptability in dynamic environments, and susceptibility to "catastrophic amnesia," failing to meet the actual needs of current network security protection. Therefore, developing a robustness enhancement method and system for malicious traffic detection models that can effectively solve the above problems has become an urgent and critical issue in the field of network security. Summary of the Invention

[0011] To address the aforementioned problems in existing technologies, this invention aims to propose a robustness enhancement method and system for malicious traffic detection models based on reinforcement learning and incremental learning. This addresses core issues such as insufficient robustness, poor compliance of adversarial examples, susceptibility to "catastrophic forgetting" in dynamic threat environments, and weak adaptability of existing malicious traffic detection models.

[0012] This invention constructs an intelligent malicious traffic defense framework that combines high adversarial robustness with continuous environmental adaptability by integrating reinforcement learning and incremental learning techniques. Specific research content includes key technical challenges such as reinforcement learning adversarial example generation based on Transformer policy networks, modeling of three-order composite rewards and compliance action masking mechanisms, MMD concept drift detection, and hierarchical EWC parameter protection. The aim is to address the two core challenges faced by current malicious traffic detection models—insufficient robustness of static defense models and "catastrophic forgetting" in updating defense models under dynamic environments—by deeply studying the weaknesses and limitations of existing malicious traffic detection systems and proposing a series of innovative theoretical methods and technical systems. This will improve the anti-attack capability, dynamic adaptability, and practicality of malicious traffic detection models, providing efficient and reliable technical support for network security protection.

[0013] To achieve the above objectives, the present invention adopts the following technical solution:

[0014] This invention provides a method for enhancing the robustness of a malicious traffic detection model based on reinforcement learning and incremental learning, comprising the following steps:

[0015] Step 1: Construct a malicious traffic training dataset and a test dataset, collect benign traffic samples and malicious traffic samples, perform data preprocessing, and extract historical attack features, context features and traffic statistical features of the traffic samples to provide data support for subsequent processes;

[0016] Step 2: Based on reinforcement learning technology and combined with Transformer policy network, construct a compliant adversarial example generation model. Through joint modeling of state-action space, third-order composite reward function and action masking mechanism, generate efficient adversarial examples with high concealment and strong compliance.

[0017] Step 3: Construct a concept drift detection mechanism based on MMD to perform drift detection on the adversarial examples generated in Step 2 and the malicious traffic samples collected in real time, quantify the degree of deviation between the distribution of current traffic data and historical training data, and start the incremental training process when the MMD value exceeds the preset threshold.

[0018] Step 4: Based on incremental learning technology and combined with the hierarchical EWC parameter protection strategy, the malicious traffic detection model is trained incrementally in a collaborative manner. By evaluating the importance of parameters, differentiated protection strengths are applied to different levels of the model to achieve the dual goals of maintaining historical attack knowledge stably and adapting to new threats.

[0019] Step 5: Use the malicious traffic detection model trained and updated in Step 4 to detect malicious traffic samples to be tested, output classification results, and feed the detection results back to the compliance adversarial sample generation model in Step 2 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness and adaptability of the model.

[0020] In the above scheme, the specific steps of step 1 are as follows:

[0021] Step 1.1: Collect samples. Collect benign traffic samples and malicious traffic samples from public network security datasets and real network environments to build an initial dataset with a balanced ratio of benign and malicious samples.

[0022] Step 1.2: Perform data preprocessing, including denoising, deduplication, and standardization of the collected traffic samples, removing invalid samples, extracting historical attack features, context features, and traffic statistics features of the samples, and converting the feature data into a format that the model can recognize.

[0023] Step 1.3: Divide the dataset. Divide the preprocessed dataset into a training set and a test set according to a preset ratio. The training set is used for the initial training of the model and subsequent incremental training, while the test set is used to verify the performance of the model.

[0024] In the above scheme, the specific steps of step 2 are as follows:

[0025] Step 2.1: Construct a joint model of state and action space, where state space s t This integrates historical attack characteristics, contextual characteristics, and traffic statistics. Historical attack characteristics include recent success rate, alarm type, and interception frequency; contextual characteristics include protocol type, source and destination information, and duration; and traffic statistics include packet size distribution, throughput fluctuation, and connection duration. Action space a t This includes traffic modification operations such as timing adjustments and packet size adjustments;

[0026] Step 2.2: Construct a Transformer policy network. Input the traffic feature sequence into a Transformer encoder consisting of a multi-layer multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures the correlation information of individual traffic features in different dimensions. The output features of each layer of the Transformer encoder are processed by residual connection and layer normalization. The Transformer encoder is used to perform deep modeling of high-dimensional, long-sequence traffic state features, capturing the complex temporal and spatial dependencies between data packets. The final output features of the encoder are input into a fully connected layer and mapped to a vector matching the action space dimension through an activation function to generate preliminary traffic modification action instructions.

[0027] Step 2.3: Implement an action masking mechanism. Based on mainstream network transmission protocol specifications such as TCP / UDP / IP, construct a network protocol compliance rule base and formulate illegal action judgment standards. Illegal action judgment standards include violations of packet size, tampering with key protocol fields, violations of timing adjustment, and illegal modification of source and destination information. Simultaneously, the state space s from Step 2.1 will be... t High-dimensional flow characteristics and action space a t The system associates traffic modification actions, clarifies the types of actions that can be executed and the scope of action modifications under different traffic states, and constructs a mapping rule base from state space to action space. At each time step t, a mask vector (Mask) is dynamically generated by combining the network protocol compliance rule base and the mapping rule base from state space to action space. t ,

[0028]

[0029] in The value is 0 or 1, where 0 indicates an illegal action that cannot be executed at time step t, and 1 indicates a legal action that can be executed at time step t. This is used to forcibly filter out illegal actions that violate network protocol specifications and ensure the compliance of action instructions.

[0030] Step 2.4: Design a third-order composite reward function, including core attack reward, stealth reward, and resource cost penalty. The third-order composite reward function is designed as follows:

[0031]

[0032] in The core attack reward is the success rate of generating adversarial examples to bypass the detection model. The reward is for concealment, specifically the similarity between the generated adversarial example and the original traffic sample. This is a resource cost penalty, manifested in the resource and time consumption of the agent when generating adversarial examples. , , The weights corresponding to these three factors are used to balance the composite reward function. This drives the agent to achieve the optimal balance between "attack effectiveness," "stealth and survival," and "resource cost," quantifying the benefit of each action taken by the agent.

[0033] Step 2.5: Generate and optimize adversarial examples. The agent combines the action modification instructions generated by the policy network in Step 2.2 and the action masking mechanism in Step 2.3 to obtain legal actions filtered by the mask. It modifies the features of the original malicious traffic sample to generate adversarial examples. The adversarial examples are input into the current malicious traffic detection model. The reward value is calculated based on the detection results. The Transformer policy network parameters are updated by backpropagation, and the quality of the adversarial examples is iteratively optimized.

[0034] In the above scheme, the specific steps of step 3 are as follows:

[0035] Step 3.1: Extract features. Extract the core features of the adversarial examples generated in Step 2 and the malicious traffic samples collected in real time, and align them with the features of the historical traffic samples used during model training.

[0036] Step 3.2: Calculate the MMD value using the Maximum Mean Difference (MMD) algorithm, according to the formula:

[0037]

[0038] Calculate the distributional differences between the current traffic sample characteristics and the historical traffic sample characteristics, and quantify the degree of deviation between the two, where: For sample set In the The mean of each feature dimension For sample set In the The mean of each feature dimension This represents the total number of feature dimensions.

[0039] Step 3.3: Perform drift detection and alarm. Preset MMD threshold. When the calculated MMD value exceeds the threshold, it is determined that concept drift has occurred, the drift alarm based on sliding window is activated, and the incremental training process in step 4 is triggered; if the threshold is not exceeded, the current detection model is maintained and traffic detection continues.

[0040] In the above scheme, the specific steps of step 4 are as follows:

[0041] Step 4.1: Evaluate the importance of parameters, consolidate the EWC algorithm with elastic weights, calculate the importance of each parameter in the model to the historical attack detection task using the Fisher information matrix, introduce prediction weights for the key parameters of each historical task, and correct the data for each task.

[0042] Step 4.2: Design a hierarchical parameter protection mechanism. Based on hierarchical importance, differentiated EWC regularization coefficients, i.e., protection weights, are set for the bottom feature extraction layer, the middle semantic understanding layer, and the top decision classification layer of the malicious traffic detection model. These coefficients directly determine the constraint strength of parameter updates during incremental training. A higher protection weight is given to the more general feature extraction layer, while the top decision layer is allowed a larger parameter adjustment space to adapt to new attack types. A hierarchical EWC regularization term is introduced into the loss function of incremental training. The total loss function is designed as follows:

[0043]

[0044] in The incremental task loss represents the model hierarchy. The mean of the Fisher information matrix for the parameters of this layer. For the current parameters in incremental training, The original parameters for historical tasks; protection weights are set through hierarchical configuration. This enables differentiated protection at different levels;

[0045] Step 4.3: Perform collaborative incremental training. Use the traffic samples with conceptual drift selected in Step 3 as incremental training data. Combine the hierarchical EWC parameter protection strategy described in Step 4.2 to perform incremental training on the current malicious traffic detection model. The hierarchical differentiated constraints solve the contradiction between protection strength and update range in the traditional single constraint method. This enables the model to learn new threat characteristics while providing appropriate protection for historical key parameters, thereby avoiding "catastrophic forgetting".

[0046] Step 4.4: Verify and adjust the model. Use the incrementally trained model to detect on the test dataset to verify the model's detection accuracy, robustness, and adaptability. If the preset performance indicators are not met, adjust the parameter protection strength or training parameters and retrain incrementally until the requirements are met.

[0047] In the above scheme, step 5 specifically involves the following steps:

[0048] Step 5.1: Model Deployment and Detection. Deploy the malicious traffic detection model that meets the preset performance indicators after incremental training in Step 4, and detect the malicious traffic samples to be tested.

[0049] Step 5.2: Output and feedback of detection results. The model classifies each sample to be tested and outputs a classification result of "benign" or "malicious". At the same time, it records key detection data and feeds the above classification results and key detection data back to the compliance adversarial sample generation model in Step 2 to provide data support for the optimization of sample generation strategy.

[0050] Step 5.3: Closed-loop iterative optimization. Based on the feedback detection data, the adversarial sample generation model adjusts the sample generation strategy and re-executes the process of steps 2-5 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness of the model and its adaptability to new malicious traffic.

[0051] This invention provides a robustness enhancement system for a malicious traffic detection model based on reinforcement learning and incremental learning, comprising the following modules:

[0052] Data preprocessing module: Used to build training and test datasets for malicious traffic, collect benign and malicious traffic samples, perform data preprocessing, and extract historical attack features, context features, and traffic statistical features of traffic samples to provide data support for subsequent modules;

[0053] Adversarial Example Generation Module: Based on reinforcement learning technology and combined with Transformer policy network, it constructs a compliant adversarial example generation model. Through joint modeling of state and action space, third-order composite reward function and action masking mechanism, it generates highly concealed and compliant adversarial examples.

[0054] Concept drift detection module: Used to build a concept drift detection mechanism based on MMD, to perform drift detection on adversarial samples generated by the adversarial sample generation module and malicious traffic samples collected in real time, to quantify the degree of deviation between the distribution of current traffic data and historical training data, and to start the incremental training process when the MMD value exceeds the preset threshold.

[0055] Incremental Training Module: This module is used to perform collaborative incremental training on the malicious traffic detection model based on incremental learning technology and combined with a hierarchical EWC parameter protection strategy. By evaluating the importance of parameters, it applies differentiated protection strengths to different levels of the model, achieving the dual goals of maintaining historical attack knowledge stably and adapting to new threats.

[0056] Traffic detection and closed-loop optimization module: This module applies the malicious traffic detection model trained and updated by the incremental training module to the detection of malicious traffic samples to be tested, outputs classification results, and feeds back the detection results to the adversarial example generation module to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness and adaptability of the model.

[0057] In the above scheme, the data preprocessing module is specifically implemented as follows:

[0058] Samples are collected from publicly available cybersecurity datasets and real-world network environments, including both benign and malicious traffic samples, to construct an initial dataset with a balanced ratio of benign and malicious samples. The collected traffic samples are then denoised, deduplicated, and standardized to remove invalid samples. Historical attack features, contextual features, and traffic statistics features of the samples are extracted, and the feature data is converted into a format recognizable by the model. The preprocessed dataset is divided into a training set and a test set according to a preset ratio. The training set is used for the initial training of the model and subsequent incremental training, while the test set is used to verify the model's performance.

[0059] In the above scheme, the adversarial example generation module is specifically implemented as follows:

[0060] Construct a joint model of state and action space, where state space s tThis integrates historical attack characteristics, contextual characteristics, and traffic statistics. Historical attack characteristics include recent success rate, alarm type, and interception frequency; contextual characteristics include protocol type, source and destination information, and duration; and traffic statistics include packet size distribution, throughput fluctuation, and connection duration. Action space a t This includes traffic modification operations such as timing adjustment and packet size adjustment;

[0061] A Transformer policy network is constructed, and the traffic feature sequence is input into a Transformer encoder consisting of a multi-layer multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures the correlation information of individual traffic features in different dimensions, and residual connections and layer normalization are performed on the output features of each layer of the Transformer encoder. The Transformer encoder is used to perform deep modeling of high-dimensional, long-sequence traffic state features, capturing the complex temporal and spatial dependencies between data packets. The final output features of the encoder are input into a fully connected layer, and the activation function is used to map them into vectors that match the action space dimension to generate preliminary traffic modification action instructions.

[0062] Implement an action masking mechanism, based on mainstream network transmission protocol specifications such as TCP / UDP / IP, construct a network protocol compliance rule base, and formulate illegal action judgment standards. Illegal action judgment standards include violations of packet size, tampering with key protocol fields, violations of timing adjustments, and illegal modification of source and destination information. Simultaneously, the state space s... t High-dimensional flow characteristics and action space a t The system associates traffic modification actions, clarifies the types of actions that can be executed and the scope of action modifications under different traffic states, and constructs a mapping rule base from state space to action space. At each time step t, a mask vector (Mask) is dynamically generated by combining the network protocol compliance rule base and the mapping rule base from state space to action space. t ,

[0063]

[0064] in The value is 0 or 1, where 0 indicates an illegal action that cannot be executed at time step t, and 1 indicates a legal action that can be executed at time step t. This is used to forcibly filter out illegal actions that violate network protocol specifications and ensure the compliance of action instructions.

[0065] Design a third-order composite reward function, including core attack reward, stealth reward, and resource cost penalty. The third-order composite reward function is designed as follows:

[0066]

[0067] in The core attack reward is the success rate of generating adversarial examples to bypass the detection model. The reward is for concealment, specifically the similarity between the generated adversarial example and the original traffic sample. This is a resource cost penalty, manifested in the resource and time consumption of the agent when generating adversarial examples. , , The weights corresponding to these three factors are used to balance the composite reward function. This drives the agent to achieve the optimal balance between "attack effectiveness," "stealth and survival," and "resource cost," quantifying the benefit of each action taken by the agent.

[0068] Adversarial examples are generated and optimized. The agent combines the action modification instructions generated by the policy network and the action masking mechanism to obtain the original malicious traffic sample by modifying its features based on the legal actions filtered by the mask, generating adversarial examples. The adversarial examples are then input into the current malicious traffic detection model, and the reward value is calculated based on the detection results. The Transformer policy network parameters are updated through backpropagation, and the quality of the adversarial examples is iteratively optimized.

[0069] In the above scheme, the specific implementation of the concept drift detection module is as follows:

[0070] The core features of the adversarial samples generated by the adversarial sample generation module and the malicious traffic samples collected in real time are extracted and aligned with the features of the historical traffic samples used during model training.

[0071] The MMD value is calculated using the Maximum Mean Difference (MMD) algorithm, according to the formula:

[0072]

[0073] Calculate the distributional differences between the current traffic sample characteristics and the historical traffic sample characteristics, and quantify the degree of deviation between the two, where: For sample set In the The mean of each feature dimension For sample set In the The mean of each feature dimension This represents the total number of feature dimensions.

[0074] The system performs drift detection and alarms, with a preset MMD threshold. When the calculated MMD value exceeds the threshold, it is determined that concept drift has occurred, a drift alarm based on a sliding window is activated, and an incremental training process is triggered. If the threshold is not exceeded, the current detection model is maintained, and traffic detection continues.

[0075] In the above scheme, the incremental training module is specifically implemented as follows:

[0076] The EWC (Elastic Weight Consolidation) algorithm is used to evaluate parameter importance. Fisher's information matrix is ​​used to calculate the importance of each parameter in the model for historical attack detection tasks. Predictive weights are introduced for key parameters of each historical task, and the data for each task is corrected. Different strengths of protection mechanisms are designed for the bottom feature extraction layer, the middle semantic understanding layer, and the top decision classification layer of the malicious traffic detection model. The feature extraction layer, which has strong versatility, is given higher protection weights, while the top decision layer allows for greater parameter adjustment space to adapt to new attack types. Traffic samples exhibiting concept drift, selected by the concept drift detection module, are used as incremental training data. Combined with the hierarchical EWC parameter protection strategy, the current malicious traffic detection model is incrementally trained. While learning new threat features, the update magnitude of historical key parameters is constrained to avoid "catastrophic forgetting." The incrementally trained model is used to detect on the test dataset to verify the model's detection accuracy, robustness, and adaptability. If the preset performance indicators are not met, the parameter protection strength or training parameters are adjusted, and incremental training is repeated until the requirements are met.

[0077] In the above scheme, the specific implementation of the flow detection and closed-loop optimization module is as follows:

[0078] After training the malicious traffic detection model to meet the preset performance indicators, the incremental training module is deployed to detect malicious traffic samples. The model classifies each sample and outputs a "benign" or "malicious" classification result, while recording key detection data. These classification results and key detection data are then fed back to the compliant adversarial sample generation model in the adversarial sample generation module, providing data support for optimizing the sample generation strategy. Closed-loop iterative optimization involves the adversarial sample generation model adjusting its sample generation strategy based on the feedback detection data and re-executing the above process to achieve a closed-loop iteration of "generation-detection-update-optimization," continuously improving the model's robustness and adaptability to new types of malicious traffic.

[0079] Compared with the prior art, the beneficial effects of this invention are as follows:

[0080] 1. Improve the quality of adversarial examples and enhance model robustness: By combining reinforcement learning with Transformer policy networks, a joint modeling of state-action space, a third-order composite reward function, and an action masking mechanism are constructed. The generated adversarial examples have high concealment, high compliance, and efficient attack power, solving the problems of pattern collapse and poor compliance of existing adversarial examples. Using these adversarial examples to train the detection model can significantly improve the model's anti-attack ability and robustness, and extend the effective lifespan of the detection model.

[0081] 2. Mitigating "catastrophic forgetting" and achieving continuous model adaptation: The innovative approach combines MMD concept drift detection with a hierarchical EWC parameter protection mechanism. Differentiated protection strengths are applied based on the importance of different levels of model parameters to historical tasks. This allows for efficient learning of new threat characteristics while maximizing the retention of historical knowledge, enabling rapid response to new threats and stable coverage of historical threats. This effectively addresses dynamic environmental changes and resolves the contradiction between preserving historical knowledge and adapting to new threats in existing incremental learning methods.

[0082] 3. Dynamic response to attack evolution and adaptation to complex network environments: Through the MMD concept drift detection mechanism, it can automatically sense the evolution of malicious traffic attack patterns and trigger the incremental training process in a timely manner, enabling the detection model to dynamically adapt to changes in attack patterns. Model updates can be completed without manual intervention, improving the automation level and practicality of the malicious traffic detection system, and making it suitable for complex and ever-changing network security environments.

[0083] 4. Closed-loop iterative optimization to continuously improve defense effectiveness: Construct a closed-loop iterative framework of "adversarial example generation - traffic detection - concept drift detection - incremental training - sample optimization", and feed the detection results back to the adversarial example generation module in real time to continuously optimize the sample generation strategy and the performance of the detection model, forming a virtuous cycle to ensure that the model can continuously cope with new malicious attacks and improve the overall defense effectiveness.

[0084] 5. Modular design enhances system maintainability and scalability: Through the design of multiple modules such as data preprocessing module, adversarial sample generation module, and concept drift detection module, the robustness enhancement process of malicious traffic detection model is modularized. Each module has independent functions and clear interfaces, which facilitates subsequent maintenance, upgrades and expansions. The parameters and logic of each module can be flexibly adjusted according to the needs of actual application scenarios.

[0085] In summary, this invention integrates core technologies such as reinforcement learning and incremental learning, combined with key methods such as Transformer policy networks, MMD concept drift detection, and hierarchical EWC parameter protection, to comprehensively improve the robustness, dynamic adaptability, and practical scalability of the malicious traffic detection model. It effectively solves the core pain points of existing models and provides efficient and reliable technical support for network security protection. Attached Figure Description

[0086] Figure 1 This is a schematic diagram of the overall workflow of the present invention. Detailed Implementation

[0087] The embodiments of the present invention will be described in detail below. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, any modifications or equivalent substitutions made to the present invention should be covered within the scope of the claims of the present invention.

[0088] Furthermore, to better illustrate the present invention, numerous specific details are set forth in the following detailed embodiments. Those skilled in the art will understand that the present invention can be practiced without these specific details.

[0089] This invention provides a method for enhancing the robustness of a malicious traffic detection model based on reinforcement learning and incremental learning, comprising the following steps:

[0090] Step 1: Construct a malicious traffic training dataset and a test dataset, collect benign traffic samples and malicious traffic samples, perform data preprocessing, and extract historical attack features, context features and traffic statistical features of the traffic samples to provide data support for subsequent processes;

[0091] Step 2: Based on reinforcement learning technology and combined with Transformer policy network, construct a compliant adversarial example generation model. Through joint modeling of state-action space, third-order composite reward function and action masking mechanism, generate efficient adversarial examples with high concealment and strong compliance.

[0092] Step 3: Construct a concept drift detection mechanism based on MMD to perform drift detection on the adversarial examples generated in Step 2 and the malicious traffic samples collected in real time, quantify the degree of deviation between the distribution of current traffic data and historical training data, and start the incremental training process when the MMD value exceeds the preset threshold.

[0093] Step 4: Based on incremental learning technology and combined with the hierarchical EWC parameter protection strategy, the malicious traffic detection model is trained incrementally in a collaborative manner. By evaluating the importance of parameters, differentiated protection strengths are applied to different levels of the model to achieve the dual goals of maintaining historical attack knowledge stably and adapting to new threats.

[0094] Step 5: Use the malicious traffic detection model trained and updated in Step 4 to detect malicious traffic samples to be tested, output classification results, and feed the detection results back to the compliance adversarial sample generation model in Step 2 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness and adaptability of the model.

[0095] In the above technical solution, the specific steps of step 1 are as follows:

[0096] Step 1.1: Collect samples. Collect benign traffic samples and malicious traffic samples from public network security datasets and real network environments to build an initial dataset with a balanced ratio of benign and malicious samples.

[0097] Step 1.2: Perform data preprocessing, including denoising, deduplication, and standardization of the collected traffic samples, removing invalid samples, extracting historical attack features, context features, and traffic statistics features of the samples, and converting the feature data into a format that the model can recognize.

[0098] Step 1.3: Divide the dataset. Divide the preprocessed dataset into a training set and a test set according to a preset ratio. The training set is used for the initial training of the model and subsequent incremental training, while the test set is used to verify the performance of the model.

[0099] In the above technical solution, the specific steps of step 2 are as follows:

[0100] Step 2.1: Construct a joint model of state and action space, where state space s t This integrates historical attack characteristics, contextual characteristics, and traffic statistics. Historical attack characteristics include recent success rate, alarm type, and interception frequency; contextual characteristics include protocol type, source and destination information, and duration; and traffic statistics include packet size distribution, throughput fluctuation, and connection duration. Action space a t This includes traffic modification operations such as timing adjustment and packet size adjustment;

[0101] Step 2.2: Construct a Transformer policy network, use the Transformer encoder to perform deep modeling of high-dimensional, long-sequence traffic state features, capture the complex temporal and spatial dependencies between data packets, and generate preliminary traffic modification action instructions;

[0102] Step 2.3: Implement the action masking mechanism, construct a mapping rule base from state space to action space, and dynamically generate a mask vector Mask at each time step t. t It forcibly filters out illegal actions that violate network protocol specifications, ensuring the compliance of action commands;

[0103] Step 2.4: Design a third-order composite reward function, which includes core attack reward, stealth reward and resource cost penalty, to drive the agent to achieve the optimal balance between "attack effectiveness", "stealth survival" and "resource cost", and quantify the benefits of each action of the agent;

[0104] Step 2.5: Generate and optimize adversarial examples. The agent modifies the features of the original malicious traffic sample based on the legal actions filtered by the mask, generates adversarial examples, inputs the adversarial examples into the current malicious traffic detection model, calculates the reward value based on the detection results, backpropagates to update the Transformer policy network parameters, and iteratively optimizes the quality of adversarial examples.

[0105] In the above technical solution, step 3 specifically involves the following steps:

[0106] Step 3.1: Extract features. Extract the core features of the adversarial examples generated in Step 2 and the malicious traffic samples collected in real time, and align them with the features of the historical traffic samples used during model training.

[0107] Step 3.2: Calculate the MMD value. Using the Maximum Mean Difference (MMD) algorithm, calculate the distribution difference between the current traffic sample characteristics and the historical traffic sample characteristics to quantify the degree of deviation between the two.

[0108] Step 3.3: Perform drift detection and alarm. Preset MMD threshold. When the calculated MMD value exceeds the threshold, it is determined that concept drift has occurred, the drift alarm based on sliding window is activated, and the incremental training process in step 4 is triggered; if the threshold is not exceeded, the current detection model is maintained and traffic detection continues.

[0109] In the above technical solution, step 4 specifically involves the following steps:

[0110] Step 4.1: Evaluate the importance of parameters. Use the EWC (Elastic Weight Consolidation) algorithm to calculate the importance of each parameter in the model to the historical attack detection task using the Fisher information matrix. Introduce predicted weights to the key parameters of each historical task and correct the data for each task.

[0111] Step 4.2: Design a layered parameter protection mechanism. Design protection mechanisms of different strengths for the bottom feature extraction layer, the middle semantic understanding layer and the top decision classification layer of the malicious traffic detection model. Give higher protection weight to the feature extraction layer with stronger versatility, and allow more parameter adjustment space for the top decision layer to adapt to new attack types.

[0112] Step 4.3: Perform collaborative incremental training. Use the traffic samples with conceptual drift selected in Step 3 as incremental training data. Combine the hierarchical EWC parameter protection strategy to incrementally train the current malicious traffic detection model. While learning new threat characteristics, constrain the update range of historical key parameters to avoid "catastrophic forgetting".

[0113] Step 4.4: Verify and adjust the model. Use the incrementally trained model to detect on the test dataset to verify the model's detection accuracy, robustness, and adaptability. If the preset performance indicators are not met, adjust the parameter protection strength or training parameters and retrain incrementally until the requirements are met.

[0114] In the above technical solution, step 5 specifically involves the following steps:

[0115] Step 5.1: Model Deployment and Detection. Deploy the malicious traffic detection model that meets the preset performance indicators after incremental training in Step 4, and detect the malicious traffic samples to be tested.

[0116] Step 5.2: Output and feedback of detection results. The model classifies each sample to be tested and outputs a classification result of "benign" or "malicious". At the same time, it records key detection data and feeds the above classification results and key detection data back to the compliance adversarial sample generation model in Step 2 to provide data support for the optimization of sample generation strategy.

[0117] Step 5.3: Closed-loop iterative optimization. Based on the feedback detection data, the adversarial sample generation model adjusts the sample generation strategy and re-executes the process of steps 2-5 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness of the model and its adaptability to new malicious traffic.

[0118] The above technical solution includes the following modules:

[0119] Data preprocessing module: Used to build training and test datasets for malicious traffic, collect benign and malicious traffic samples, perform data preprocessing, and extract historical attack features, context features, and traffic statistical features of traffic samples to provide data support for subsequent modules;

[0120] Adversarial Example Generation Module: Based on reinforcement learning technology and combined with Transformer policy network, it constructs a compliant adversarial example generation model. Through joint modeling of state and action space, third-order composite reward function and action masking mechanism, it generates highly concealed and compliant adversarial examples.

[0121] Concept drift detection module: Used to build a concept drift detection mechanism based on MMD, to perform drift detection on adversarial samples generated by the adversarial sample generation module and malicious traffic samples collected in real time, to quantify the degree of deviation between the distribution of current traffic data and historical training data, and to start the incremental training process when the MMD value exceeds the preset threshold.

[0122] Incremental Training Module: This module is used to perform collaborative incremental training on the malicious traffic detection model based on incremental learning technology and combined with a hierarchical EWC parameter protection strategy. By evaluating the importance of parameters, it applies differentiated protection strengths to different levels of the model, achieving the dual goals of maintaining historical attack knowledge stably and adapting to new threats.

[0123] Traffic detection and closed-loop optimization module: This module applies the malicious traffic detection model trained and updated by the incremental training module to the detection of malicious traffic samples to be tested, outputs classification results, and feeds back the detection results to the adversarial example generation module to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness and adaptability of the model.

[0124] In the above technical solution, the data preprocessing module is specifically implemented as follows:

[0125] Collect samples by gathering benign and malicious traffic samples from publicly available cybersecurity datasets and real-world network environments to construct an initial dataset with a balanced ratio of benign to malicious samples. Perform denoising, deduplication, and standardization on the collected traffic samples, removing invalid samples and extracting historical attack features, contextual features, and traffic statistics features from the samples. Convert the feature data into a format recognizable by the model. Divide the preprocessed dataset into a training set and a test set according to a preset ratio (e.g., 8:2). The training set is used for initial model training and subsequent incremental training, while the test set is used to verify the model's performance.

[0126] In the above technical solution, the adversarial example generation module is specifically implemented as follows:

[0127] Construct a joint model of state and action space, where state space s t By comprehensively integrating historical attack characteristics (recent success rate, alarm type, interception frequency), contextual characteristics (protocol type, source and destination information, duration), and traffic statistics characteristics (packet size distribution, throughput fluctuation, connection duration), the action space a t This includes traffic modification operations such as timing adjustment and packet size adjustment; constructing a Transformer policy network, utilizing a Transformer encoder to perform deep modeling of high-dimensional, long-sequence traffic state features, capturing complex temporal and spatial dependencies between data packets, and generating preliminary traffic modification action instructions; implementing an action masking mechanism, constructing a mapping rule base from state space to action space, and dynamically generating a mask vector Mask at each time step t. tThe system forcibly filters illegal actions that violate network protocol specifications to ensure the compliance of action commands. A three-order composite reward function is designed, comprising core attack rewards, stealth rewards, and resource cost penalties, driving the agent to achieve an optimal balance between "attack effectiveness," "stealth survival," and "resource cost," quantifying the benefit of each action. Based on the masked and filtered legal actions, the agent modifies the features of the original malicious traffic samples to generate adversarial examples. These adversarial examples are then input into the current malicious traffic detection model, and reward values ​​are calculated based on the detection results. Backpropagation updates the Transformer policy network parameters, iteratively optimizing the quality of the adversarial examples.

[0128] In the above technical solution, the specific implementation of the concept drift detection module is as follows:

[0129] Feature extraction: The core features of adversarial samples generated by the adversarial sample generation module and malicious traffic samples collected in real time are extracted and aligned with the features of historical traffic samples used during model training. The maximum mean difference (MMD) algorithm is used to calculate the distribution difference between the features of the current traffic sample and the features of historical traffic samples, quantifying the degree of deviation between the two. A preset MMD threshold is set. When the calculated MMD value exceeds the threshold, concept drift is determined to have occurred, a drift alarm based on a sliding window is activated, and the incremental training process of the incremental training module is triggered. If the threshold is not exceeded, the current detection model is maintained and traffic detection continues.

[0130] In the above technical solution, the incremental training module is specifically implemented as follows:

[0131] To assess parameter importance, the EWC (Elastic Weight Consolidation) algorithm is employed. This algorithm utilizes the Fisher information matrix to calculate the importance of each parameter in the model for historical attack detection tasks, introducing predicted weights for key parameters of each historical task and correcting the data for each task. Different levels of protection mechanisms are designed for the bottom-level feature extraction layer, the middle-level semantic understanding layer, and the top-level decision classification layer of the malicious traffic detection model. The feature extraction layer, with its higher versatility, receives higher protection weights, while the top-level decision layer allows for greater parameter adjustment space to adapt to new attack types. Traffic samples exhibiting concept drift, selected by the concept drift detection module, are used as incremental training data. Combined with the hierarchical EWC parameter protection strategy, the current malicious traffic detection model is incrementally trained. This learns new threat features while constraining the update magnitude of historical key parameters to avoid "catastrophic forgetting." The incrementally trained model is then used to detect on a test dataset to verify its detection accuracy, robustness, and adaptability. If the preset performance indicators are not met, the parameter protection strength or training parameters are adjusted, and incremental training is repeated until the requirements are met.

[0132] In the above technical solution, the specific implementation of the flow detection and closed-loop optimization module is as follows:

[0133] After training the malicious traffic detection model to meet the preset performance indicators, the incremental training module is deployed to detect malicious traffic samples. The model classifies each sample and outputs a "benign" or "malicious" classification result, while recording key detection data. These classification results and key detection data are then fed back to the compliant adversarial sample generation model in the adversarial sample generation module, providing data support for optimizing the sample generation strategy. Closed-loop iterative optimization involves the adversarial sample generation model adjusting its sample generation strategy based on the feedback detection data and re-executing the above process to achieve a closed-loop iteration of "generation-detection-update-optimization," continuously improving the model's robustness and adaptability to new types of malicious traffic.

[0134] To make the objectives, technical solutions, and advantages of this invention clearer, examples are provided below for illustration.

[0135] Example 1

[0136] Step 1: Construct the malicious traffic training dataset and test dataset, including the following steps:

[0137] Step 1.1: Collect samples. Collect 5,000 benign traffic samples and 5,000 malicious traffic samples (including various attack types such as DDoS attacks, web attacks, and botnet attacks) from public network security datasets (such as CIC-IDS-2017 and NSL-KDD) and a company's actual network environment to build a balanced initial dataset with a 1:1 ratio of benign and malicious samples.

[0138] Step 1.2: Perform data preprocessing. Use Python's Pandas library to denoise and deduplicatize the collected traffic samples, removing invalid and abnormal samples. Use the StandardScaler function of the Scikit-learn library to standardize the feature data, transforming the feature values ​​to the [0,1] interval. Extract historical attack features (recent success rate, alarm type, interception frequency), context features (protocol type, source IP, destination IP, duration), and traffic statistics features (packet size distribution, throughput fluctuation, connection duration) from the samples, and convert the feature data into a tensor format that the model can recognize.

[0139] Step 1.3: Divide the dataset. Divide the preprocessed dataset into a training set (8000 samples) and a test set (2000 samples) in an 8:2 ratio. The training set is used for the initial training of the model and subsequent incremental training, while the test set is used to verify the performance of the model.

[0140] Step 2: Construct a compliance adversarial example generation model to generate efficient adversarial examples, including the following steps:

[0141] Step 2.1: Construct a joint model of state and action space, where state space s t By comprehensively integrating historical attack characteristics (recent success rate, alarm type, interception frequency), contextual characteristics (protocol type, source IP, destination IP, duration), and traffic statistics characteristics (packet size distribution, throughput fluctuation, connection duration), various characteristics are combined into a 128-dimensional state vector using vector concatenation; action space a t It includes three types of actions: timing adjustment (adjusting the data packet sending interval), packet size adjustment (adjusting the number of bytes in the data packet), and protocol field fine-tuning (fine-tuning the TCP / UDP protocol header fields), totaling 16 specific operations;

[0142] Step 2.2: Construct a Transformer policy network. Using the PyTorch framework, a Transformer policy network is constructed based on the existing Transformer encoder structure. It includes four encoder layers, each containing a multi-head self-attention mechanism (8 heads) and a feedforward neural network (256 hidden layers). The input is a state vector. The 128-dimensional flow state vector constructed in Step 2.1 is used as the input to the Transformer policy network. Before input, layer normalization is performed to standardize the feature values ​​to the [0,1] interval. The multi-head self-attention mechanism assigns weights to the high-dimensional flow features in the state vector. Finally, the attention-weighted feature vector is obtained by concatenation and fusion. This weighted feature vector is then input into the feedforward neural network, where a ReLU activation function is used for non-linear feature transformation. Layer normalization is then performed to complete the processing of a single encoder layer. The signal undergoes the same iterative process through all four encoder layers. The feature vector output from the last encoder layer is input into a Softmax activation function, converting it into a 16-dimensional action probability distribution output, generating preliminary flow modification action instructions.

[0143] Step 2.3: Implement the action masking mechanism, construct a mapping rule base from state space to action space, and define illegal actions based on the TCP / UDP protocol specification (such as modifying the protocol type field, exceeding the maximum limit of packet size, etc.). At each time step t, based on the current state vector s... t Dynamically generate a 16-dimensional mask vector. t (Legal actions are 1, illegal actions are 0), forcibly filtering illegal actions to ensure the compliance of action instructions;

[0144] Step 2.4: Design a third-order composite reward function, R = 0.4 × R_core + 0.3 × R_concealment - 0.3 × R_cost, where R_core is the core attack reward (10 points for successfully bypassing the detection model, -5 points for failing to bypass it), R_concealment is the concealment reward (8 points for feature similarity ≥ 0.9 between adversarial examples and benign examples, 4 points for 0.7~0.9, and -3 points for < 0.7), and R_cost is the resource cost penalty (0 points for ≤ 3 modification actions, -2 points for 3~5 modification actions, and -5 points for > 5 modification actions), driving the agent to achieve the optimal balance between "attack effectiveness", "concealment survival" and "resource cost".

[0145] Step 2.5: Generate and optimize adversarial examples. The reinforcement learning agent is trained using the proximal policy optimization (PPO) algorithm for 1000 iterations, with each iteration containing 100 sample batches and each batch containing 64 samples. The agent modifies the features of the original malicious traffic samples based on the masked, legal actions. The agent obtains compliant actions based on the action probability distribution output by the Transformer policy network and the masked filtering, and performs targeted modifications on the samples. All modifications are limited to protocol compliance. For temporal feature modifications, within the legal interval of 0.01~0.5s, the data packet sending interval is slightly adjusted according to the action instruction. For packet size feature modifications, within the range of 46~1500... Within the legal range of bytes, the packet payload length is increased or decreased according to the action instruction; for protocol field modification, within the legal range of protocol window size, the window size field value is fine-tuned within 5000 bytes to generate adversarial examples. The adversarial examples are then input into the initial malicious traffic detection model (built based on CNN-LSTM), the reward value is calculated based on the detection results, the Transformer policy network parameters are updated through backpropagation, the quality of adversarial examples is iteratively optimized, and finally 10,000 efficient adversarial examples are generated.

[0146] Step 3: Concept drift detection triggers the incremental training process, including the following steps:

[0147] Step 3.1: Extract features. Extract the core features (consistent with the features extracted in Step 1.2) from the 10,000 adversarial samples generated in Step 2 and the 500 malicious traffic samples collected in real time, and align them with the features of the historical traffic samples used during the initial training of the model.

[0148] Step 3.2: Calculate the MMD value. The MMD algorithm is implemented using the Scikit-learn library in Python. The kernel function is a Gaussian kernel (with a bandwidth of 0.1). The distribution difference between the current traffic sample characteristics and the historical traffic sample characteristics is calculated to obtain the MMD value.

[0149] Step 3.3: Perform drift judgment and alarm. The preset MMD threshold is 0.05. The calculated MMD value is 0.07, which exceeds the preset threshold. It is determined that concept drift has occurred, and the drift alarm based on the sliding window (window size is 100) is activated, and the incremental training process in step 4 is triggered.

[0150] Step 4: Collaborative incremental training to update the malicious traffic detection model, including the following steps:

[0151] Step 4.1: Evaluate parameter importance. Using the EWC algorithm, the importance of each parameter in the initial detection model to the historical attack detection task is calculated using the Fisher information matrix. Predictive weights (ranging from 0.8 to 1.0) are introduced for the key parameters of each historical task, and the data for each task is corrected.

[0152] Step 4.2: Design a hierarchical parameter protection mechanism. Design protection mechanisms of different strengths for the bottom feature extraction layer (CNN layer), the middle semantic understanding layer (LSTM layer), and the top decision classification layer (fully connected layer) of the CNN-LSTM detection model. The protection weight of the feature extraction layer is 0.9, the protection weight of the semantic understanding layer is 0.7, and the protection weight of the decision classification layer is 0.5.

[0153] Step 4.3: Perform collaborative incremental training. Use the 1200 traffic samples (including adversarial samples and real-time collected samples) selected in Step 3 that exhibit conceptual drift as incremental training data. Combined with the hierarchical EWC parameter protection strategy, incrementally train the initial detection model. The training rounds are 200, the learning rate is 0.001, and the batch size is 64. While learning new threat features, the update range of historical key parameters is constrained to avoid "catastrophic forgetting".

[0154] Step 4.4: Verify and adjust the model. Use the incrementally trained model for detection on the test dataset. The model's detection accuracy is 98.2%, precision is 97.8%, recall is 98.5%, and F1 score is 98.1%, all of which meet the preset performance indicators (accuracy ≥ 97%). No parameter adjustment is required, and the model update is completed.

[0155] Step 5: Closed-loop iterative optimization to continuously improve model performance, including the following steps:

[0156] Step 5.1: Model Deployment and Detection. Deploy the malicious traffic detection model that meets the preset performance indicators (detection accuracy ≥ 97%) after incremental training in Step 4, and detect the malicious traffic samples to be tested. For the samples to be tested, first perform the preprocessing operation in Step 1.2 to extract their historical attack features, context features and traffic statistics features, convert them into tensor formats that the model can recognize, and then input them into the updated detection model for detection.

[0157] Step 5.2: Output and feedback of detection results. The model classifies each sample to be tested and outputs a classification result of "benign" or "malicious". At the same time, it records key detection data (including the attack success rate of adversarial examples, the feature similarity between adversarial examples and benign examples, the model's false negative rate, false positive rate, etc.). The above classification results and key detection data are fed back to the compliant adversarial example generation model in Step 2 to provide data support for the optimization of the sample generation strategy.

[0158] Step 5.3: Closed-loop iterative optimization. Based on the feedback detection data, the adversarial sample generation model adjusts the sample generation strategy and re-executes the process of steps 2-5 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness of the model and its adaptability to new malicious traffic.

[0159] Example 2:

[0160] Suppose we have an enterprise perimeter network environment, which serves as the connection point between the enterprise and the Internet. It faces malicious traffic threats such as DDoS, SQL injection, and botnets. We need to use the method of this invention to enhance the anti-attack capability and dynamic adaptability of the malicious traffic detection model, alleviate "catastrophic amnesia", and achieve accurate detection and continuous optimization.

[0161] Step 1: Collect public datasets (CIC-IDS-2017, NSL-KDD) and actual enterprise traffic samples, and collect 5000 benign and 5000 malicious samples from each to construct a 1:1 balanced dataset; after denoising, deduplication, and standardization preprocessing, extract three core features: historical attacks, context, and traffic statistics, and convert them into tensor format, and divide them into a training set (8000 samples) and a test set (2000 samples) in an 8:2 ratio for model training and performance verification.

[0162] Step 2: Construct a compliant adversarial sample generation model based on reinforcement learning and Transformer policy network. Integrate three types of features into a 128-dimensional state vector. The action space includes 16 types of operations in three categories: temporal sequence, packet size, and protocol field fine-tuning. Filter illegal actions through an action masking mechanism. Design a third-order composite reward function (R = 0.4 × R_core + 0.3 × R_concealment + 0.3 × R_cost). Train the agent for 1000 rounds using the PPO algorithm, and finally generate 10,000 efficient adversarial samples.

[0163] Step 3: Construct a concept drift detection mechanism using the MMD algorithm, extract the core features of adversarial samples (10,000) and real-time collected malicious traffic samples (500) and align them with historical samples, use a Gaussian kernel (bandwidth 0.1) to calculate the MMD value; preset threshold 0.05, when the calculated value (0.07) exceeds the threshold, it is determined that concept drift has occurred and incremental training is triggered.

[0164] Step 4: Incremental training was performed based on the hierarchical EWC parameter protection strategy. The importance of parameters was evaluated using the Fisher information matrix, and differentiated protection weights were set for each layer of the CNN-LSTM model (bottom layer 0.9, middle layer 0.7, top layer 0.5). 1200 drift samples were used as incremental data, and the model was trained for 200 rounds (learning rate 0.001, batch size 64). After validation, the model accuracy reached 98.2%, and the update was completed.

[0165] Step 5: Deploy the updated model to the enterprise boundary detection node, perform detection on the traffic to be tested after preprocessing, output "benign / malicious" results and record key data, feed it back to the adversarial example generation model adjustment strategy, repeat steps 2-5 to achieve closed-loop iteration, and continuously improve model performance.

[0166] Through the above steps, this invention successfully enhances the robustness of the enterprise boundary network detection model, alleviates "catastrophic forgetting," can automatically detect the evolution of attack patterns, complete model updates without manual intervention, effectively prevents malicious traffic attacks, ensures the stability of enterprise business and the security of core data, and reduces the manual cost of protection.

Claims

1. A method for enhancing the robustness of a malicious traffic detection model based on reinforcement learning and incremental learning, characterized in that, Includes the following steps: Step 1: Construct a malicious traffic training dataset and a test dataset, collect benign traffic samples and malicious traffic samples, perform data preprocessing, and extract historical attack features, context features and traffic statistical features of the traffic samples to provide data support for subsequent processes; Step 2: Based on reinforcement learning technology and combined with Transformer policy network, construct a compliant adversarial example generation model. Through joint modeling of state-action space, third-order composite reward function and action masking mechanism, generate efficient adversarial examples with high concealment and strong compliance. Step 3: Construct a concept drift detection mechanism based on MMD to perform drift detection on the adversarial examples generated in Step 2 and the malicious traffic samples collected in real time, quantify the degree of deviation between the distribution of current traffic data and historical training data, and start the incremental training process when the MMD value exceeds the preset threshold. Step 4: Based on incremental learning technology and combined with the hierarchical EWC parameter protection strategy, the malicious traffic detection model is trained incrementally in a collaborative manner. By evaluating the importance of parameters, differentiated protection strengths are applied to different levels of the model to achieve the dual goals of maintaining historical attack knowledge stably and adapting to new threats. Step 5: Use the malicious traffic detection model trained and updated in Step 4 to detect malicious traffic samples to be tested, output classification results, and feed the detection results back to the compliance adversarial sample generation model in Step 2 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness and adaptability of the model.

2. The robustness enhancement method for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 1, characterized in that, The specific steps of step 1 are as follows: Step 1.1: Collect samples. Collect benign traffic samples and malicious traffic samples from public network security datasets and real network environments to construct an initial dataset with a balanced ratio of benign and malicious samples. Step 1.2: Perform data preprocessing, including denoising, deduplication, and standardization of the collected traffic samples, removing invalid samples, extracting historical attack features, context features, and traffic statistics features of the samples, and converting the feature data into a format that the model can recognize. Step 1.3: Divide the dataset. Divide the preprocessed dataset into a training set and a test set according to a preset ratio. The training set is used for the initial training of the model and subsequent incremental training, while the test set is used to verify the performance of the model.

3. The robustness enhancement method for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 1, characterized in that, The specific steps of step 2 are as follows: Step 2.1: Construct a joint model of state and action space, where state space s t It comprehensively integrates historical attack characteristics, contextual characteristics, and traffic statistics characteristics. Historical attack characteristics include recent success rate, alarm type, and interception frequency. Contextual characteristics include protocol type, source and destination information, and duration. Traffic statistics characteristics include packet size distribution, throughput fluctuation, and connection duration. Action space a t This includes traffic modification operations such as timing adjustments and packet size adjustments; Step 2.2: Construct a Transformer policy network. Input the traffic feature sequence into a Transformer encoder consisting of a multi-layer multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures the correlation information of individual traffic features in different dimensions. The output features of each layer of the Transformer encoder are processed by residual connection and layer normalization. The Transformer encoder is used to perform deep modeling of high-dimensional, long-sequence traffic state features, capturing the complex temporal and spatial dependencies between data packets. The final output features of the encoder are input into a fully connected layer and mapped to a vector matching the action space dimension through an activation function to generate preliminary traffic modification action instructions. Step 2.3: Implement an action masking mechanism. Based on mainstream network transmission protocol specifications such as TCP / UDP / IP, construct a network protocol compliance rule base and formulate illegal action judgment standards. Illegal action judgment standards include violations of packet size, tampering with key protocol fields, violations of timing adjustment, and illegal modification of source and destination information. Simultaneously, the state space s from Step 2.1 will be... t High-dimensional flow characteristics and action space a t The system associates traffic modification actions, clarifies the types of actions that can be executed and the scope of action modifications under different traffic states, and constructs a mapping rule base from state space to action space. At each time step t, a mask vector (Mask) is dynamically generated by combining the network protocol compliance rule base and the mapping rule base from state space to action space. t , in The value is 0 or 1, where 0 indicates an illegal action that cannot be executed at time step t, and 1 indicates a legal action that can be executed at time step t. This is used to forcibly filter out illegal actions that violate network protocol specifications and ensure the compliance of action instructions. Step 2.4: Design a third-order composite reward function, including core attack reward, stealth reward, and resource cost penalty. The third-order composite reward function is designed as follows: in The core attack reward is the success rate of generating adversarial examples to bypass the detection model. The reward is for concealment, specifically the similarity between the generated adversarial example and the original traffic sample. This is a resource cost penalty, manifested in the resource and time consumption of the agent when generating adversarial examples. , , The weights corresponding to the three factors are used to balance the composite reward function, driving the agent to achieve the optimal balance between "attack effectiveness", "stealth survival" and "resource cost", and quantifying the benefits of each action of the agent; Step 2.5: Generate and optimize adversarial examples. The agent combines the action modification instructions generated by the policy network in Step 2.2 and the action masking mechanism in Step 2.3 to obtain legal actions filtered by the mask. It modifies the features of the original malicious traffic sample to generate adversarial examples. The adversarial examples are input into the current malicious traffic detection model. The reward value is calculated based on the detection results. The Transformer policy network parameters are updated by backpropagation, and the quality of the adversarial examples is iteratively optimized.

4. The robustness enhancement method for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 1, characterized in that, The specific steps of step 3 are as follows: Step 3.1: Extract features. Extract the core features of the adversarial examples generated in Step 2 and the malicious traffic samples collected in real time, and align them with the features of the historical traffic samples used during model training. Step 3.2: Calculate the MMD value using the Maximum Mean Difference (MMD) algorithm, according to the formula: Calculate the distributional differences between the current traffic sample characteristics and the historical traffic sample characteristics, and quantify the degree of deviation between the two, where: For sample set In the The mean of each feature dimension For sample set In the The mean of each feature dimension This represents the total number of feature dimensions. Step 3.3: Perform drift detection and alarm. Preset MMD threshold. When the calculated MMD value exceeds the threshold, it is determined that concept drift has occurred, the drift alarm based on sliding window is activated, and the incremental training process in step 4 is triggered; if the threshold is not exceeded, the current detection model is maintained and traffic detection continues.

5. The robustness enhancement method for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 1, characterized in that, The specific steps of step 4 are as follows: Step 4.1: Evaluate the importance of parameters, consolidate the EWC algorithm with elastic weights, calculate the importance of each parameter in the model to the historical attack detection task using the Fisher information matrix, introduce prediction weights for the key parameters of each historical task, and correct the data for each task. Step 4.2: Design a hierarchical parameter protection mechanism. Based on hierarchical importance, differentiated EWC regularization coefficients, i.e., protection weights, are set for the bottom feature extraction layer, the middle semantic understanding layer, and the top decision classification layer of the malicious traffic detection model. These coefficients directly determine the constraint strength of parameter updates during incremental training. A higher protection weight is given to the more general feature extraction layer, while the top decision layer is allowed a larger parameter adjustment space to adapt to new attack types. A hierarchical EWC regularization term is introduced into the loss function of incremental training. The total loss function is designed as follows: in The incremental task loss represents the model hierarchy. The mean of the Fisher information matrix for the parameters of this layer. These are the current parameters during incremental training. The original parameters for historical tasks; protection weights are set through hierarchical configuration. This enables differentiated protection at different levels; Step 4.3: Perform collaborative incremental training. Use the traffic samples with conceptual drift selected in Step 3 as incremental training data. Combine the hierarchical EWC parameter protection strategy described in Step 4.2 to perform incremental training on the current malicious traffic detection model. The hierarchical differentiated constraints solve the contradiction between protection strength and update range in the traditional single constraint method. This enables the model to learn new threat characteristics while providing appropriate protection for historical key parameters, thereby avoiding "catastrophic forgetting". Step 4.4: Verify and adjust the model. Use the incrementally trained model to detect on the test dataset to verify the model's detection accuracy, robustness, and adaptability. If the preset performance indicators are not met, adjust the parameter protection strength or training parameters and retrain incrementally until the requirements are met.

6. The robustness enhancement method for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 1, characterized in that, The specific steps of step 5 are as follows: Step 5.1: Model Deployment and Detection. Deploy the malicious traffic detection model that meets the preset performance indicators after incremental training in Step 4, and detect the malicious traffic samples to be tested. Step 5.2: Output and feedback of detection results. The model classifies each sample to be tested and outputs a classification result of "benign" or "malicious". At the same time, it records key detection data and feeds the above classification results and key detection data back to the compliance adversarial sample generation model in Step 2 to provide data support for the optimization of sample generation strategy. Step 5.3: Closed-loop iterative optimization. Based on the feedback detection data, the adversarial sample generation model adjusts the sample generation strategy and re-executes the process of steps 2-5 to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness of the model and its adaptability to new malicious traffic.

7. A robustness enhancement system for a malicious traffic detection model based on reinforcement learning and incremental learning, characterized in that, Includes the following modules: Data preprocessing module: Used to build training and test datasets for malicious traffic, collect benign and malicious traffic samples, perform data preprocessing, and extract historical attack features, context features, and traffic statistical features of traffic samples to provide data support for subsequent modules; Adversarial Example Generation Module: Based on reinforcement learning technology and combined with Transformer policy network, it builds a compliant adversarial example generation model. Through joint modeling of state and action space, third-order composite reward function and action masking mechanism, it generates highly concealed and compliant adversarial examples. Concept drift detection module: Used to build a concept drift detection mechanism based on MMD, to perform drift detection on adversarial samples generated by the adversarial sample generation module and malicious traffic samples collected in real time, to quantify the degree of deviation between the distribution of current traffic data and historical training data, and to start the incremental training process when the MMD value exceeds the preset threshold. Incremental Training Module: This module is used to perform collaborative incremental training on the malicious traffic detection model based on incremental learning technology and combined with a hierarchical EWC parameter protection strategy. By evaluating the importance of parameters, it applies differentiated protection strengths to different levels of the model, achieving the dual goals of maintaining historical attack knowledge stably and adapting to new threats. Traffic detection and closed-loop optimization module: This module applies the malicious traffic detection model trained and updated by the incremental training module to the detection of malicious traffic samples to be tested, outputs classification results, and feeds back the detection results to the adversarial example generation module to achieve a closed-loop iteration of "generation-detection-update-optimization" and continuously improve the robustness and adaptability of the model.

8. The robustness enhancement system for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 7, characterized in that, The specific implementation of the data preprocessing module is as follows: Collect samples: Collect benign and malicious traffic samples from public network security datasets and real network environments to construct an initial dataset with a balanced ratio of benign and malicious samples. The collected traffic samples are denoised, deduplicated, and standardized to remove invalid samples. Historical attack features, context features, and traffic statistics features of the samples are extracted and the feature data is converted into a format that the model can recognize. The preprocessed dataset is divided into a training set and a test set according to a preset ratio. The training set is used for the initial training of the model and subsequent incremental training, while the test set is used to verify the performance of the model.

9. The robustness enhancement system for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 7, characterized in that, The specific implementation of the adversarial sample generation module is as follows: Construct a joint model of state and action space, where state space s t It comprehensively integrates historical attack characteristics, contextual characteristics, and traffic statistics characteristics. Historical attack characteristics include recent success rate, alarm type, and interception frequency. Contextual characteristics include protocol type, source and destination information, and duration. Traffic statistics characteristics include packet size distribution, throughput fluctuation, and connection duration. Action space a t This includes traffic modification operations such as timing adjustment and packet size adjustment; A Transformer policy network is constructed, and the traffic feature sequence is input into a Transformer encoder consisting of a multi-layer multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism captures the correlation information of individual traffic features in different dimensions, and residual connections and layer normalization are performed on the output features of each layer of the Transformer encoder. The Transformer encoder is used to perform deep modeling of high-dimensional, long-sequence traffic state features, capturing the complex temporal and spatial dependencies between data packets. The final output features of the encoder are input into a fully connected layer, and the activation function is used to map them into vectors that match the action space dimension to generate preliminary traffic modification action instructions. Implement an action masking mechanism, based on mainstream network transmission protocol specifications such as TCP / UDP / IP, construct a network protocol compliance rule base, and formulate illegal action judgment standards. Illegal action judgment standards include violations of packet size, tampering with key protocol fields, violations of timing adjustments, and illegal modification of source and destination information. Simultaneously, the state space s... t High-dimensional flow characteristics and action space a t The system associates traffic modification actions, clarifies the types of actions that can be executed and the scope of action modifications under different traffic states, and constructs a mapping rule base from state space to action space. At each time step t, a mask vector (Mask) is dynamically generated by combining the network protocol compliance rule base and the mapping rule base from state space to action space. t , in The value is 0 or 1, where 0 indicates an illegal action that cannot be executed at time step t, and 1 indicates a legal action that can be executed at time step t. This is used to forcibly filter out illegal actions that violate network protocol specifications and ensure the compliance of action instructions. Design a third-order composite reward function, including core attack reward, stealth reward, and resource cost penalty. The third-order composite reward function is designed as follows: in The core attack reward is the success rate of generating adversarial examples to bypass the detection model. The reward is for concealment, specifically the similarity between the generated adversarial example and the original traffic sample. This is a resource cost penalty, manifested in the resource and time consumption of the agent when generating adversarial examples. , , The weights corresponding to these three factors are used to balance the composite reward function. This drives the agent to achieve the optimal balance between "attack effectiveness," "stealth and survival," and "resource cost," quantifying the benefit of each action taken by the agent. Adversarial examples are generated and optimized. The agent combines the action modification instructions generated by the policy network and the action masking mechanism to obtain the original malicious traffic sample by modifying its features based on the legal actions filtered by the mask, generating adversarial examples. The adversarial examples are then input into the current malicious traffic detection model, and the reward value is calculated based on the detection results. The Transformer policy network parameters are updated through backpropagation, and the quality of the adversarial examples is iteratively optimized.

10. The robustness enhancement system for a malicious traffic detection model based on reinforcement learning and incremental learning according to claim 7, characterized in that, The specific implementation of the concept drift detection module is as follows: The core features of the adversarial samples generated by the adversarial sample generation module and the malicious traffic samples collected in real time are extracted and aligned with the features of historical traffic samples used during model training. The MMD value is calculated using the Maximum Mean Difference (MMD) algorithm, according to the formula: Calculate the distributional differences between the current traffic sample characteristics and the historical traffic sample characteristics, and quantify the degree of deviation between the two, where: For sample set In the The mean of each feature dimension For sample set In the The mean of each feature dimension This represents the total number of feature dimensions. The system performs drift detection and alarms, with a preset MMD threshold. When the calculated MMD value exceeds the threshold, it is determined that concept drift has occurred, a drift alarm based on a sliding window is activated, and an incremental training process is triggered. If the threshold is not exceeded, the current detection model is maintained, and traffic detection continues.

Images