Intrusion Detection Method Based on Deep Reinforcement Learning and Structured Data Transformer

Abstract

This invention discloses an intrusion detection method based on deep reinforcement learning and structured data Transformer. The method includes: S1, collecting traffic data and inputting the dataset into an agent; S2, the agent classifies the currently input traffic data using a classifier; S3, the environment determines the category selected by the agent and the true category, and inputs the reward feedback result and newly sampled traffic data into the agent; S4, when the stored data reaches the batch size for data updates, adaptively updating the weights of each type of sample distribution; S5, updating the value function using an improved DQN algorithm, enabling the agent to learn from experience and improve its decisions; S6, terminating training when the set number of learning rounds is reached; S7, testing on new traffic data and evaluating the intrusion detection classification performance indicators. This invention effectively prevents the situation where minority category samples are overwhelmed by majority category samples, improving the problem of imbalanced data sample distribution in intrusion detection tasks.

Description

Technical Field

[0001] This invention belongs to the technical field of network information security, specifically relating to an intrusion detection method based on deep reinforcement learning and structured data Transformer. Background Technology

[0002] With the rapid development of computer and network technologies, concepts such as smart homes, smart vehicles, and smart cities have become a reality, and people are increasingly reliant on the internet for information transmission and data processing. However, cybersecurity threats are escalating, especially the increasing number of intrusions into networks, which can lead to significant economic losses, privacy breaches, and infrastructure damage. Therefore, to protect networks and systems from intrusions, specialized intrusion detection systems have become a crucial component of cybersecurity. However, a major challenge in intrusion detection is the imbalanced sample distribution; typically, the number of normal samples far exceeds the number of abnormal samples. This causes models to learn better from normal samples but poorly from abnormal samples.

[0003] Common methods for addressing imbalanced sample distribution include data sampling techniques, which balance the dataset by adding outlier samples or removing normal samples; ensemble learning, such as Boosting and Stacking, which integrates the predictions of multiple classifiers to compensate for the poor prediction of minority class attacks by a single classifier; and generative adversarial networks (GANs) to generate synthetic minority class samples, increasing the proportion of minority class attacks in the dataset. These generated samples are similar to but distinct from real data, improving the model's generalization ability. While these methods alleviate the impact of imbalanced sample distribution on intrusion detection to some extent, they also have limitations. Data sampling techniques may lead to information loss, and adding normal samples or removing outlier samples can affect the accuracy of the data sample distribution in the real network. Ensemble learning, while improving the robustness of intrusion detection models, increases computational and storage overhead in practical applications. While GAN-generated synthetic samples can increase the proportion of minority class attacks, the quality and diversity of the generated samples are issues that need attention. Summary of the Invention

[0004] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide an intrusion detection method based on deep reinforcement learning and structured data Transformer. By introducing a dual experience replay pool design in the deep Q-network (DQN), a smaller number of more challenging minority class samples are specially stored to effectively prevent minority class samples from being overwhelmed by majority class samples, thereby improving the problem of unbalanced data sample distribution in intrusion detection tasks.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides an intrusion detection method based on deep reinforcement learning and structured data Transformer, comprising the following steps:

[0007] S1. Collect traffic data and input the dataset into the agent;

[0008] S2. The agent uses a classifier to classify the dataset and obtain the classification results of the traffic data; the classifier includes an improved deep Q network and a Transformer model, and the improved deep Q network includes a dual experience replay pool and an experience replay pool switching window.

[0009] S3. The intrusion detection environment compares the classification results of the agent with the true category and generates reward feedback; the reward feedback and newly collected traffic data are input into the agent for the agent's learning and decision optimization; the currently processed traffic data, the classification results of the traffic data, the reward feedback and the newly collected traffic data are combined into an experience quadruple and stored in a dual experience replay pool; the dual experience replay pool includes a first experience replay pool and a second experience replay pool.

[0010] S4. Repeat steps S1 to S3 until the stored data reaches the preset batch size, and then adaptively update the distribution weights of various samples.

[0011] S5. The intrusion detection environment uses reward feedback to update the agent's value function, and uses various samples from the dual experience replay pools to train the agent and update the agent's decision-making strategy; the agent selects various samples from the first experience replay pool or the second experience replay pool through the experience replay pool switching window.

[0012] S6. Repeat steps S1 to S5 until the agent reaches the set number of learning rounds and the agent completes training.

[0013] S7. Input the unknown traffic data into the trained agent for intrusion detection; evaluate the intrusion detection results using intrusion detection classification performance metrics.

[0014] As a preferred technical solution, the Transformer model includes multiple encoding layers, each encoding layer including a multi-head attention layer and a feedforward neural network; the multi-head attention layer and the feedforward neural network are connected through residual connections and layer normalization connections.

[0015] As a preferred technical solution, step S2 includes:

[0016] Traffic features are divided into categorical features and data features. The categorical features are converted into fixed-length embedding vectors using an embedding layer, and the numerical features are normalized.

[0017] The embedding vectors of categorical features and numerical features are concatenated and input into the Transformer model. The learned attention weights adaptively capture the correlation information between the input traffic feature sequences to generate contextual embeddings.

[0018] The context is embedded through multiple hidden layers to further extract traffic features, and the output is a Q-value vector equal to the number of classification categories. Each Q-value vector corresponds to each classification category.

[0019] As a preferred technical solution, step S3 further includes: switching the experience replay pool switching window according to the set conditions, and storing the classification results in the first experience replay pool and the second experience replay pool respectively.

[0020] As a preferred technical solution, step S3 further includes: when the classification result is stored in the dual experience replay pool, whether to add the current experience to the second experience replay pool is determined according to the weight of each category.

[0021] As a preferred technical solution, when the classification results are stored in the dual experience replay pool, the weights of each category updated in the previous batch are called, the experience replay pool switching window is switched using the weights of each category, and the classification results are stored in the first experience replay pool or the second experience replay pool.

[0022] As a preferred technical solution, after completing a batch of neural network updates, the update loss of the current batch is compared with the update loss of the previous batch. If the update loss of the current batch increases, the distribution weights of various types of samples are updated to adjust the distribution quantity of various types of samples in the second experience replay pool.

[0023] As a preferred technical solution, when the weights of various sample distributions are updated adaptively, the weight of each category is calculated by counting the number of types of the maximum value in the classification results of the statistical traffic data. The weight of the category that appears more often will be reduced accordingly, and finally the weight values ​​of all categories are adjusted.

[0024] As a preferred technical solution, step S3 includes:

[0025] The intrusion detection environment compares the agent's classification results with the actual traffic data. If the agent's chosen classification action matches the actual situation, it receives a positive reward; if the agent's chosen action differs from the actual situation, it receives a negative reward.

[0026] As a preferred technical solution, step S5 includes:

[0027] The agent optimizes the value function based on reward feedback;

[0028] An improved deep Q-network is used to generate a list of Q-values ​​based on newly collected traffic data. The best action is selected based on the Q-values ​​to make intrusion detection decisions, and the model parameters are updated through the backpropagation algorithm to optimize the model's performance in intrusion detection tasks.

[0029] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0030] (1) By introducing a second experience replay pool, this invention is specifically designed for storing and managing a smaller number of more challenging minority class samples, effectively preventing minority class samples from being overwhelmed by majority class samples, thereby solving the sample imbalance problem in intrusion detection.

[0031] (2) The present invention sets an experience replay pool switching window so that the model can switch between two experience replay pools, avoiding over-focusing on one experience replay pool and causing forgetting of other experience replay pools, maintaining comprehensive learning of all categories, and helping to maintain the balance of the model among different types of samples.

[0032] (3) The distribution of samples of each category in the experience replay pool of this invention can be flexibly adjusted, so that the model pays more attention to the samples that are considered difficult at the current stage, thereby improving the learning effect of a few categories, enhancing the model's learning ability when facing imbalanced samples, and improving the generalization ability of the overall model.

[0033] (4) The Transformer model of structured data is used to extract and represent traffic features. This model has strong sequence modeling and context understanding capabilities, which helps to extract key features in traffic data more accurately and effectively, and improves the classification performance and robustness of the model. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of an intrusion detection method based on deep reinforcement learning and structured data Transformer according to an embodiment of the present invention;

[0036] Figure 2This is a scene diagram illustrating the interaction between the intelligent agent, the classifier, and the intrusion detection environment in an embodiment of the present invention.

[0037] Figure 3 This is a structural diagram of the Transformer model for structured data in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of the structure of the improved deep Q-network according to an embodiment of the present invention. Detailed Implementation

[0039] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0040] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0041] Deep Q-Network (DQN) is a reinforcement learning algorithm that combines the ideas of deep neural networks and Q-learning. It uses neural networks to approximate the Q-value function (i.e., the action-value function, which measures the value of taking an action in a specific state) to learn and make decisions about complex environments.

[0042] Please see Figure 1 The intrusion detection method based on deep reinforcement learning and structured data Transformer provided in this embodiment includes the following steps:

[0043] S1. Collect traffic data and input the dataset into the agent.

[0044] The traffic data includes network traffic data collected by network traffic data collection tools and data sources.

[0045] S2. The agent uses a classifier to classify the dataset and obtain the classification results of the traffic data; the classifier includes an improved deep Q network and a Transformer model, and the improved deep Q network includes a dual experience replay pool and an experience replay pool switching window.

[0046] In this implementation, to identify and distinguish between normal traffic and abnormal traffic of corresponding attack types, an intrusion detection model is constructed that involves interaction between an agent and an intrusion detection environment, such as... Figure 2 As shown, the intrusion detection environment is a customized reinforcement learning environment, made suitable for intrusion detection tasks through customized action space, state space, and reward function. The agent is the entity responsible for analyzing and classifying network traffic. Specifically, the agent needs to process and analyze the input network traffic data, extract network traffic features, convert it into a precise state space, and determine the available action space. In each time step where the agent interacts with the environment, the agent receives traffic data sampled from the environment and extracts relevant traffic features. After analyzing the traffic data, the agent classifies the currently input traffic data, classifying it as normal traffic or abnormal traffic of a corresponding attack type.

[0047] This embodiment combines an improved Deep Q-Network with a Transformer model to form a classifier, which the agent then uses to classify the current input traffic data. Specifically, the classifier uses a Transformer model to extract features and learn representations from the input data, and then uses an improved DQN for reinforcement learning training to obtain the classification results for the input traffic data.

[0048] Regarding feature extraction and representation learning using the Transformer model, the details are as follows:

[0049] In intrusion detection, structured data is usually presented in tabular form, containing multiple attributes such as source IP address, target IP address, protocol, etc., and corresponding labels such as normal traffic, abnormal traffic, and attack type. The Transformer model is used to model structured data such as network traffic or system logs and transform them into sequences for feature extraction and classification.

[0050] like Figure 3 As shown, the Transformer model includes multiple encoding layers, each of which includes a multi-head attention layer and a feedforward neural network; the multi-head attention layer and the feedforward neural network are connected through residual connections and layer normalization connections.

[0051] When extracting traffic features, the traffic features are first divided into categorical features and numerical features. Categorical features are converted into fixed-length vector representations using an embedding layer, while numerical features are normalized. The embedding vectors of the categorical features and the vectors of the numerical features are then concatenated and input into the Transformer model. Then, in the self-attention mechanism, the model adaptively captures the correlation information between the input traffic feature sequences through learned attention weights, thereby generating contextual embeddings. The formula for calculating the attention weights is as follows:

[0052]

[0053] Here, the query matrix Q represents the query matrix, which is used to characterize the traffic features at the current time step; the key matrix K (Key) and the value matrix V (Value) are the traffic features at different time steps in the input sequence.

[0054] Finally, after obtaining the context embedding, it is input into multiple hidden layers to further extract traffic features, and the final output is a Q-value vector equal to the number of classification categories, with each Q-value corresponding to each classification category.

[0055] S3. The intrusion detection environment compares the classification results of the agent with the true category and generates reward feedback; the reward feedback and newly collected traffic data are input into the agent for the agent's learning and decision optimization; the currently processed traffic data, the classification results of the traffic data, the reward feedback and the newly collected traffic data are combined into an experience quadruple (current state, current action, reward, next state) and stored in a dual experience replay pool; the dual experience replay pool includes a first experience replay pool and a second experience replay pool.

[0056] The intrusion detection environment compares the classification actions performed by the agent with the actual traffic data. If the agent's chosen classification action matches the actual situation, it receives a positive reward. If the agent's chosen action differs from the actual situation, resulting in a false positive or false negative, the agent receives a negative reward. Therefore, the intrusion detection agent optimizes its model based on the reward feedback, aiming to obtain as many positive rewards as possible to maximize the cumulative reward.

[0057] Then, the reward feedback and the newly collected traffic data are input into the agent for the agent's learning and decision optimization in step S5.

[0058] The specific steps for storing experience in the dual experience replay pools are as follows:

[0059] like Figure 4As shown on the left, this implementation uses an improved DQN to add a second experience replay pool. The experience in this pool is determined by the weights based on the proportion of samples of each category in the data, which determines whether to add them to the experience replay pool. A switching window for the experience replay pool is also set up, which can switch between the first and second experience replay pools.

[0060] When storing experience, the experience quadruple (current state, current action, reward, next state) is stored simultaneously in two experience replay pools. These two pools play different roles during the storage process. Specifically, the first pool stores a large number of experience categories, but these are relatively common. The second pool determines whether to add the current experience to its own pool based on the weight of each category, and switches the experience replay pool switching window accordingly, storing potentially difficult-to-classify minority category samples in the second pool.

[0061] In practice, to achieve window switching for the experience replay pool, all or one of the following steps can be used:

[0062] The experience replay pool switching window is switched according to the set conditions, and the classification results are stored in the first experience replay pool and the second experience replay pool respectively.

[0063] When the classification results are stored in the dual experience replay pool, the weight of each category determines whether to add the current experience to the second experience replay pool. When the classification results are stored in the dual experience replay pool, the weight of each category updated in the previous batch is called, the experience replay pool switching window is switched using the weight of each category, and the classification results are stored in the first experience replay pool or the second experience replay pool.

[0064] At this point, the first and second experience pools complete one round of training and store the corresponding experience, and then update the weights of each sample in this round of training.

[0065] S4. Repeat steps S1 to S3 until the stored data reaches the preset batch size, and then adaptively update the distribution weights of various samples.

[0066] In step S4, after a batch of neural network updates is completed, it is checked whether the loss of the current batch has increased compared to the previous batch. At this point, the dual empirical replay pool enters the empirical replay stage. If the loss has increased, it indicates the existence of potentially difficult-to-classify minority class samples. Therefore, weight updates will be performed to adjust the distribution of each class of samples in the second empirical replay pool. The weights of each class of samples in the second empirical replay pool are an adaptive dynamic update process, i.e., adaptive sample distribution. The update process updates the weights based on a given list of Q-values. This method calculates the weight corresponding to each class by counting the number of types with the highest Q-value in the list. The weight of the class that appears more frequently is reduced accordingly. Finally, the weight values ​​of all classes are adjusted.

[0067] In a more concrete example, assuming there are n traffic data samples and m discrete actions (representing different classification labels), the Q-value update formula can be summarized as follows:

[0068] (s,α)=(1-α)·Q(s,a)+α·(r+γ·maxa′Q(s′,a′))

[0069] s∈S,S={d1,s2,…s n}

[0070] a∈A,A={a1,a2,…a m}

[0071] Where s represents the feature information of the current network traffic, a represents the classification label predicted by the agent, r is the feedback on whether the agent's prediction is correct or not, γ is the discount factor used to measure the importance of future rewards, and s′ is the feature information of the next network traffic after the agent takes action a.

[0072] weight w i The update formula is:

[0073]

[0074] Where y represents the number of categories in the data, N represents the number of samples in each category in the Q-value list, and n i n represents the number of samples in class i that are currently undergoing weight updates. j This represents the number of samples in the j-th category.

[0075] S5. The intrusion detection environment uses reward feedback to update the agent's value function, and uses various samples from the dual experience replay pools to train the agent and update the agent's decision-making strategy; the agent selects various samples from the first or second experience replay pool through the experience replay pool switching window.

[0076] Step S5 involves replaying the experiences stored in the two experience recycling pools to train and update the agent's decision-making policy. (See also...) Figure 2 In the improved DQN algorithm, the Q-value vector represents the potential value of different actions. The best action can be selected based on the Q-value for intrusion detection decision-making, and the model parameters are updated through the backpropagation algorithm to optimize the model's performance in intrusion detection tasks.

[0077] During the update process, the agent continuously interacts with the intrusion detection environment, adapting its action space, state space, and reward function to suit the intrusion detection task. On one hand, the agent continuously optimizes the value function through reward feedback to guide the strategy; on the other hand, it continuously adjusts the sample distribution by switching between the first and second experience replay pools through an experience replay pool switching window, thereby optimizing the agent's decision-making capabilities.

[0078] S6. Repeat steps S1 to S5 until the agent reaches the set number of learning rounds and the agent completes training.

[0079] S7. Input the unknown traffic data into the trained agent for intrusion detection; evaluate the intrusion detection results using intrusion detection classification performance metrics.

[0080] The intrusion detection classification performance metrics mentioned above include accuracy, recall, precision, and F1 score.

[0081] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An intrusion detection method based on deep reinforcement learning and structured data Transformer, characterized in that, Includes the following steps: S1. Collect traffic data and input the dataset into the agent; S2. The agent uses a classifier to classify the dataset and obtain the classification results of the traffic data; the classifier includes an improved deep Q network and a Transformer model, and the improved deep Q network includes a dual experience replay pool and an experience replay pool switching window. S3. The intrusion detection environment compares the agent's classification results with the true category and generates reward feedback. The reward feedback and newly collected traffic data are input into the agent for the agent's learning and decision optimization; the currently processed traffic data, the classification results of the traffic data, the reward feedback and the newly collected traffic data are combined into an experience quadruple and stored in a dual experience replay pool. The dual experience replay pool includes a first experience replay pool and a second experience replay pool; S4. When the stored data reaches the preset batch size, perform adaptive updates of the distribution weights of various samples. S5. The intrusion detection environment uses reward feedback to update the agent's value function, and uses various samples from the dual experience replay pool to train the agent and update the agent's decision-making strategy. The agent selects various samples from the first or second experience replay pool through the experience replay pool switching window; S6. When the agent reaches the set number of learning rounds, the agent completes the training. S7. Input the unknown traffic data into the trained agent for intrusion detection; evaluate the intrusion detection results using intrusion detection classification performance metrics.

2. The intrusion detection method based on deep reinforcement learning and structured data Transformer as described in claim 1, characterized in that, The Transformer model includes multiple encoding layers, each of which includes a multi-head attention layer and a feedforward neural network; the multi-head attention layer and the feedforward neural network are connected through residual connections and layer normalization connections.

3. The intrusion detection method based on deep reinforcement learning and structured data Transformer as described in claim 1, characterized in that, Step S2 includes: Traffic features are divided into categorical features and data features. The categorical features are converted into fixed-length embedding vectors using an embedding layer, and the numerical features are normalized. The embedding vectors of categorical features and numerical features are concatenated and input into the Transformer model. The learned attention weights adaptively capture the correlation information between the input traffic feature sequences to generate contextual embeddings. The context is embedded through multiple hidden layers to further extract traffic features, and the output is a Q-value vector equal to the number of classification categories. Each Q-value vector corresponds to each classification category.

4. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 1, characterized in that, Step S3 further includes: switching the experience replay pool switching window according to the set conditions, and storing the classification results in the first experience replay pool and the second experience replay pool respectively.

5. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 1, characterized in that, Step S3 further includes: when the classification result is stored in the dual experience replay pool, whether to add the current experience to the second experience replay pool is determined according to the weight of each category.

6. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 5, characterized in that, When the classification results are stored in the dual experience replay pool, the weights of each category updated in the previous batch are called, the experience replay pool switching window is switched using the weights of each category, and the classification results are stored in the first experience replay pool or the second experience replay pool.

7. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 1, characterized in that, After a batch of neural network updates is completed, the update loss of the current batch is compared with the update loss of the previous batch. If the update loss of the current batch increases, the weights of the distribution of each type of sample are updated to adjust the number of samples of each type in the second experience replay pool.

8. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 7, characterized in that, When the weights of various sample distributions are updated adaptively, the weight of each category is calculated by counting the number of types of the maximum value in the classification results of the statistical traffic data. The weight of the category that appears more often will be reduced accordingly, and finally the weight values ​​of all categories are adjusted.

9. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 1, characterized in that, Step S3 includes: The intrusion detection environment compares the agent's classification results with the actual traffic data. If the agent's chosen classification action matches the actual situation, it receives a positive reward; if the agent's chosen action differs from the actual situation, it receives a negative reward.

10. The intrusion detection method based on deep reinforcement learning and structured data Transformer according to claim 1, characterized in that, Step S5 includes: The agent optimizes the value function based on reward feedback; An improved deep Q-network is used to generate a list of Q-values ​​based on newly collected traffic data. The best action is selected based on the Q-values ​​to make intrusion detection decisions, and the model parameters are updated through the backpropagation algorithm to optimize the model's performance in intrusion detection tasks.

Images