Network attack detection method based on deep reinforcement learning

Abstract

A network attack detection method based on deep reinforcement learning. The original dataset is preprocessed, and then the agent is constructed. This involves initializing the agent's environment, specifying the agent's interaction with the environment, and developing a training strategy and value function. Features are selected based on the state and fed into the detection model for prediction. The detection results are fed back to the agent training module as feedback, and the action's Q(s, a) is calculated, refreshing the Q-table. This process is repeated until the number of features in the optimal feature subset reaches a maximum, indicating model convergence, or until the training step is completed, generating the optimal feature subset. The processing method designed for new features reflects their importance to intrusion attack detection. If they are important, a dedicated optimal feature subset is deployed for the new feature. This demonstrates the flexibility of the optimal feature subset, allowing for autonomous responses to different attack scenarios.

Description

Technical Field

[0001] The present invention relates to a network attack detection method based on deep reinforcement learning, and belongs to the field of information security technology. Background Art

[0002] Numerous studies have been conducted on network attack detection methods, but most approaches neglect the feature processing of raw data and instead emphasize improving detection algorithms. Based on the principle that "data and features determine the upper limit of machine learning, while models and algorithms merely approximate this upper limit," this paper investigates the problem of feature processing of raw data. Feature selection methods extract the optimal feature subset from raw data and features by eliminating irrelevant, redundant, abnormal, and insignificant features. This approach further improves model training accuracy and reduces runtime and resource consumption, representing a search-based optimization problem. Traditional feature selection methods are primarily classified into three categories: filter, wrapper, and embedded methods. Specifically, these methods include Pearson correlation coefficient, chi-squared validation, distance measurement, and variance selection. These methods are primarily based on mathematical features and incorporate multidisciplinary approaches such as search technology and statistics. While significant research has been achieved, they also suffer from drawbacks, such as the relatively complex computational process, the exponential growth of data features due to the increasing dimensionality, and the inability to adapt to data development and changes, making them static methods. Therefore, in an era where data features are dynamically developing and changing, the optimal feature set should be selected flexibly. In addition, once the optimal feature set is determined, it should not remain unchanged. Instead, different methods should be proposed to optimize and update it based on the actual situation.

[0003] We are currently in the era of big data. Faced with massive amounts of high-dimensional data, traditional feature selection methods are no longer sufficient. With the advancement of technologies like data mining and machine learning, feature selection criteria have evolved beyond simply mathematically calculating correlations between individual data features. Furthermore, in dynamically changing network environments, an increasing number of intrusion attacks with novel features are emerging. Attackers can exploit these features to devise attacks that circumvent existing defenses and detection measures, resulting in serious information leakage. For example, in networks with dynamically changing topologies, such as in-vehicle networks and satellite communication networks, coupled with large amounts of training data, the search for an optimal feature set can be extremely slow, unable to keep pace with the pace of environmental change. Furthermore, if novel features are not considered, even if an optimal feature set is derived from existing features, attackers can still exploit these features to devise intrusion attacks and obtain sensitive data.

[0004] In summary, traditional feature selection methods can no longer select the optimal feature set, and thus cannot provide good defense. Later, some scholars proposed feature selection algorithms based on simple machine learning, feature selection based on traditional deep learning models, and several related improved algorithms to solve the security problems brought by massive high-dimensional features and new features. However, there are also certain defects. For example, Nugroho et al. analyzed and sorted out the performance of various machine learning algorithms in the study of intrusion detection in IoT devices in the past five years in "A Review of intrusion detection system in IoT with machine learning approach: current and future research." Among them, support vector machine (SVM) and artificial neural network (ANN) have the largest input to the intrusion classification process, and the output is also good, which can obtain good accuracy. Kilincer et al. conducted research on a variety of intrusion detection open source datasets in "Machine learning methods for cyber security intrusion detection: Datasets and comparative study.", using K-nearest neighbor (KNN) and decision tree (DT) algorithms for classification, and achieved more successful research results. This research is considered to be helpful in using machine learning methods on the basis of artificial intelligence to help study intrusion detection mechanisms; and Hosseini et al. conducted research on a variety of intrusion detection open source datasets in "Machine learning methods for cyber security intrusion detection: Datasets and comparative study.", using K-nearest neighbor (KNN) and decision tree (DT) algorithms for classification, and achieved more successful research results. This research is considered to be helpful in using machine learning methods on the basis of artificial intelligence to help study intrusion detection mechanisms; and Hosseini et al. conducted research on a variety of intrusion detection open source datasets in "Machine learning methods for cyber security intrusion detection: Datasets and comparative study.", using K-nearest neighbor (KNN) and decision tree (DT) algorithms for classification, and achieved more successful research results. In the paper, in order to improve the training effect and performance, SVM and ANN, ANN and decision tree (DT) were combined successively, and finally the feature dimension was successfully reduced and the training time was optimized.While these methods are fast, they cannot extract deep network data information, identify new cyberattacks, or be applied to highly volatile networks. In their paper "Deep learning-based framework for the detection of cyberattack using feature engineering," Akhtar et al. employed a convolutional neural network (CNN) classification model to detect DoS attacks, fully extracting data features and achieving high accuracy. In their paper "Deep transfer learning based intrusion detection system for electric vehicle networks," Mehedi et al. proposed a LeNet model based on deep transfer learning. Compared with mainstream machine learning, deep learning, and baseline deep transfer learning models, this model significantly improved intrusion detection accuracy and exhibited superior security performance. Deep learning utilizes multi-layer neural networks to extract deep features from raw data and identify cyberattacks through continuous iterative training. While this approach can effectively process massive amounts of data features to a certain extent, it is still a static model and suffers from limitations in handling unknown new features. Summary of the Invention

[0005] In order to effectively solve the selection problem of massive features and new unknown features, the present invention proposes a network attack detection method based on deep reinforcement learning, which is mainly used in the field of intrusion attack detection. The flexibility is reflected in the way of handling new features. The scheme first proposes a method of combining feature selection with anomaly detection, and uses the results of anomaly detection as feedback for the intelligent agent. The reward mechanism is designed based on the feedback. The intelligent agent sets a corresponding reward for each feature. In this way, after the model converges, the optimal feature set can be directly selected based on the training experience. At this time, we set this optimal feature set as a universal optimal feature subset (fixing the length of the set to max). Then, the present invention performs relevant processing on the new features, which can prevent attackers from leaking sensitive information through the new features. When a new feature appears, it is first assumed to be a member of the optimal feature subset (considering the correlation between the features, the new feature is taken as the first feature), and then repeat step 1 to select max-1 features to form the optimal feature subset. In this way, if the detection metric is significantly improved compared to the general optimal feature subset, this optimal feature subset is set as the exclusive optimal feature subset for "a new feature" or "a new intrusion attack centered on this new feature." Otherwise, the new feature is deemed to be of little research significance and can be temporarily ignored, with subsequent detection still using the general feature subset. Finally, because the algorithm combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, the optimal feature subset selected by the present invention is relatively flexible, allowing us to determine whether it makes sense to design a dedicated feature subset for a new feature. Therefore, it can be applied to dynamically changing network environments and detect new intrusion attacks.

[0006] The technical solution of the present invention:

[0007] A network attack detection method based on deep reinforcement learning, the steps are as follows:

[0008] (1) Construction of the environment state model of the feature selection agent;

[0009] The environment state model is the environment required by the agent, including the definition of the reward function and the design of the interaction feedback rules. The specific contents are as follows:

[0010] (1.1) First use U t To represent the discounted future cumulative reward obtained by the agent at time t, and then consider the specific application context;

[0011] Discounted Future Accumulated Rewards U t :The intelligent body can perceive the state of the environment and respond to the feedback signal r provided by the environment t, by learning to select actions, the discounted future accumulated rewards are maximized; due to the randomness of the environment, the randomness of the agent's state and the actions taken will continue to increase with the increase of the number of steps. Therefore, in order to reduce uncertainty and randomness, the discount factor γ is introduced to reduce the strong correlation between steps, and the discounted future accumulated rewards U t As a discounted future cumulative reward, its expression is:

[0012]

[0013] Among them, R t is the feedback received by the agent at time t, γ∈[0,1] is the discount coefficient used to increase the immediate reward rather than the delayed reward;

[0014] When γ is close to 0, it means that more emphasis is placed on current rewards; when γ is close to 1, it means that more emphasis is placed on future rewards. Since the application context here is intrusion detection, the network traffic is discrete and independent of each other, which is a discrete value problem of categorical data. Therefore, γ here should be as close to 0 as possible to reduce the continuity between network traffic.

[0015] (1.2) After defining the reward function, design the interactive feedback rules between the agent and the environment, that is, the reward mechanism;

[0016] Design a dual reward mechanism. Consider the evaluation indicators including the accuracy, precision, recall rate of the detection results, and the running time of the model. Design the following reward formula:

[0017]

[0018] Among them, ω represents the weight matrix for measuring the corresponding evaluation indicators, which is used to measure the importance of each evaluation indicator, set the preference and priority, etc.; R represents the reward matrix, and each evaluation indicator corresponds to a reward component; r_a represents the feedback of accuracy, r_p represents the feedback of precision, r_r represents the feedback of recall, and r_t represents the feedback of running time; it should be noted that the false alarm rate and missed alarm rate indicators are not considered in the formula. This is because they are linearly related to the precision and recall rates. However, if you want to study them separately, you can also add them to the formula and set the corresponding weights.

[0019] At each iteration, the newly selected features are added to the selected feature set. If the agent's training metrics using the new feature set decrease, the reward for the new features is set to -100 (to ensure that subsequent training of the agent can fully avoid this feature). If the metrics improve, first record the feedback corresponding to the improved detection metrics, namely accuracy r_a, precision r_p, recall r_r, and runtime r_t. Then, calculate the new reward based on the weights corresponding to each metric. For example, if accuracy increases to 90%, set r_a = 0.9. It should be noted that this can be handled flexibly according to the actual situation. If the accuracy of the detection results is of great concern, the weights corresponding to other metrics can be set to numbers close to 0, or even 0. If multiple detection metrics are of concern at the same time, then reasonable weights can be set according to the needs.

[0020] The advantage of this is that different scenarios have different importance and priorities for the three indicators, so it can be applied to a wider range of scenarios, and the evaluation criteria can be selected according to actual needs to improve generalization. It can also conduct separate research on a certain indicator, such as finding the feature subset with the highest accuracy.

[0021] (2) Construction of the value function of the feature selection agent;

[0022] The value function is the expectation of reward, which is mainly used to evaluate the quality of different states and guide the choice of action of the agent. The value function is also used to evaluate the quality of the agent in the state s at a certain time t. The specific content is as follows:

[0023] (2.1) First, we calculate the value function Q(s,a), which is used to evaluate the expected return of the current agent starting from state s, performing action a and obeying the policy ε. The formula is as follows:

[0024] Q π (s,a)=E π [U t |S t =s,A t =a]

[0025] Among them, S t Indicates the state of the agent at time t, A t represents the action performed by the agent at time t, E π The training strategy of the agent;

[0026] After obtaining the value of all possible actions a corresponding to the current state s, combined with the training strategy, the agent needs to select the optimal action, that is, based on the strategy π, take the maximum value among all Q(s,a), which is action a t The decision-making process is as follows:

[0027]

[0028] Among them, Q * (s t ,a t ) is the maximum value of all actions at the t-th time step;

[0029] (2.2) After constructing the value function, the agent regularly selects all possible actions in the current state according to the established strategy; therefore, the strategy is defined as follows:

[0030] In reinforcement learning, there are two crucial concepts: exploration and exploitation. The former refers to the agent selecting the optimal action from known actions based on the principle of "maximizing action value," while the latter refers to the agent selecting other unknown actions. Furthermore, in a given state, the agent can only perform one action; it cannot perform both simultaneously. Strategy is used to balance exploration and exploitation.

[0031] A greedy strategy is used, which means that when the agent makes a decision, there is a probability ∈(0<∈<1) of randomly selecting an unknown action, and the remaining probability 1-∈ is to select the action with the greatest value among the existing actions; when the agent selects a feature and finally adds it to the optimal feature set, it needs to remove it from the action space or reset its Q value, that is, reduce the Q value corresponding to the action that selects the feature as much as possible to ensure that the feature is not selected as much as possible in subsequent training.

[0032] Beneficial Effects of the Present Invention: While traditional feature selection methods are somewhat effective, they rely on extensive mathematical calculations, consuming significant computational power and failing to effectively address novel features. This, in particular, poses a range of security threats to intrusion attacks targeting novel feature structures. Therefore, the present invention proposes a network attack detection method based on deep reinforcement learning.

[0033] Deep reinforcement learning combines the massive data processing capabilities of deep learning with the flexible decision-making capabilities of reinforcement learning. Primarily built for reinforcement learning, it first defines the agent's environment and designs interaction rules and reward mechanisms. Next, it defines a value function and training strategy, providing a single criterion for the agent to select actions. Finally, through repeated iterative training, the agent selects features with high rewards and avoids those with low rewards, ultimately generating an optimal feature subset.

[0034] When faced with new features, they can be directly added to the optimal feature set, and the new optimal feature set can be used as training data to train the model again. Then, based on various detection indicators, it can be decided whether it is necessary to set a dedicated optimal feature subset for the new features / new intrusion attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a framework diagram of the intelligent agent detection model based on deep reinforcement learning described in the present invention.

[0036] Figure 2 This is a flowchart of the feature selection based on deep reinforcement learning described in the present invention.

[0037] Figure 3 This is a flow chart of processing novel features according to the present invention. DETAILED DESCRIPTION

[0038] In order to express the purpose, technical solutions and advantages of the present invention more clearly, the present invention will be further described in detail below through implementation steps and drawings.

[0039] A network attack detection method based on deep reinforcement learning includes how to preprocess raw data, how to reflect the interaction process between the intelligent agent and the environment, and how to process new feature processes.

[0040] Reference Figure 2 The specific operation process of how to reflect the interaction between the agent and the environment and select features is as follows:

[0041] Step 1. Initialize the environment in which the agent is located, define the state space to represent the selected features, the action space to represent the available actions for selecting features from the original features, set the training strategy and the maximum number of optimal feature subsets max.

[0042] Step 2. Select an action according to the greedy strategy. The action with the largest Q value at the current moment is selected with a probability of ∈, and the possible actions in the environment are randomly explored with a probability of 1-∈.

[0043] Step 3. After the agent performs the action, it obtains the state S at the next moment t+1 .

[0044] Step 4. The detection model predicts whether there is abnormal behavior based on the state characteristics currently selected by the agent and combined with the corresponding machine learning algorithm.

[0045] Step 5. Calculate the detection index based on the predicted value and record it as the agent's feedback. The formula is as follows:

[0046]

[0047] Where ω represents the weight matrix, which is used to measure the importance of each indicator, set preferences, and priorities; r_a, r_p, r_r, and r_t correspond to the accuracy, precision, recall, and running time of the detection results, respectively.

[0048] Step 6. Add the features corresponding to the state to the optimal feature subset.

[0049] Step 7. Calculate the Q value of the action at the previous moment. The formula is as follows:

[0050] Q π (s,a)=E π [U t |S t =s,A t =a]

[0051] Among them, π represents the training strategy, and the Q value represents the expectation of the discounted future accumulated rewards corresponding to the current state and action.

[0052] Step 8. Scan the optimal feature subset. If the number of features contained in it reaches the maximum value max, the model ends when it converges; otherwise, return to step 2 and repeat.

[0053] Reference Figure 3 , how to process new features, the specific operation process is as follows:

[0054] Step 9. Convert the new features into state variables through normalization, one-hot encoding, etc.

[0055] Step 10. Assume that it belongs to the optimal feature subset and add it to form new training data, which is input into the detection model.

[0056] Step 11. Observe whether the detection indicators (accuracy, precision, recall, and running time) are improved compared to the existing optimal feature set.

[0057] Step 12. If the detection index is improved, the new feature is assigned a label, marking the exclusive optimal feature subset of the new feature / new intrusion attack.

[0058] Step 13. If the detection index does not improve, the new features are considered to be of little significance, and subsequent tests can use the universal optimal feature subset.

Claims

1. A network attack detection method based on deep reinforcement learning, characterized in that: Here are the steps: (1) Construction of the environment state model of the feature selection agent; The environment state model is the environment required by the agent, including the definition of the reward function and the design of the interaction feedback rules. The specific contents are as follows: (1.1) First use U t To represent the discounted future cumulative reward obtained by the agent at time t, and then consider the specific application context; Discounted Future Accumulated Rewards U t :The intelligent body can perceive the state of the environment and respond to the feedback signal r provided by the environment t , by learning to select actions that maximize discounted future accumulated rewards; By introducing the discount coefficient γ to reduce the strong correlation between steps, the future cumulative reward U t As a discounted future cumulative reward, its expression is: Among them, R t is the feedback received by the agent at time t, γ∈[0,1] is the discount coefficient used to increase the immediate reward rather than the delayed reward; When γ is close to 0, it means that more emphasis is placed on current returns; when γ is close to 1, it means that more emphasis is placed on future returns; (1.2) After defining the reward function, design the interactive feedback rules between the agent and the environment, that is, the reward mechanism; Design a dual reward mechanism. Consider the evaluation indicators including the accuracy, precision, recall rate of the detection results, and the running time of the model. Design the following reward formula: Where ω represents the weight matrix of the corresponding evaluation indicators, which is used to measure the importance of each evaluation indicator, set the preference and priority; R represents the reward matrix, and each evaluation indicator corresponds to a reward component; r_a represents the feedback of accuracy, r_p represents the feedback of precision, r_r represents the feedback of recall, and r_t represents the feedback of running time; At each iteration, the newly selected features are added to the selected feature set. If the agent's performance using the new feature set decreases, the reward for the new features is set to -100. If the performance improves, the feedback corresponding to the improved detection indicators is first recorded, i.e., accuracy r_a, precision r_p, recall r_r, and running time r_t. Then, a new reward is calculated based on the weights of the indicators. (2) Construction of the value function of the feature selection agent; The value function is the expectation of reward, which is used to evaluate the quality of different states and guide the choice of actions of the agent. The value function is also used to evaluate the quality of the agent's state s at a certain time t. The specific content is as follows: (2.1) First, we calculate the value function Q(s,a), which is used to evaluate the expected return of the current agent starting from state s, performing action a and obeying the policy ε. The formula is as follows: Q π (s,a)=E π [U t |S t =s,A t =a] Among them, S t Indicates the state of the agent at time t, A t represents the action performed by the agent at time t, and the agent's training strategy π; After obtaining the value of all possible actions a corresponding to the current state s, combined with the training strategy, the agent needs to select the optimal action, that is, based on the strategy π, take the maximum value among all Q(s,a), which is action a t The decision-making process is as follows: Among them, Q * (s t ,a t ) is the maximum value of all actions at the tth moment; (2.2) After constructing the value function, the agent regularly selects all possible actions in the current state according to the established strategy; therefore, the strategy is defined as follows: A greedy strategy is used, which means that when the agent makes a decision, there is a probability of ∈ to randomly select an unknown action, 0<∈<1, and the remaining probability of 1-∈ is to select the action with the greatest value among the existing actions; when the agent selects a feature and finally adds it to the optimal feature set, it needs to remove it from the action space or reset its Q value, that is, the Q value corresponding to the action that selects the feature is reduced as much as possible to ensure that the feature is not selected as much as possible in subsequent training.

Images