An intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning

Abstract

The application belongs to the technical field of network security situation awareness, and discloses an intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning, which comprises the following steps: obtaining a network traffic dataset; preprocessing the obtained network traffic dataset to obtain a normalized dataset; constructing a deep attention Q network SEA-DDQN model based on double values, performing network security intrusion situation prediction ISP, and realizing the ISP process. The application adopts the above-mentioned intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning, continuously improves the prediction ability through continuous interaction with the traffic environment, so that the model can flexibly cope with complex and constantly changing threat scenarios; SEA-DDQN utilizes the advantages of reinforcement learning to optimize the adaptability and decision-making process of the model, thereby enhancing the anti-interference ability of the model in a dynamic environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security situation awareness, and in particular to an intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning. BACKGROUND

[0002] With the increasing complexity and diversity of network threats, the importance of cyber security situation awareness (CSA) is increasingly prominent. As a core component of CSA, situation prediction bears the key responsibility of identifying and responding to potential security threats in the future. Traditional machine learning (ML) methods usually rely on supervised learning on network security data to predict and identify security incidents. However, these methods often face the challenges of insufficient adaptability and limited generalization ability when dealing with dynamic network environments and their changing attack patterns, which limits their effectiveness in real-time monitoring and emergency response.

[0003] Therefore, in order to solve the problems of dependence on labeled data, insufficient prediction ability for new attacks, poor adaptability in dynamic environments, and increased computational resource consumption and reduced training efficiency caused by "state explosion" when dealing with large-scale data sets due to the high dimensionality of attack behavior characteristics in existing supervised learning and unsupervised learning methods for intrusion situation prediction (ISP), the present application proposes a deep attention Q network (SEA-DDQN) adaptive reinforcement learning framework based on double value to predict network security intrusion situation. SUMMARY

[0004] The purpose of the present application is to provide an intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning, which continuously improves its prediction ability through continuous interaction with the traffic environment, enabling the model to flexibly cope with complex and changing threat scenarios; SEA-DDQN utilizes the advantages of reinforcement learning to optimize the adaptability and decision-making process of the model, thereby enhancing the model's anti-interference ability in dynamic environments.

[0005] To achieve the above purpose, the present application provides an intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning, comprising the following steps:

[0006] Step S1, obtaining NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 network traffic data sets;

[0007] Step S2, preprocessing the obtained network traffic data set to obtain a normalized data set;

[0008] Step S3, constructing a deep attention Q network SEA-DDQN model based on double value, and performing network security intrusion situation prediction ISP;

[0009] Step S4, realizing the ISP process based on the SEA-DDQN model.

[0010] Preferably, in step S2, the obtained network traffic data set is preprocessed to obtain a normalized data set, and the specific process is as follows:

[0011] Step S21, defining the network traffic data set as , wherein is the number of traffic samples, and the feature space is , as follows:

[0012] ;

[0013] , wherein represents a feature, represents an index of the number of features in the data set; represents a feature subset; ; is a traffic feature subset; is a content feature subset; is a time statistical feature subset; is a host behavior feature subset; represents a union symbol, and represents that the four feature subsets are combined into a feature space ;

[0014] Step S22, performing hot one-hot encoding processing on discrete features ;

[0015] First, the discrete feature subset ; wherein represents a discrete feature; represents a discrete feature index; the corresponding value range of each discrete feature is respectively: ; ; ; wherein , and respectively represent the value range of the discrete feature, ; for any discrete feature, define an indicator function , as follows:

[0016] ;

[0017] , wherein ; represents a sample in the discrete feature the value of the feature vector

[0018] Then, the feature space is reconstructed by the hot one-hot encoding transformation That is , as follows:

[0019] ;

[0020] where d is the total number of features after hot one-hot encoding processing; represents the hot one-hot encoding processing function; represents the vector splicing operation; is the index of all non-discrete features in the original feature; is the discrete feature subset;

[0021] Finally, the original training set and test set are respectively subjected to hot one-hot encoding to generate their respective extended feature matrices , and are normalized, as follows:

[0022] ;

[0023] where, represents the normalized feature; and represent the maximum and minimum values of the feature ; is an indicator function, which takes the value 1 when , otherwise 0; the attack categories in the data set are classified and mapped into a set of digital categories .

[0024] Preferably, based on the NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 training sets and test sets after data preprocessing, the network ISP reinforcement learning environment is constructed and defined; the ISP process is abstracted as a Markov decision process MDP, and is formally defined as a four-tuple , as follows:

[0025] ;

[0026] where the state is a set of feature vector spaces of each network flow; the action is a set of flow type decisions defined in NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 at time The state and action of the agent are represented as and ; the reward the reward or punishment obtained by the agent for predicting the traffic correctly or not; the calculation of the reward vector is based on the comparison between the prediction result of the agent and the actual label to evaluate the accuracy of the prediction of the agent; the discount factor a discount coefficient for balancing the immediate reward and long-term return.

[0027] Preferably, the SEA-DDQN model agent is constructed to include an action selection network and a target Q value network; the SEA-DDQN model agent performs prediction and target Q value calculation on the captured network traffic;

[0028] the action selection network and the target Q value network The Q value function of the action selection network and the target Q value network is respectively composed of a four-layer fully connected feedforward neural network, and an attention mechanism and a ReLu activation function are used between the fully connected layers, wherein each of the two hidden layers contains 128 neurons;

[0029] The state of the SEA-DDQN model agent at each time step is represented by a feature vector of the network traffic, and the state space of the agent is defined as Therefore, the SEA-DDQN model agent obtains the state vector of each piece of network traffic as follows:

[0030] ;

[0031] wherein each dimension represents a feature of the network traffic.

[0032] Preferably, an attention mechanism is introduced to optimize the attention degree of the SEA-DDQN model agent to the network traffic features, enhance the discriminative features and suppress the irrelevant features, and the specific process is as follows:

[0033] For the network traffic feature state vector input at each time step, after processing by the fully connected layer, a 128-dimensional feature vector is obtained, wherein represents the network traffic feature vector processed by the fully connected layer, represents a dimension index; then it is sent to the attention layer to enhance the attention degree to important features, and the attention weight of the attention layer is as follows:

[0034] ;

[0035] wherein, is a dimension reduction projection matrix; is a dimension increasing reconstruction matrix; denotes the dimension reduction ratio, which is a hyperparameter; denotes the Sigmoid gating function, which compresses the input value to the interval [0, 1], and the generated attention weight is used to measure the importance of each feature; the generated attention weight is fused with the original feature by modulation, as follows:

[0036] ;

[0037] wherein, denotes the fused feature representation; is the Hadamard product, i.e., the element-wise multiplication operation; this operation allows each feature value to be dynamically adjusted by the corresponding weight .

[0038] Preferably, the SEA-DDQN model agent generates an action list according to the input features of the deep neural network, and the list is presented in the form of an action vector; the final Q value is used to evaluate whether the attack behavior is successfully predicted or not;

[0039] First, the class labels in the NSL-KDD, UNSW-NB15, CICIDS-2017, and MQTT-IoT-IDS2020 datasets are respectively mapped into a set of digital classes, as follows:

[0040] ; wherein, 0 represents Normal; 1 represents DoS; 2 represents Probe; 3 represents U2R; 4 represents R2L;

[0041] ; wherein, 0 represents normal traffic; 1 represents abnormal traffic;

[0042] ; wherein, 0 represents Bots; 1 represents Brute Force; 2 represents DDos; 3 represents Dos; 4 represents Normal; 5 represents Port Scanning; 6 represents Web Attacks;

[0043] ; wherein, 0 represents Normal; 1 represents Bruteforce; 2 represents Scan_A; 3 represents Scan_sU; 4 represents Sparta;

[0044] Then, the action space of the SEA-DDQN model agent is defined as:

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] where each action corresponds to a predicted class decision in the action list; represents the class index.

[0050] Preferably, a dynamic weight reward mechanism is designed to balance class bias by constructing a class-sensitive reward and determine the class weight using a reverse frequency weighting strategy to enhance the sensitivity of the SEA-DDQN model agent to different attack classes, balance the influence of class bias in the dataset on the learning process, and the specific process is as follows:

[0051] First, define the specific form of the reward function as follows:

[0052] ;

[0053] where represents the weight of the attack class; represents correct prediction; represents incorrect prediction;

[0054] Then, use the reverse frequency weighting strategy to assign weights by considering the frequency of each attack class, as follows:

[0055] ;

[0056] ;

[0057] where represents the attack class weight set of ; represents the weight of attack; represents the weight of no attack; represents the weight of attack; represents the weight of attack; represents the weight of attack; represents the attack class weight set of ; represents the weight of attack; represents the weight of weight of attack; represents weight of attack; represents weight of attack; represents weight of attack;

[0058] The weight rule is updated as follows:

[0059] ;

[0060] wherein, is the total number of samples, is the number of attack types, represents the total number of attack samples;

[0061] For the two-prediction task, a positive reward is given when the SEA-DDQN model agent correctly predicts the sample, and a negative penalty is given otherwise, as follows:

[0062] ;

[0063] wherein, represents reward setting rule for each flow in the data set;

[0064] To simulate the delay feedback scenario in the real network, the n-step reward accumulation mechanism is adopted, and the reward actually stored in the experience replay pool is the n-step cumulative reward , rather than the single-step immediate reward ; the agent stores the transition sequence of the last n steps during the interaction with the environment, and calculates the discounted cumulative reward after n steps , as follows:

[0065] .

[0066] Preferably, the priority sampling strategy PER is adopted to optimize the sampling process of the experience network flow. Due to the importance difference of the attack categories, TD-Error is introduced as a key indicator to measure the importance of samples, and by defining the priority update formula, the dynamic adjustment of the sample sampling probability is realized, as follows:

[0067] ;

[0068] wherein, represents the probability of the th experience network flow being sampled; is a constant, which is used to ensure that all experience network flows have a probability of being sampled, avoiding that some samples have a sampling probability of zero due to too small TD-Error; This represents the current network traffic count in the experience replay pool. This is a priority parameter used to control the degree of influence of TD-Error on the sampling probability; For the first The TD-Error corresponding to each empirical network traffic is shown below:

[0069] ;

[0070] in, Represents the target Q-value network; Indicates the first One state; Indicates the first Actions in each state; This represents the Q-value estimate of the target network; This represents the maximum value of the Q-value estimate for the target network; Indicates the first One state; Indicates the first Actions in each state; This represents the Q-value estimate of the current network.

[0071] Preferably, in step S4, the ISP process is implemented based on the SEA-DDQN model, including the following steps:

[0072] Step S41: Based on the SEA-DDQN model, convert the state feature vector space of each network traffic sample into a single vector. As an action selection network The input is used to capture the characteristics of each traffic sample through hierarchical nonlinear transformation;

[0073] Step S42: In the input layer, convert the network traffic sample state feature vector space... The d-dimensional feature vectors in the model are mapped to a 128-dimensional hidden space as follows:

[0074] ;

[0075] in, This represents the preactivation vector of the first layer of the neural network; This represents the weights of the first layer of the neural network; This indicates the bias of the second layer of the neural network;

[0076] Step S43: Through the attention layer and using The activation function implements a non-linear transformation, enabling the neural network to learn more complex representations of traffic features, as shown below:

[0077] ;

[0078] ;

[0079] wherein, is element-wise multiplication, used to combine the weighted results of attention mechanism with original features; denotes the result after ReLU activation function, representing the nonlinearly transformed feature representation learned by the network;

[0080] Step S44, the second layer fully connected layer and the subsequent layers and the above layers are the same, as follows:

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] wherein, denotes the pre-activation vector of the second layer neural network; denotes the weight of the second layer neural network; denotes the bias of the second layer neural network; denotes the result after ReLU activation function of the second layer;

[0086] Step S45, in the last output layer, the Q value of each action in the action space of the agent is generated , whose output value represents the expected reward accumulation of the corresponding action under the current state , as follows:

[0087] ;

[0088] wherein, is the weight of the fourth layer neural network; is the result after ReLU activation function of the third layer, representing the nonlinearly transformed feature representation learned by the network; is the bias of the fourth layer neural network; is the Q value of the first action in the action space element vector; denotes the Q value estimation of the network; is the expected cumulative reward corresponding to the element in the action space ;

[0089] For the Q value vector obtained by the action selection network for the current state , the agent adopts The policy to make the best prediction action selection as follows:

[0090] ;

[0091] wherein, represents the size of the prediction action space; represents the optimal prediction action under the current state ; represents the exploration rate of the current time step; represents the probability of selecting an action under the state ;

[0092] The update rule of is as follows:

[0093] ;

[0094] wherein, is the initial exploration rate, is the minimum exploration rate threshold, is the decay coefficient, is the current learning time step.

[0095] The calculation process of the target Q value network is the same as that of the action selection network , and the input is , and the target Q value is calculated as follows:

[0096] ;

[0097] wherein, is the reward obtained by taking action under the current state ; is the target network Q value;

[0098] Based on PER, the sampling probability is assigned according to the importance of experience, the sampling bias is introduced, and the mean square error loss function with importance sampling weight is constructed to compensate for the bias introduced by non-uniform sampling, as follows:

[0099] ;

[0100] wherein, represents the loss function; represents the bth flow in the experience replay pool; represents the state; represents the action; represents the state and action The predicted Q value; This represents the target q value of the b-th flow; This indicates the probability that the b-th traffic item will be sampled. Indicates the importance sampling coefficient; The number of network traffic samples drawn from the experience replay pool;

[0101] In the initial stage of learning, as learning progresses, The value gradually increases and eventually approaches 1, as shown below:

[0102] ;

[0103] in, This represents the value of β at time step t; This represents the value of β at time step t-1; This represents the initial value of β; This represents the final target value for the β value; the β value is a parameter used to control the degree of forgetting or retention of past experiences.

[0104] In action selection network Medium parameters Parameters are updated in real time through backpropagation. The update formula is as follows:

[0105] ;

[0106] in, Represents the parameters of the neural network; Indicates the learning rate; Indicates parameters The gradient;

[0107] Target Q-value network parameters Updates are achieved by combining Polyak average soft updates and hard updates, specifically by performing an action selection network every 100 time steps. Medium parameters Network with target Q value parameters The parameters of the target Q-value network are updated by performing a weighted average; the action selection network is updated every 1000 time steps. Medium parameters Directly copy to the network with the target Q value parameters In the middle, as shown below:

[0108] ;

[0109] in, parameters representing the target q-value network; for average soft update coefficient.

[0110] Therefore, the present application adopts the above-mentioned one kind based on SEA-DDQN adaptive reinforcement learning's invasion situation prediction method, beneficial effects are as follows:

[0111] (1) The present application adopts the architecture combining DDQN and attention mechanism, through separating agent, the captured network flow is predicted and the target Q value is calculated, the overestimation deviation of agent in predicting network flow is reduced. At the same time, the attention mechanism is introduced to optimize the attention degree distribution of agent to network flow characteristics, enhance discriminative features and suppress irrelevant features.

[0112] (2) The present application introduces n-step reward accumulation mechanism and lag reward distribution strategy to simulate the inherent delay of threat confirmation in real network environment. In this way, the agent must learn to predict the future cumulative return based on the historical state sequence, so as to realize the forward prediction and autonomous decision of potential invasion situation, instead of relying on immediate supervision signal.

[0113] (3) The present application combines the priority experience replay (PER) strategy mechanism, introduces the time difference error (TD-Error) as the importance index of experience network flow sample, and preferentially uses the experience network flow sample with large TD error for learning, so as to preferentially use those experience network flow samples with the greatest influence on the current strategy for updating, and improve the ISP efficiency.

[0114] (4) The present application uses benchmark NSL-KDD, UNSW-NB15, CICIDS-2017 and NSL-KDD, UNSW-NB15 data set to evaluate SEA-DDQN, and compares the results with other mainstream research methods. The experimental results show that the prediction accuracy of SEA-DDQN is higher. This shows that SEA-DDQN has higher accuracy and robustness in processing complex network flow data. BRIEF DESCRIPTION OF DRAWINGS

[0115] Figure 1 is the ISP model implementation flowchart of the present application;

[0116] Figure 2 is the DRL agent environment interaction flowchart of the present application;

[0117] Figure 3 is the flowchart of the feature processing process of the attention mechanism of the present application;

[0118] Figure 4 is the flowchart of the priority experience sampling process of the present application;

[0119] Figure 5 is a flow chart of the ISP prediction attack class decision process of the present application;

[0120] Figure 6 is a graph of the accuracy and exploration rate under different discount factors in an embodiment of the present application;

[0121] Figure 7 is a comparison graph of the average loss under different discount factors in an embodiment of the present application;

[0122] Figure 8 is an accuracy distribution graph of the last 50 rounds under different discount factors in an embodiment of the present application; wherein the orange part represents γ = 0.5; the blue part represents γ = 0.005; and the green part represents γ = 0.9;

[0123] Figure 9 is a confusion matrix graph under different discount factors in an embodiment of the present application; wherein (a) is a confusion matrix with a discount factor of 0.9; (b) is a confusion matrix with a discount factor of 0.5; and (c) is a confusion matrix with a discount factor of 0.005;

[0124] Figure 10 is an accuracy comparison graph under different parameter values of Batch-size & β_final in an embodiment of the present application;

[0125] Figure 11 is an ablation comparison graph in an embodiment of the present application; wherein (a) is a comparison of F1-Score of each class; and (b) is a comparison of overall performance;

[0126] Figure 12 is a minority class comparison graph in an embodiment of the present application; wherein (a) is a performance comparison of R2L class; and (b) is a performance comparison of R2L class (reduce key class);

[0127] Figure 13 is a confidence curve graph of different attack types in an embodiment of the present application;

[0128] Figure 14 is a performance box plot in an embodiment of the present application; wherein different colors represent different attack classes; cyan green represents DoS; apricot orange represents Normal; mist blue represents Probe; rose pink represents R2L; and tender yellow green represents U2R;

[0129] Figure 15 is a training loss comparison graph under different random seeds in an embodiment of the present application;

[0130] Figure 16 is a 95% confidence interval graph under different random seeds in an embodiment of the present application;

[0131] Figure 17is the NSL-KDD exploration graph in the embodiment of the application; wherein, (a) is the accuracy rate and exploration rate of NSL-KDD data set varying with training rounds; (b) is the average loss of NSL-KDD with training rounds;

[0132] Figure 18 is the UNSW-NB15 exploration graph in the embodiment of the application; wherein, (a) is the accuracy rate and exploration rate of UNSW-NB15 varying with training rounds; (b) is the average loss of UNSW-NB15 with rounds.

[0133] Figure 19 is the CICIDS2017 exploration graph in the embodiment of the application; wherein, (a) is the accuracy rate and exploration rate of CICIDS2017 varying with training rounds; (b) is the average loss of CICIDS2017 with rounds.

[0134] Figure 20 is the MQTT-IoT-IDS2020 exploration graph in the embodiment of the application; wherein, (a) is the accuracy rate and exploration rate of MQTT-IoT-IDS2020 varying with training rounds; (b) is the average loss of MQTT-IoT-IDS2020 with rounds. DETAILED DESCRIPTION

[0135] The technical solutions of the application are further described below through the drawings and embodiments.

[0136] Embodiment 1

[0137] As shown in Figure 1 , a method for intrusion situation prediction based on SEA-DDQN adaptive reinforcement learning comprises the following steps:

[0138] Step S1, obtaining NSL-KDD, UNSW-NB15, CICIDS-2017 and NSL-KDD, UNSW-NB15 network flow data sets.

[0139] Obtaining NSL-KDD, UNSW-NB15, CICIDS-2017 and NSL-KDD, UNSW-NB15 network flow data sets. Among them, the NSL-KDD data set and the UNSW-NB15 data set contain 41 features and 1 category label, the CICIDS-2017 data set contains 52 features and 1 category label, the MQTT-IoT-IDS2020 data set contains 31 features and one category label, as shown in Table 1, Table 2, Table 3 and Table 4 respectively. The features cover basic network connection attributes, content features and flow-based statistical information, which can fully reflect the characteristics of network flow.

[0140] The NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020-Biflow data sets used in the present application contain 148517, 257673, 524288 and 173371 network traffic records respectively. Among them, the CICIDS-2017 data set originally containing 1048576 records is reduced by half through stratified sampling method, ensuring that the sample proportion of each category can be highly consistent with the original data set.

[0141] Table 1 NSL-KDD data set characteristics

[0142] ;

[0143] Table 2 UNSW-NB15 data set characteristics

[0144] ;

[0145] Table 3 CICIDS-2017 data set characteristics

[0146] ;

[0147] Table 4 MQTT-IoT-IDS2020 data set characteristics

[0148] ;

[0149] Step S2, preprocessing the obtained network traffic data set to obtain a normalized data set.

[0150] The preprocessing methods of NSL-KDD, UNSW-NB15, CICIDS-2017 and NSL-KDD, UNSW-NB15 data sets are consistent. The present embodiment takes the NSL-KDD data set as an example, and the specific processing process is as follows:

[0151] Step S21, define the network traffic data set as , wherein is the number of traffic samples, and the feature space is , as shown below:

[0152] ;

[0153] , wherein represents a feature subset, ; represents a feature, ; represents a union symbol, and represents that the four feature subsets are combined into a feature space ;

[0154] is a flow feature subset, which provides a direct description of the data packet transmission duration and the communication protocol adopted from the most basic network connection level, is the basic element to understand the network behavior pattern, and contains , and other features.

[0155] is a content feature subset, which reflects the data interaction scale and directionality of the communication parties by quantifying the actual transmission content in the data packet, and contains , and other features.

[0156] is a time statistical feature subset, which summarizes and analyzes the network traffic from the time dimension, reveals the frequency of network connection and the activity status of the server within a specific time window, and contains , and other features.

[0157] is a host behavior feature subset, which focuses on the network behavior pattern of a specific host, provides the connection frequency and data transmission volume of the destination host, and contains and other features.

[0158] Step S22, for discrete features , , , hot one-hot encoding processing is performed.

[0159] First, the discrete feature subset ; wherein, represents a discrete feature; represents a discrete feature index. The corresponding value range of each discrete feature is respectively: ; ; ; wherein, , and respectively represent the value range of the discrete feature, . For any discrete feature, define the indicator function as follows:

[0160] ;

[0161] wherein, ; represents the value of the sample in the discrete feature .

[0162] Then, through hot one-hot encoding transformation , that is , the feature space is reconstructed as follows:

[0163] ;

[0164] where d is the total number of features after the hot one encoding process; represents the hot one encoding processing function; represents the vector splicing operation; is the index of all non-discrete features in the original 41-dimensional feature; is the discrete feature subset.

[0165] Finally, the original training set and test set are respectively hot one encoded to generate their respective extended feature matrices , and normalized as follows:

[0166] ;

[0167] where, represents the normalized feature; and respectively represent the maximum and minimum values of the feature ; is an indicator function, which takes the value 1 when , otherwise 0.

[0168] The NSL-KDD data set contains 59 attack categories, which are divided into 5 main categories and mapped into a set of digital categories , which is convenient for the model to process, as shown in Table 5.

[0169] Table 5 NSL-KDD data set record categories

[0170] ;

[0171] In Table 5, the normal traffic has no attack behavior, which is classified as Normal. Attacks such as Back and Worm occupy system or network resources through a large number of legal requests, making normal users unable to obtain services, which are classified as Dos. Attacks such as Nmap and Ipsweep frequently perform port scanning and IP scanning, which are classified as Probe. Attacks such as Snmpguess and tp_write obtain access permissions to local systems by remotely sending data packets, which are classified as R2L. Attacks such as Perl and Rootkit are local users who elevate their own permissions to root users through system vulnerabilities, which are classified as U2R.

[0172] Similarly, in the CICIDS-2017 and MQTT-IoT-IDS2020-Biflow datasets, they are divided into 7 main categories and 5 main categories and mapped into a set of digital categories, the original record categories of CICIDS-2017 and the record categories used in this study are shown in Tables 6 and 7, respectively, and MQTT-IoT-IDS2020-Biflow is shown in Table 8. In the UNSW-NB15 dataset, it is two-predicted, and its record categories are shown in Table 9.

[0173] Table 6 Record categories of CICIDS-2017 original dataset

[0174] ;

[0175] Table 7 Record categories of CICIDS-2017 dataset of the present application

[0176] ;

[0177] Table 8 Record categories of MQTT-IoT-IDS2020-Biflow dataset

[0178] ;

[0179] Table 9 Record categories of UNSW-NB15 dataset

[0180] ;

[0181] Step S3, a deep attention Q network (SEA-DDQN) model based on double value is constructed, and network security intrusion situation prediction (ISP) is performed.

[0182] Step S31, based on the NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 training set and test set after data preprocessing, the network ISP reinforcement learning environment is constructed and defined.

[0183] The ISP process is abstracted as a Markov decision process MDP, and it is formally defined as a four-tuple , as follows:

[0184] ;

[0185] wherein the state is a set of feature vector spaces for each network flow; the action is a set of flow type decisions defined in NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020, at a specific time The state and action of the agent are denoted as and respectively. The reward is a scalar signal that the agent receives for predicting traffic correctly or incorrectly.

[0186] Since each traffic has independence, the MDP is reduced to a conditional reward maximization problem. Figure 2 The basic process of the agent interacting with the environment in the deep reinforcement learning used in the present application is shown in

[0187] Step S32, constructing a SEA-DDQN model agent, including an action selection network and a target Q value network.

[0188] The SEA-DDQN model agent includes two key neural networks, namely the action selection network and the target Q value network. The SEA-DDQN model agent performs prediction (i.e. action selection) and target Q value calculation on the captured network traffic.

[0189] The Q value functions of the action selection network and the target Q value network are respectively composed of four layers of fully connected feedforward neural networks, and attention mechanisms and ReLu activation functions are used between the fully connected layers, wherein each of the two hidden layers contains 128 neurons, as shown below:

[0190] ;

[0191] wherein, denotes the action selection network or the target network; is the fourth fully connected layer; is the third fully connected layer; is the second fully connected layer; is the first fully connected layer; denotes inputting the state S into the first fully connected layer; is the Relu layer; is the attention layer.

[0192] In this application, the features other than the category label in the four data sets used by the application are used as the state representation of the SEA-DDQN model agent; the category label is used to calculate the reward vector based on the model prediction result. The calculation of the reward vector is based on the comparison between the prediction result of the agent and the actual label, and the agent only obtains the action for calculating the reward vector, without performing any actual action on the environment. The action selection of the agent is only used to evaluate the accuracy of its prediction, and the feedback is given to the SEA-DDQN model agent through the reward function to guide its learning process.

[0193] The state of the SEA-DDQN model agent at each time step is represented by the feature vector of network traffic , and the state space of the agent is defined as Therefore, the state vector of the SEA-DDQN model agent for each piece of network traffic is obtained as follows:

[0194] ;

[0195] where each dimension represents a certain feature of the network traffic.

[0196] Step S33, introduce attention mechanism to optimize the attention allocation of the SEA-DDQN model agent to the network traffic features, enhance discriminative features and suppress irrelevant features.

[0197] Step S331, for the input network traffic feature state vector at each time step, after processing by the fully connected layer, a 128-dimensional feature vector is obtained, where represents the network traffic feature vector processed by the fully connected layer, represents the dimension index; then it is sent to the attention layer to enhance the attention to important features, and the attention weight is as follows:

[0198] ;

[0199] where is the dimension reduction projection matrix, which projects the original feature vector to a low-dimensional space to reduce the computational complexity and extract the main feature trend; is the dimension lifting reconstruction matrix, which is used to map the dimension-reduced features back to the original feature space to generate attention weights matching the original feature dimensions; represents the dimension reduction ratio, which is a hyperparameter; represents the Sigmoid gating function, as follows:

[0200] ;

[0201] wherein, represents a linear combination calculated by the previous layer of the model; is a Sigmoid function.

[0202] Here, the Sigmoid gating function compresses the input value to the interval , and the generated attention weight can be regarded as the original "attention coefficient" of the feature, which is used to measure the importance of each feature.

[0203] Step S332, the generated attention weight is fused with the original feature by modulation, as follows:

[0204] ;

[0205] wherein, represents the fused feature representation; is the Hadamard product, that is, the element-wise multiplication operation.

[0206] This operation makes each feature value be dynamically adjusted by the corresponding weight . Since , the feature with a larger weight will be enhanced, while the feature with a smaller weight will be weakened. The whole attention mechanism processing feature process is as shown in Figure 3 .

[0207] Step S34, the SEA-DDQN model agent generates an action list according to the input feature of the deep neural network, which is presented in the form of an action vector; the final Q value is used to evaluate whether the attack behavior is successfully predicted.

[0208] Step S341, first, respectively map the class labels in NSL-KDD, UNSW-NB15, CICIDS-2017, and MQTT-IoT-IDS2020 data sets into a set of digital categories, as follows:

[0209] ; wherein, 0 represents Normal; 1 represents DoS; 2 represents Probe; 3 represents U2R; and 4 represents R2L.

[0210] ; wherein, 0 represents normal traffic; and 1 represents abnormal traffic.

[0211] ; wherein 0 represents Bots; 1 represents Brute Force; 2 represents DDos; 3 represents Dos; 4 represents Normal; 5 represents Port Scanning; and 6 represents Web Attacks.

[0212] ; wherein 0 represents Normal; 1 represents Bruteforce; 2 represents Scan_A; 3 represents Scan_sU; and 4 represents Sparta.

[0213] Step S342, the action space of the SEA-DDQN model agent is defined as:

[0214] ;

[0215] ;

[0216] ;

[0217] ;

[0218] wherein each action corresponds to a predicted category decision in the action list; represents a category index.

[0219] Step S35, a dynamic weight reward mechanism is designed, and a category-sensitive reward is constructed to effectively cope with the imbalance of different category attack data.

[0220] In reinforcement learning, the design of the reward function is crucial for guiding the agent to learn the optimal strategy. The present application proposes a dynamic reward function design method, which balances the category bias by constructing a category-sensitive reward, and determines the category weight by using the inverse frequency weighting strategy to enhance the sensitivity of the SEA-DDQN model agent to different attack categories, and balance the influence of category bias in the data set on the learning process.

[0221] Step S351, the specific form of the reward function is as follows:

[0222] ;

[0223] wherein, represents the weight of the attack category; represents correct prediction; represents incorrect prediction.

[0224] Step S352, the reverse frequency weighting strategy, by considering the frequency of each attack category, the lower frequency category gets higher weight, so as to give more attention in the reward function to balance the category bias, as follows:

[0225] ;

[0226] ;

[0227] wherein, represents a set of attack category weights of represents the weight of attack represents the weight of attack represents the weight of attack represents the weight of attack represents a set of attack category weights of represents the weight of attack represents the weight of attack represents the weight of attack represents the weight of attack represents the weight of attack .

[0228] The weight rule is updated as follows:

[0229] ;

[0230] wherein, is the total number of samples, is the number of attack types, represents the total number of attack samples.

[0231] Through this design, the reward function can accurately feedback the predicted behavior of the agent, thereby improving the prediction accuracy of different attack category samples.

[0232] Since UNSW-NB15 is a two-prediction task in this study, a positive reward is given when the SEA-DDQN model agent correctly predicts the sample, otherwise a negative penalty is given, as follows:

[0233] ;

[0234] in, express The reward setting rules for each traffic item in the dataset.

[0235] To simulate latency feedback scenarios in real-world networks, an n-step reward accumulation mechanism is adopted, and the actual reward stored in the experience replay pool is the n-step accumulated reward. Instead of one-step instant rewards The agent stores the transition sequence of the most recent n steps during its interaction with the environment, and calculates the discounted cumulative reward after reaching n steps. As shown below:

[0236] .

[0237] Step S36: Based on the Priority Experience Playback (PER) strategy mechanism, the Time Differential Error (TD-Error) is introduced as an indicator of the importance of experience network traffic samples. Experience network traffic samples with large TD errors are used for learning first, thereby prioritizing the use of experience network traffic samples that have the greatest influence on the current strategy for updating, thus improving ISP efficiency.

[0238] During the interaction between the agent and the environment, experiences (states, actions, rewards, and the next state) are stored in an experience replay pool and periodically replayed for learning. This approach balances the effects of rewards and penalties by weighting samples of different categories, enabling the agent to learn the characteristics of different attack categories more evenly.

[0239] A Priority Sampling (PER) strategy is adopted to optimize the sampling process of empirical network traffic. Taking into full account the differences in importance of different attack categories in NSL-KDD and CICIDS-2017, TD-Error is introduced as a key indicator to measure sample importance. A priority update formula is defined to dynamically adjust the sample sampling probability, as shown below:

[0240] ;

[0241] in, Indicates the first The probability that a given network traffic sample will be taken; It is a small constant used to ensure that all empirical network traffic has a certain probability of being sampled, and to avoid some samples having a zero sampling probability due to the TD-Error being too small; This represents the current network traffic count in the experience replay pool. This is a priority parameter used to control the degree of influence of TD-Error on the sampling probability; For the first TD-Error corresponding to an experience network flow, as shown below:

[0242]

[0243] wherein, denotes a target Q-value network; denotes a first state; denotes an action of the first state; denotes a second state; denotes an action of the second state; denotes a Q-value estimate of the target network; denotes a maximum value of the Q-value estimate of the target network; denotes a third state; denotes an action of the third state; denotes a fourth state; denotes an action of the fourth state; denotes a Q-value estimate of the current network.

[0244] The priority experience sampling process is as shown in Figure 4 .

[0245] Step S4, based on the SEA-DDQN model, an ISP process is implemented.

[0246] Step S41, based on the SEA-DDQN model, a state feature vector space of each network flow sample is taken as an input of an action selection network , and features of each flow sample are captured through hierarchical nonlinear transformation, as shown below:

[0247]

[0248] wherein, denotes the action selection network; denotes an input of the first fully connected layer.

[0249] Step S42, a d-dimensional feature vector in the state feature vector space of the network flow sample is mapped to a 128-dimensional hidden space in the input layer, as shown below:

[0250]

[0251] wherein, denotes a pre-activation vector of the first neural network; denotes a weight of the first neural network; denotes a bias of the second neural network.

[0252] Step S43, through an attention layer and using ​​​The activation function realizes a nonlinear transformation, so that the neural network learns more complex traffic feature representation, as follows:

[0253]

[0254]

[0255] where, is an element-wise multiplication, which is used to combine the weighted results of the attention mechanism with the original features; represents the result after ReLU activation function processing, which represents the feature representation learned by the network after nonlinear transformation.

[0256] Step S44, the second layer full connection layer and the subsequent layers and the above layers are the same, as follows:

[0257]

[0258]

[0259]

[0260]

[0261] where, represents the pre-activation vector of the second layer neural network; represents the weight of the second layer neural network; represents the bias of the second layer neural network; represents the result after ReLU activation function processing of the second layer.

[0262] Step S45, in the last output layer, the Q value of each corresponding to the agent action space is generated, that is, the 5 neurons of the output layer correspond to 5 possible actions, and the output value represents the expected reward accumulation of the corresponding action under the current state , as follows:

[0263]

[0264] where, is the weight of the fourth layer neural network; is the result after ReLU activation function processing of the third layer, which represents the feature representation learned by the network after nonlinear transformation; is the bias of the fourth layer neural network; is the Q value of the first action in the action space element vector; represents the Q value estimation of the network; is the Q value of the corresponding action space ​​​​​​​Expected cumulative reward of the middle element.

[0265] For the current state Passing through the action selection network The resulting Q-value vector, the agent adopts The policy to select the best predicted action, as follows:

[0266] ;

[0267] Wherein, Indicates the size of the predicted action space; Indicates the optimal predicted action under the current state ; Indicates the exploration rate of the current time step; Indicates the probability of selecting an action In state The entire predicted attack class decision process, as shown in Figure 5 .

[0268] In the reinforcement learning process, as the agent accumulates experience, it is hoped that it will gradually reduce the frequency of exploration and increase the proportion of utilization. Let the exploration rate decay over time, then The update rule is as follows:

[0269] ;

[0270] Wherein, Is the initial exploration rate, Is the minimum exploration rate threshold, Is the decay coefficient, Is the current learning time step.

[0271] In the early stage of learning, the agent will select a random predicted action with a probability of As learning progresses, Decay exponentially. This can ensure that the agent mainly relies on the learned strategy for prediction in the later training period, and does not completely lose the ability to explore, thereby maintaining a stable balance between exploration and utilization in the long term.

[0272] Step S46, the role of the target Q-value network Is to provide a stable target value for training the action selection network , which provides a target Q-value by calculating the Q-value of the next state , so as to realize the approximation of the Q-value function.

[0273] The calculation process of the target Q-value network The calculation process of the action selection network , its input is , the target Q value is calculated as follows:

[0274] ;

[0275] wherein, is the current state action is taken the reward obtained; is the target network Q value.

[0276] Since PER assigns sampling probabilities according to the importance of experience, which leads to a sampling distribution different from the original experience distribution, introducing sampling bias, the loss function of the present application is the mean square error loss with importance sampling weight to compensate for the bias introduced by non-uniform sampling, as follows:

[0277] ;

[0278] wherein, denotes the loss function; denotes the bth flow in the experience replay pool; denotes the state; denotes the action; denotes the predicted Q value of the state and the action of the bth flow by the action selection network; denotes the target q value of the bth flow; denotes the probability of the bth flow being sampled; denotes the importance sampling coefficient; is the number of network flows drawn from the experience replay pool; at this time, in is .

[0279] In the initial stage of learning, in order to reduce the weight influence of high priority network flows in the experience pool, the value is relatively low. As learning proceeds, the value gradually increases and eventually approaches 1, as follows:

[0280] ;

[0281] wherein, denotes the value of β at time step t; denotes the value of β at time step t-1 (i.e. the previous time step); denotes the initial value of β; denotes the final target value of β; β is a parameter for controlling the degree of "forgetting" or "retaining" of past experience (experience pool).

[0282] Step S48: The network used to calculate the target Q value in this invention. and They have the same structure, but their parameters are updated differently. A Polyak average soft update strategy and a hard update strategy are used to smoothly update the target network parameters, reducing the instability caused by gradient fluctuations during the learning process.

[0283] In action selection network Medium parameters Parameters are updated in real time through backpropagation. The update formula is as follows:

[0284] ;

[0285] in, Represents the parameters of the neural network; Indicates the learning rate; Indicates parameters The gradient.

[0286] In order for the target Q-value network to effectively keep up with the learning progress of the action selection network while maintaining stability, the target Q-value network... parameters Updates are achieved by combining Polyak average soft updates and hard updates, specifically by performing an action selection network every 100 time steps. Medium parameters Network with target Q value parameters The parameters of the target Q-value network are updated using a weighted average; the action selection network is updated every 1000 time steps. Medium parameters Directly copy to the network with the target Q value parameters In the middle, as shown below:

[0287] ;

[0288] in, The parameters of the target q-value network are represented. for Average soft update coefficient.

[0289] Example 2

[0290] The embodiment carries out multi-prediction task on NSL-KDD, CICIDS-2017 dataset, and two-prediction task on UNSW-NNB15. In the context of multi-prediction and two-prediction, TP (true positive) refers to the amount of positive samples correctly predicted by the agent, FP (false positive) refers to the amount of negative samples incorrectly predicted as positive, TN (true negative) refers to the amount of samples correctly predicted as negative, and FN refers to the amount of negative samples incorrectly predicted as positive.

[0291] TP, FP, TN and FN are used to calculate more interpretable performance indicators such as accuracy, precision, recall and F1 score, so as to comprehensively and quantitatively evaluate the performance of the agent.

[0292] In the performance evaluation system of the prediction task, accuracy As the most commonly used measure, its essence is to reflect the proportion of correct prediction by the agent in the whole sample. Specifically, it is the ratio of the number of samples correctly predicted by the agent to the total number of samples, as follows:

[0293] ;

[0294] Precision is defined as the proportion of samples that actually belong to a certain positive class in the sample set predicted by the agent to belong to that positive class, as follows:

[0295] ;

[0296] Recall is defined as the proportion of samples correctly predicted by the agent as positive among all actual positive samples, as follows:

[0297] ;

[0298] F1 score As a comprehensive performance indicator, it is defined as the harmonic mean of precision and recall. It comprehensively measures the accuracy and integrity of the agent in predicting positive samples. By integrating the information of precision and recall, F1 score can comprehensively reflect the balanced performance of the agent in predicting positive and negative samples, as follows:

[0299] ;

[0300] To provide reference and lessons for subsequent research, in this study, the parameters of the proposed SEA-DDQN model are shown in Table 10. These parameters are determined after systematic research and optimization, and they play a crucial role in the training and testing process of the model. Through experimental verification, the combination of these parameters performs excellently on the specific task and data set targeted by this study, effectively balancing the exploration and utilization capabilities of the model, and improving the accuracy and stability of its decision-making.

[0301] Table 10 SEA-DDQN model parameter table

[0302] ;

[0303] I. Sensitivity analysis of discount factor γ and Batch-size & β_final

[0304] The value of the discount factor γ has a decisive influence on the time horizon and decision-making strategy of the reinforcement learning agent. To demonstrate the selection of the γ parameter in the proposed model, in-depth sensitivity experiments were conducted on the NSL-KDD dataset, comparing three different γ values: γ = 0.9 (long-term planning), γ = 0.5 (medium-term planning, main setting), and γ = 0.005 (extremely short-sighted). The final accuracy on the test set for γ = 0.5 was 97.50%, for γ = 0.005 was 97.14%, and for γ = 0.9 was 96.75%. The experimental results consistently show that γ = 0.5 achieves the best balance in model performance and learning stability. Given that this parameter outperforms the other two groups on NSL-KDD, it is directly migrated to UNSW-NB15 and CICIDS-2017, further verifying the cross-dataset robustness of γ = 0.5.

[0305] From Figure 6 , Figure 7 and Figure 8 , it is clear that γ = 0.5 is the undisputed optimal choice. It not only achieves the highest final performance, but more importantly, this achievement is realized through the most stable and efficient learning process. In the learning process of γ = 0.5, its average loss decreases the fastest and smoothest, indicating an efficient and stable learning process. From Figure 8It can be seen that in the last 50 episodes, the exploration rate is close to 0.1, and the median line in the box is obviously higher than the other two groups, indicating that in more than 50% of the tests, its accuracy is the highest among the three. The box is very short, which means that 50% of the test results are concentrated in a very narrow high-precision interval (97.10%-97.50%). This is a sign of high stability and reliability of performance. The upper and lower lines are short and have no outliers, further proving that its output is very consistent and there is no performance anomaly or surge. This shows that γ = 0.5 enables it to quickly learn the reward and plan for long-term benefits.

[0306] The "short-sighted" strategy of γ = 0.005, although the performance is acceptable, the learning process is unstable, and its performance upper limit cannot surpass γ = 0.5. In the learning process of γ = 0.005, its average loss curve fluctuates significantly in the later period, indicating that its learning strategy is unstable. From Figure 8 It can be seen that in the last 50 episodes, the exploration rate is close to 0.1, and the median accuracy line is lower than γ = 0.5 but higher than γ = 0.9. The box is significantly longer than the box of γ = 0.5. This means that its 50% test results are distributed in a wider range (96.80%-97.15%). This indicates that its performance is volatile and not stable. The model learned by the "short-sighted" strategy has poor robustness, and its performance is more susceptible to changes in test samples, although it can sometimes achieve good results, but it is unreliable.

[0307] γ = 0.9 is significantly behind in terms of final performance due to low learning efficiency. In the learning process of γ = 0.9, its average loss value decreases the slowest and is always the highest, indicating that its learning process is inefficient. From Figure 8 It can be seen that in the last 50 episodes, the exploration rate is close to 0.1, and the median line in the box is obviously higher than the other two groups, indicating that in more than 50% of the tests, its accuracy is the highest among the three. The box is very short, which means that 50% of the test results are concentrated in a very narrow high-precision interval (97.10%-97.50%). This is a sign of high stability and reliability of performance. The upper and lower lines are short and have no outliers, further proving that its output is very consistent and there is no performance anomaly or surge. This shows that γ = 0.5 enables it to quickly learn the reward and plan for long-term benefits.

[0308] In contrast, choosing γ = 0.5 exhibits the best overall performance. It not only achieves the highest median accuracy, but more importantly, its box is short and compact, with symmetrical upper and lower lines and no outliers. This indicates that the performance of the model is highly concentrated and stable, with excellent robustness. A stable model is crucial in real network security applications, as its output must be reliable and predictable.

[0309] Figure 9The confusion matrix and detailed results in Table 11 reveal the profound impact of the discount factor γ on model behavior, particularly its ability to detect a few attack categories. The γ=0.9 setting exhibits mediocre performance across all categories due to low learning efficiency, especially failing completely in detecting U2R attacks (Recall: 16.42%), demonstrating that an overly "far-sighted" strategy is ineffective in this task. The γ=0.005 setting exhibits "short-sighted" behavior. While its overall accuracy (97.14%) is deceptively high, in-depth analysis reveals that its U2R attack detection strategy is conservative and flawed, trading high prediction accuracy for extremely high false negative rates (Recall only 47.76%). This is an unacceptable strategy in practical security applications. Its ability to detect R2L attacks shows a significant degradation, with recall dropping sharply to 84.00%, proving its inability to learn complex patterns requiring multi-step associations. This confirms that extremely low γ values ​​reduce the agent to a short-sighted classifier, fundamentally conflicting with the goal of simulating attack evolution. Conversely, choosing γ=0.5 yielded outstanding balanced performance, significantly improving detection of the most critical advanced threats (U2R), nearly doubling recall, and achieving the only available F1-Score. It achieved best performance in R2L attack detection, achieving a perfect balance between high recall and high precision. Simultaneously, it maintained top performance in most categories. This demonstrates that the "mid-range vision" provided by γ=0.5 is crucial for the agent to learn to detect various attack patterns, from simple to complex, and to make optimal trade-offs.

[0310] Table 11 Detailed Indicators under Different Discount Factors

[0311] ;

[0312] To verify the impact of Batch-size and β_final on model accuracy in Table 11, training was performed with the remaining hyperparameters set to 10w, 20w, and 24w. Figure 10 As can be seen, the three curves almost completely overlap. The average accuracies for the 10w, 20w, and 24w configurations are 97.47%, 97.46%, and 97.49%, respectively, with a standard deviation of only 0.013% and a maximum difference of no more than 0.03%. These results demonstrate that, within the examined value range, variations in Batch-size and β_final have a negligible impact on the final accuracy of the model, verifying the robustness of the proposed method to this hyperparameter setting.

[0313] II. Ablation Experiment.

[0314] To evaluate the contribution of each key component in the proposed network intrusion detection model, an ablation experiment on the NSL-KDD dataset was designed. The attack class inverse frequency weight mechanism and the PER module were removed respectively, and compared with the complete model. The experiment evaluated the overall performance and the performance of each category, focusing on the detection ability of the model for minority class attacks (U2R, R2L).

[0315] As shown in Table 12 and Figure 11 It can be seen that the complete model achieved the best performance in accuracy, precision, recall and F1-Score. Removing the class weight or PER will cause the overall performance of the model to decline, and removing the PER mechanism has the most significant impact on the overall performance (accuracy decreases by 0.87%), which indicates that PER plays a key role in improving the overall learning efficiency and stability of the model. For the majority or more common categories such as DoS, Normal, Probe, the three model variants all maintained a high level of performance (F1-Score is higher than 0.96), and the difference between them is small.

[0316] Table 12 Comparison of overall performance indicators of the complete model, removing class weight and removing PER mechanism

[0317] ;

[0318] Although the overall performance gap seems small, the performance difference in each category, especially in the minority class attacks, reveals the importance of each component. U2R and R2L are the rarest attack types in the data and the most difficult to detect categories, and their F1-Score comparison shows that the complete model has a significant advantage over the other two model variants. Figure 12As shown. For the U2R category, the full model achieved the highest F1-Score (0.5825), with a precision of 0.8333 significantly higher than the recall of 0.4478, indicating that the model's prediction results for this category are very reliable, but there is still room for improvement in detection capability. After removing the class weights, the model's precision decreased by 0.7143, causing the F1-Score to drop to 0.5505. This indicates that the class weight module effectively ensures that the model learns minority class features, preventing them from being overwhelmed by the majority class. After removing PER, the model maintained a high precision (0.8113), but the F1-Score was similar to the full model (0.5767). This indicates that PER has a positive effect on improving the detection stability of rare attacks such as U2R, but it is not the most critical factor. For the R2L category, the full model achieved the highest F1-Score (0.9417) in the R2L category, achieving a balance between high precision (0.9182) and high recall (0.9664). Removing class weights decreased the model's recall, with the F1-Score dropping to 0.9243. This indicates that without class weights, the model's ability to detect R2L attacks weakens. Removing PER resulted in a relatively smaller performance drop (F1-Score: 0.9383), suggesting that for R2L attacks, class weights are more critical than PER.

[0319] Ablation studies reveal that the class weight module is a core component for improving the model's performance in detecting minority class attacks (such as U2R and R2L). Its main function is to prevent the model training process from being dominated by the majority class, ensuring that the loss function effectively reflects the classification error of minority class samples, thereby significantly improving the recall and F1-score for the minority class. The PER mechanism is key to improving the overall performance and stability of the model. By focusing on those "hard-to-learn" samples (usually misclassified or uncertain samples, often belonging to the minority class), the PER mechanism effectively accelerates the model's convergence speed and improves overall performance metrics. While its optimization of minority class performance is not as direct as that of class weights, it provides an important supplement. The complete model integrates class weights and the PER mechanism, achieving an optimal balance between overall performance and performance across classes, especially in the detection of the crucial minority class, validating the effectiveness and necessity of the model design.

[0320] III. Sensitivity analysis of random seeds and evaluation of model stability.

[0321] To verify the potential impact of random seeds on model reproducibility, this study systematically evaluates the influence of different random initialization seeds on model performance on the NSL-KDD dataset, demonstrating the reliability and stability of the results. Experiments were repeated on the standard test set using five different random seeds (42, 2023, 2024, 3407, 12345), and statistical analysis was performed on overall and category-specific performance metrics.

[0322] The model of this invention exhibits high repeatability and stability in performance across the entire system and most categories. Figure 13 , Figure 14 As shown in Table 13, the overall classification accuracy of the model exhibits extremely high stability under different random seeds. Its mean is as high as 97.60%, with a very low standard deviation (0.15%) and a very narrow 95% confidence interval ([97.46%, 97.73%]). This indicates that the overall performance of the model is not sensitive to the choice of random seed and has excellent reproducibility. For the four attack categories with large sample sizes—DoS, Normal, Probe, and R2L—the standard deviation of the accuracy is less than 1%, the confidence intervals are concentrated, and the box plot distribution is compact. This proves that the model's ability to identify these major categories is reliable and consistent. Due to the extremely small number of samples in the U2R category in the training data (severe class imbalance), its performance is most sensitive to the randomness of model initialization, with a standard deviation of accuracy as high as 11.06%. However, despite the fluctuations in absolute values, its performance is statistically significantly better than random guessing, and multiple experiments have provided an expected range of its performance (mean 46.74%, 95% CI [37.04%, 56.43%]), which provides a reliable benchmark for subsequent research.

[0323] Table 13 Statistical analysis of model performance under different random seeds

[0324] ;

[0325] To rigorously determine the convergence of model training, this invention proposes a "double threshold" criterion: the exploration rate ε must first drop below 0.1, and the average difference in consecutive losses over the last 50 episodes must be less than 0.03. This criterion's design takes into account both the "exploration-exploitation" game dynamics of reinforcement learning and the local fluctuations in the loss function. First, ε < 0.1 ensures that the policy has entered the "exploitation"-dominated phase. At this point, the agent's traversal of the state space almost stops, and subsequent weight updates mainly rely on the first-order information of the policy gradient. If training continues, the incremental gain comes only from small perturbations, and their marginal contribution to the final performance can be ignored. Second, requiring the average loss fluctuation over 50 consecutive episodes to be below 0.03 is to statistically eliminate Gaussian noise introduced by stochastic gradients: such as... Figure 15 , Figure 16As shown, the average loss range of all random seeds within 95% confidence interval does not exceed 0.0256 (maximum appears in seed 2023), indicating that the loss sequence has entered a stationary phase, and there is no trend decline or periodic oscillation; the threshold of 0.03 is about 1.2 times the maximum observed range, which can accommodate the difference between seeds and environmental randomness.

[0326] Four, CICIDS-2017, NSL-KDD, MQTT-IoT-IDS2020 multi-ISP and UNNSW-NB15 two-ISP.

[0327] For the multi-ISP analysis of the CICIDS2017 dataset, Table 14 shows that the method proposed by the present application maintains strong and balanced detection performance in the seven types of traffic. Specifically, the F1-score of DDoS and DoS attacks is as high as 99.46% and 98.47%, respectively, and the recall rate is close to or exceeds 99%, indicating almost no false negatives for large-scale blocking attacks; Brute Force also achieves an F1 of 97.63%, with an accuracy of 98.44% and a recall of 96.82%; Port Scanning maintains an accuracy of 90.37% with a high sensitivity of 99.75% recall, with an F1 of 94.83%, effectively balancing the risk of false positives; Web Attacks and Normal traffic are stable in the F1 interval of 90% and 98%, reflecting the robust characterization of complex application layer anomalies and normal behavior; only Bots has an evaluation of 67.78% due to the scarcity of samples, becoming the only short board, but the remaining six categories are better than the same period reported in the field, verifying the wide applicability and reliability of the model in multi-attack scenarios.

[0328] Table 14 CICIDS2017 multi-prediction

[0329] ;

[0330] Table 15 shows the performance of representative methods for multi-classification tasks on the CICIDS2017 dataset. The accuracy of mainstream research has generally exceeded 97%. Among them, Zihan W et al. lead with the highest accuracy of 99.35% and F1-score of 99.17%; Yaser Alhasawi et al. and Jieling L et al. also follow closely behind with F1-scores of 98.90% and 98.54%, respectively, showing balanced detection capabilities for each type of attack. In contrast, W. Elmasry et al. achieved an accuracy of 98.95%, but Precision and Recall both dropped to about 95.8%, indicating that the recall for the minority class is still insufficient. The method of the present application maintains a high accuracy of 98.14% while stabilizing Precision, Recall and F1-score at around 98.2%, with a difference of less than 1 percentage point from the optimal result, verifying its effectiveness and competitiveness in overall detection performance and class balance.

[0331] Table 15 CICIDS2017 overall multi-prediction comparison

[0332] ;

[0333] The multi-ISP experimental results for the NSL-KDD dataset presented in Table 16 show that SEA-DDQN can achieve efficient and accurate prediction in complex network attack scenarios. Specifically, the model performs particularly well in DoS attack detection, with an accuracy of 98.71%, precision of 99.19%, recall of 99.52%, and F1-score of 99.35%, indicating that it can identify DoS attacks with extremely high confidence, providing reliable technical support for preventing such attacks. In the detection of Probe attacks, the model's accuracy is 94.54%, with precision, recall, and F1-score of 95.10%, 99.38%, and 97.19%, respectively, showing strong identification capabilities for Probe attacks and effectively detecting potential network reconnaissance behaviors in the early detection stage. For the more complex R2L attacks, the model's accuracy is 88.98%, with precision, recall, and F1-score of 91.82%, 96.64%, and 94.17%, respectively, although the accuracy is relatively low, the recall is high, indicating that the model can capture most of the real attack samples when identifying R2L attacks, which helps to take protective measures in a timely manner. In the prediction of normal traffic, the model's accuracy, precision, recall, and F1-score are 94.78%, 98.56%, 96.12%, and 97.32%, respectively, which can also accurately distinguish between normal network behavior and various attack behaviors.

[0334] Table 16 NSL-KDD multi-prediction

[0335] ;

[0336] Table 17 shows that compared with existing research, the schemes of C. P. R. Kanna et al. and Chadia E L A et al. achieve 98.67% and 98.92% in accuracy, respectively, but the former sacrifices part of the precision at the cost of 100% recall, and the latter falls back to 95.44% in recall, resulting in no absolute advantage in F1-score; the works of Zhendong W et al. and Wei et al. also achieve 98.60% and 92.95% in accuracy, respectively, but their precision or recall fluctuates significantly, and the comprehensive F1-score is lower than 98%. In contrast, the method in this paper maintains a highly consistent balanced performance in 97.50% accuracy, 97.54% precision, 97.50% recall, and 97.48% F1-score, neither over-biased to a certain index nor robust in overall performance.

[0337] Table 17 Comparison of NSL-KDD overall multi-prediction

[0338] ;

[0339] Table 18 compares the two ISP results of SEA-DDQN on the UNSW-NB15 dataset with existing research. From the key indicators of accuracy, precision, recall, and F1-score, most of the literature presents a clear trade-off between recall and precision: Yousefnezhad et al. push the F1-score to 93.34% with an extremely high recall of 99.72%, but the precision drops to 87.37%; Zhendong W et al., Li Jieling et al., and Marwa K et al. also sacrifice precision to achieve more than 90% recall, resulting in F1-score hovering between 84% and 89%. In contrast, the method in this paper achieves a high consistency of four indicators in 95.32% accuracy, 95.47% precision, 95.32% recall, and 95.30% F1-score, indicating that SEA-DDQN can maintain extremely low false positive rate while still fully capturing attack samples, achieving balanced and robust two-class detection performance.

[0340] Table 18 Comparison of UNSW-NB15 two-prediction

[0341] ;

[0342] The multi-ISP experimental results of the MQTT-IoT-IDS2020 dataset presented in Table 19 show that the proposed model achieves excellent and balanced detection performance for all attack types: Sparta and Scan_A almost reach full marks in the four indicators, with F1-score of 100% and 99.99%, respectively; Normal traffic is also stably identified, with F1-score of 99.66%; Scan_sU and Bruteforce are relatively slightly lower, with F1-score of 99.48% and 98.33%, respectively, fully verifying the strong generalization ability and robustness of the model in complex MQTT Internet of Things environment.

[0343] Table 19 MQTT-IoT-IDS2020 Multi-Prediction

[0344] ;

[0345] Table 20 shows that compared with existing research, the existing methods generally have a significant gap in the four indicators: Khan et al. and Ullah-DT et al. have an overall accuracy of more than 98%, but the recall rate drops to 86.71% and 82.06%, respectively, causing the F1-score to stay around 90%; Pandey et al.'s precision and recall simultaneously drop to about 87%, and F1 further drops to 85.64%; Lucia et al.'s recall rate is as high as 99.17%, but the overall accuracy is missing and the precision is only 92.14%, with F1-score of 95.53%. In contrast, Otokwala et al., Shirodkar and Akbar et al. have pushed a single indicator to 100% or more than 99.5%, but still have a slight gap in recall or precision. The method in this paper locks the accuracy, precision, recall and F1-score at 99.59% at the same time, while maintaining high detection sensitivity, it completely eliminates the trade-off defect of false positives and false negatives, and is better than the best existing results.

[0346] Table 20 MQTT-IoT-IDS2020 Overall Multi-Prediction Comparison

[0347] ;

[0348] V. Agent Exploration Analysis

[0349] During the experiment, the agent continuously adjusts its strategy through interaction with network traffic to achieve the best balance between exploration and exploitation. This study records the accuracy and exploration rate changes of the model at each training stage on the test set in detail. Figures 17-20In the learning process of the four data sets NSL-KDD, UNSW-NB15, CICIDS2017 and MQTT-IoT-IDS2020 respectively, the exploration rate of the agent on all data sets presents a monotone decreasing trend, and the accuracy of traffic prediction continues to improve with the training, and finally tends to be convergent, which gradually changes from extensive exploration in the early stage to stable utilization in the later stage, reflecting that the agent effectively identifies the high return action in the strategy space and forms a stable decision strategy. In addition, the loss function curve on all data sets presents a downward trend, and there is slight shock in the early stage of loss, which is mainly caused by the unstable strategy in the exploration stage. With the decrease of exploration rate, the loss curve gradually becomes smooth, which embodies the controllability and stability of the strategy convergence process. The agent on the four types of ISP data sets shows a reasonable exploration-exploitation trade-off mechanism, which can effectively avoid local optimum in the early stage of training and stably improve the detection performance in the later stage.

[0350] In summary, the application continuously enhances the adaptability and optimization of the model through continuous interaction with the traffic environment, improves the feature discrimination ability through the combination of attention mechanism, and improves the learning efficiency by focusing on samples with large information through priority experience replay, which has higher accuracy and robustness in processing complex network traffic data.

[0351] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application but not to limit it, although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that: it can still modify or equivalently replace the technical solutions of the present application, and these modifications or equivalent replacements also cannot make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. An intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning, characterized in that, The method comprises the following steps: Step S1, obtaining NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 network traffic data sets; Step S2, preprocessing the obtained network traffic data set to obtain a normalized data set; Step S3, constructing a deep attention Q network SEA-DDQN model based on double values to perform network security intrusion situation prediction ISP; Step S31, based on the NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 training set and test set after data preprocessing, constructing and defining a network ISP reinforcement learning environment; The ISP process is abstracted as a Markov Decision Process (MDP) and is formally defined as a four-tuple As follows: ； wherein the state is a set of feature vector spaces for each network traffic; the action is a set of traffic type decisions defined in NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020, at time The state and action of the agent are denoted as and respectively; the reward is the reward or penalty obtained by the agent for predicting the traffic correctly or not; the calculation of the reward vector evaluates the accuracy of the agent’s prediction based on the comparison between the agent’s prediction and the actual label; the discount factor is a discount coefficient used to balance the immediate reward and the long-term return; Step S32, constructing a SEA-DDQN model agent, including an action selection network and a target Q value network; the SEA-DDQN model agent performs prediction and target Q value calculation on the captured network traffic; Action selection network and target Q-value network The Q-value functions are each composed of a four-layer fully connected feedforward neural network with attention mechanisms between the fully connected layers and ReLu activation functions, with two hidden layers each containing 128 neurons. The SEA-DDQN model agent at each time step is represented by a feature vector of network traffic The state space of the agent is defined as Thus, the state vector of each piece of network traffic obtained by the SEA-DDQN model agent is As follows: ； wherein each dimension represents a certain feature of network traffic; Step S33, introducing an attention mechanism to optimize the attention degree distribution of the SEA-DDQN model agent to network traffic features, enhancing discriminative features and suppressing irrelevant features, and the specific process is as follows: Step S331, for each time step input network traffic feature state vector After processing through the full connection layer, a 128-dimensional feature vector is obtained Wherein represents the network traffic feature vector processed through the full connection layer, represents the dimension index; then sent to the attention layer to enhance the attention degree to important features, and the attention weight As follows: ； where, is a dimensionality reduction projection matrix; is a dimensionality increase reconstruction matrix; denotes the dimensionality reduction ratio, which is a hyperparameter; denotes a Sigmoid gating function, which compresses the input value to interval, and the generated attention weight is used to measure the importance of each feature; Step S332, generating attention weights with the original features are fused by modulation as follows: ； wherein, represents the fused feature representation; is a Hadamard product, i.e. an element-wise multiplication operation; this operation causes each feature value to be dynamically adjusted by the corresponding weight ; Step S34: The SEA-DDQN model agent, based on the input features of the deep neural network... The system generates a list of actions, which is presented as action vectors; the final Q-value is used to evaluate whether the attack behavior was successfully predicted. Step S341, first, respectively map the class labels in the NSL-KDD, UNSW-NB15, CICIDS-2017 and MQTT-IoT-IDS2020 data sets into a digital class set, as follows: ; where 0 means Normal; 1 means DoS; 2 means Probe; 3 means U2R; 4 means R2L; ; wherein 0 indicates normal traffic; 1 indicates abnormal traffic; ; where 0 means Bots; 1 means Brute Force; 2 means DDos; 3 means Dos; 4 means Normal; 5 means Port Scanning; 6 means Web Attacks; ; where 0 means Normal; 1 means Bruteforce; 2 means Scan_A; 3 means Scan_sU; 4 means Sparta; Step S342, then, define the action space of the SEA-DDQN model agent as: ； ； ； ； wherein each action corresponding to a predicted class decision in the list of actions, respectively; denotes a class index; denotes a defined symbol, the symbol to the left is defined by the one to the right. Step S35, designing a dynamic weight reward mechanism, balancing class bias by constructing a class-sensitive reward, and determining the class weight by using a reverse frequency weighting strategy to enhance the sensitivity of the SEA-DDQN model agent to different attack classes, balance the influence of class bias in the data set on the learning process, and the specific process is as follows: Step S351, first, define the reward function The specific form is as follows: ； wherein, represents a weight of an attack category; represents a correct prediction; represents a wrong prediction; Step S352, then, use the reverse frequency weighting strategy to assign weights by considering the frequency of each attack class, as follows: ； ； wherein represents a set of attack category weights of represents a weight of an attack; represents a weight of an attack; represents a weight of an attack; represents a weight of an attack; represents a weight of an attack; represents a set of attack category weights of represents a weight of an attack; represents a weight of an attack; represents a weight of an attack; represents a weight of an attack; represents a weight of an attack; The weight rule is updated as follows: ； wherein, is the total number of samples, is the number of attack types, denotes the total number of attack samples; For the prediction task, a positive reward is given when the SEA-DDQN model agent correctly predicts the sample, and a negative penalty is given otherwise, as follows: ； wherein, represents a reward setting rule for each flow of the data set; To simulate the delayed feedback scenario in real-world networks, an n-step reward accumulation mechanism is adopted, and the reward actually stored into the experience replay pool is the n-step cumulative reward , rather than the single-step immediate reward ; the agent stores the last n-step transition sequence in the interaction with the environment, and calculates the discounted cumulative reward after reaching n steps , as follows: ； Step S36, using a priority sampling strategy PER to optimize the sampling process of the experienced network traffic, since the importance of attack classes is different, introducing TD-Error as a key indicator to measure the importance of samples, and by defining a priority update formula, the sampling probability of the samples is dynamically adjusted, as follows: ； in, Indicates the first The probability that a given network traffic sample will be taken; It is a constant used to ensure that all empirical network traffic has a probability of being sampled, and to avoid some samples having a zero sampling probability due to the TD-Error being too small; This represents the current network traffic count in the experience replay pool. This is a priority parameter used to control the degree of influence of TD-Error on the sampling probability; For the first The TD-Error corresponding to each empirical network traffic is shown below: ； wherein, represents a target Q-value network; represents a state; represents a state; represents an action for a state; represents a target network; represents a Q-value estimate for a target network; represents a maximum of a Q-value estimate for a target network; represents a state; represents a state; represents an action for a state; represents a current network; represents a Q-value estimate for a current network; Step S4, based on the SEA-DDQN model, implementing the ISP process, including the following steps: Step S41, based on the SEA-DDQN model, the state feature vector space of each network traffic sample is mapped into a high-dimensional feature space through a hierarchical nonlinear transformation As the input of the action selection network , the features of each traffic sample are captured through hierarchical nonlinear transformation. Step S42, map the network traffic sample state feature vector in the d-dimensional feature vector space to a 128-dimensional hidden space in the input layer as follows: ； wherein, represents a pre-activation vector of the first layer neural network; represents a weight of the first layer neural network; represents a bias of the second layer neural network; Step S43, passing through the attention layer and using The activation function realizes a nonlinear transformation, so that the neural network learns a more complex traffic feature representation, as follows: ； ； wherein, is an element-wise multiplication, for combining the weighted results of the attention mechanism with the original features; denotes the result after ReLU activation function, representing the nonlinearly transformed feature representation learned by the network. Step S44, the second fully connected layer and the subsequent layers and the above layers are the same, as follows: ； ； ； ； wherein, represents a pre-activation vector of the second layer neural network; represents a weight of the second layer neural network; represents a bias of the second layer neural network; represents a result after the second layer is processed by a ReLU activation function. Step S45, in the last output layer, the output value of each corresponding Q value, the output value represents the expected reward accumulation of the current state the corresponding action, as follows: ； wherein, is the weight of the fourth layer neural network; is the result of the third layer after ReLU activation function, representing the feature representation learned by the network after nonlinear transformation; is the bias of the fourth layer neural network; is the Q value of the first action in the action space element vector; represents the Q value estimation of the network; is the expected cumulative reward of the corresponding action space element. For the current state Through the action selection network The resulting Q-value vector, the agent adopts The policy to select the best predicted action, as follows: ； wherein, represents the size of the predicted action space; represents the optimal predicted action under the current state ; represents the exploration rate for the current time step; represents the probability of selecting an action under the state ; The update rule for the is shown below: ； wherein, is an initial exploration rate, is a minimum exploration rate threshold, is a decay coefficient, is a current learning time step; Step S46, target Q-value network The calculation process of the action selection network The calculation process of the action selection network is the same, and the input is The target Q-value is calculated as follows: ； wherein, is the current state take an action the reward obtained; the target network Q-value; Step S47, based on PER, the sampling probability is allocated according to the importance of experience, a sampling bias is introduced, a mean square error loss function with importance sampling weight is constructed to compensate for the bias introduced by non-uniform sampling, as follows: ； wherein, represents ; represents the b-th flow in the experience replay pool; represents the state; represents the action; represents the predicted Q value of the state and action of the b-th flow by the action selection network; represents the target q value of the b-th flow; represents the probability that the b-th flow is sampled; represents the importance sampling coefficient; is the number of network flows drawn from the experience replay pool; At the initial stage of learning, as learning proceeds, The values gradually increase, eventually approaching 1, as follows: ； wherein, denotes the value of β at time step t; denotes the value of β at time step t-1; denotes the initial value of β; denotes the final target value of β; β is a parameter for controlling the degree of forgetting or retaining past experiences; Step S48, in the action selection network parameters parameters The update formula of the parameters is as follows: ； wherein, denote parameters of a neural network; denotes a learning rate; denote parameters of the gradient; Target Q-value network Parameters of the target Q-value network The update is implemented by a combination of Polyak-averaged soft and hard updates, i.e. every 100 time steps the parameters of the action selection network Parameters of the target Q-value network are directly copied to the parameters of the target Q-value network Parameters of the target Q-value network are updated by a weighted average; every 1000 time steps the parameters of the action selection network Parameters of the target Q-value network are directly copied to the parameters of the target Q-value network Parameters of the target Q-value network are updated as follows: ； wherein, denote parameters of the target q-value network; is average soft update coefficient.

2. The intrusion situation prediction method based on SEA-DDQN adaptive reinforcement learning according to claim 1, characterized in that, In step S2, the obtained network traffic data set is preprocessed to obtain a normalized data set, and the specific process is as follows: Step S21, defining the network traffic dataset as wherein is the number of traffic samples, then the feature space is as follows: ； wherein, represents a feature, represents an index of the number of features in the dataset; represents a feature subset; ; is a flow feature subset; is a content feature subset; is a time statistical feature subset; is a host behavior feature subset; represents a union symbol, and represents combining the four feature subsets into a feature space ; Step S22, for discrete features performing heat one-hot encoding processing; First, a discrete feature subset ; wherein, denotes a discrete feature; denotes a discrete feature index; the corresponding value range of each discrete feature is respectively: ; ; ; wherein, , and denote the value range of a discrete feature, ; for any discrete feature, define an indicator function as follows: ； wherein ; representing a sample at a discrete feature of the values; Then, the heat signature is transformed by a hot-sole coding transformation That is The feature space reconstruction is achieved as follows: ； wherein d is the total number of features after the hot one encoding process; represents a hot one encoding process function; represents a vector concatenation operation; is an index of all non-discrete features in the original features; is a discrete feature subset; Finally, the original training set and test set are respectively hot unique coded to generate their respective extended feature matrix and normalized as follows: ； wherein, denotes a normalized feature; and denote the maximum and minimum values of the feature , respectively; is an indicator function that takes the value 1 when and 0 otherwise; the attack classes in the dataset are classified and mapped into the set of numerical classes .

Images