Security protection method for intelligent computing network

Abstract

The invention provides an intelligent computing network-oriented security protection method, which comprises the following steps of: preprocessing a network traffic data set, and dividing the network traffic data set into a training set and a test set; performing feature selection, and exploring and searching an optimal feature subset in a feature space; modeling a network intrusion detection process as a Markov decision process, mapping a feature subset to a state and an action in a neural network, and designing a reward function; combining with an SAC reinforcement learning algorithm, establishing an experience playback pool, and constructing a reinforcement learning agent for network intrusion detection; defining an attacker agent and a defender agent which are opposite to each other, and performing parameter synchronization by using an asynchronous updating mode; respectively creating sub-agents of attackers and defenders to carry out iterative confrontation; and evaluating the classification capability of the defense agent by using the test set. Feature selection is carried out, the calculation amount is reduced, and the detection efficiency is improved; and an adversarial training mechanism is introduced to ensure that complex network attacks can be rapidly and accurately detected and responded.

Description

Technical Field

[0001] The present invention relates to the technical field of network security protection, and in particular to a security protection method for an intelligent computing network. Background Art

[0002] As an important part of the new generation of information infrastructure, intelligent computing networks have gradually evolved into a core force driving the in-depth development and transformation and upgrading of the digital economy. Intelligent computing networks have been widely infiltrated into many industries such as public services, finance, energy, transportation, commerce and trade, and industrial manufacturing, directly involving national security, social stability and personal privacy protection. With the continuous expansion of the scale of intelligent computing networks, the security threats they face are becoming more and more severe. Cyber ​​attacks cause incalculable losses and harm by penetrating intelligent computing networks, seriously threatening national political stability and social security. Based on the existing communication network, intelligent computing networks have the characteristics of deep integration of network and computing power, massive heterogeneous terminal access, and unbalanced terminal security protection capabilities. Traditional intrusion perception technology has problems such as incomplete extraction of network data features, reliance on data labels and data imbalance, resulting in the model's detection performance showing long detection time, high false alarm rate and missed alarm rate, and insufficient ability to identify new types of attacks, making it difficult to meet the needs of high detection rate, dynamic detection and automatic learning recognition. Therefore, there is an urgent need to develop a more efficient and intelligent intrusion detection technology, optimize data processing through feature selection, reduce the impact of redundant and irrelevant features, and introduce an improved reinforcement learning algorithm framework to collect and analyze historical attack data, so as to improve the ability to identify complex and hidden network attacks, achieve rapid response and adaptive learning, effectively improve the intelligent network threat detection capabilities, and ensure the effectiveness and timeliness of network security protection.

[0003] The invention patent with application number 202410931255.3 discloses a method, device, equipment and medium for intelligent computing network protection of computing power equipment. The intelligent computing network protection method for computing power equipment includes: when the operating system of the computing power equipment is in kernel state, extracting features from the acquired data packets to obtain target features; calling a preset network security model to detect attacks on the target features to obtain detection results; the preset network security model is used to determine whether the data packets corresponding to the target features have attack behaviors; compiling the preset program into eBPF bytecode and mounting it to the XDP Hook in kernel state; performing the target operation corresponding to the detection result on the data packet in the XDP Hook. The above invention discloses that by detecting data packets with attack behaviors before the data packets reach the application layer, the detection position is forward, and the network security model can be used to realize real-time and accurate analysis of data packets, and the detection process runs directly in the kernel space, which can quickly detect whether there is an attack behavior. However, the above method does not mention how to effectively screen out the features that are really critical to the detection task. There may be a situation where the amount of calculation is large when facing a large number of data packet features. The above method emphasizes early detection and real-time and accurate analysis, which may lack sufficient adaptability. Summary of the invention

[0004] In view of the technical problems of long detection time, high false alarm rate and missed alarm rate of existing network intrusion detection algorithms and insufficient ability to identify complex attack types, the present invention proposes a security protection method for intelligent computing networks, which decomposes network intrusion detection into two steps: feature selection and intrusion detection, optimizes the response speed and accuracy of network intrusion detection, significantly reduces the false alarm rate and missed alarm rate, enhances the security and defense capability of intelligent computing networks, and improves resource utilization efficiency, and enhances the intelligence and automation level of network security protection; the present invention adopts the HSWOA algorithm for feature selection, eliminates redundant and irrelevant features, reduces the amount of calculation, and improves detection efficiency; introduces an adversarial training mechanism to simulate real attack and defense scenarios, and ensures rapid and accurate detection and response in the face of complex network attacks.

[0005] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a security protection method for intelligent computing network, the steps of which are as follows:

[0006] Step 1: Preprocess the network traffic data set and divide the preprocessed data set into a training set and a test set;

[0007] Step 2: Use the mixed strategy whale optimization algorithm to select features for the preprocessed training set. By simulating the hunting behavior of whales, explore the potential optimal solution area in the feature space and search for the optimal feature subset;

[0008] Step 3: Model the network intrusion detection process as a Markov decision process, define the state, action, reward, and discount factor quadruple, map the preprocessed data set to the state and action in the neural network, and design the reward function;

[0009] Step 4: Combine the four-tuple defined in step 3 with the SAC reinforcement learning algorithm, build an experience replay pool, construct a reinforcement learning agent for network intrusion detection, define the reinforcement learning agent neural network and design a loss function to optimize the strategy and value estimation of the reinforcement learning agent;

[0010] Step 5: Based on the obtained reinforcement learning agent, define the adversarial attacking agent and defending agent, initialize the parameters of the agent's neural network, and synchronize them with the agent's policy network and value network parameters using an asynchronous update method;

[0011] Step 6: Create sub-agents for both the attacker and defender to perform iterative confrontation. In each round of training, the attacker and defender calculate the reward value based on the classification results of the defender's agent, update their own sub-agent strategies, and pass the parameters to the attacker and defender agents.

[0012] Step 7: Use step 2 to process the test set and evaluate the performance of the defender’s agent. The classification ability of the defender’s agent is evaluated by calculating the accuracy, false positive rate, and false negative rate.

[0013] Preferably, the preprocessing is to merge the CSV files in the network traffic data set into one CSV file; detect and delete missing values ​​in the merged CSV file; convert the character data in the CSV file into numerical data using one-hot encoding technology; and normalize the numerical data.

[0014] Preferably, the implementation method of the hybrid strategy whale optimization algorithm for feature selection is: initializing the training set to generate an initial population containing multiple potential feature combinations, the initial population is the initial position of the whale in the feature space, incorporating a Gaussian mutation mechanism in the prey encirclement stage, and introducing an adaptive inertia weight and a mirror selection strategy in the random prey search stage; the Gaussian mutation mechanism adjusts each feature dimension to a random number that obeys a Gaussian distribution, and the adaptive inertia weight is a dynamically adjusted parameter determined according to the current number of iterations and the total number of iterations.

[0015] Preferably, the Gaussian mutation mechanism is implemented by: according to the position X of the i-th whale in the j-dimension in the current iteration t ij (t), calculate the average μ of the i-th whale in all dimensions i and the standard deviation σ in all dimensions i; The position of the whale in the jth dimension after the Gaussian mutation operation is

[0016] The adaptive inertia weight is T is the total number of iterations, ω max and ω min are the maximum and minimum values ​​of the adaptive inertia weight respectively;

[0017] The whale's position update formula is:

[0018]

[0019] Among them, X * (t) is the position of the best whale in the current iteration t, A is the coefficient related to the convergence factor, D is the distance between the current whale and the best whale, X rand (t) is the whale individual randomly selected in the current iteration, X i (t+1) is the position of the i-th whale in the next iteration;

[0020] The implementation method of the mirror selection strategy is as follows: take a uniform random number r∈(0,1) in each feature dimension and determine whether it exceeds a threshold, perform an inverse transformation on the feature dimension that exceeds the threshold to obtain a mirror solution, merge the original solution and the mirror solution to form an extended population, and sort the individuals in the extended population according to the fitness function, and select individuals with higher fitness as the initial positions of the next generation of whales.

[0021] Preferably, the modeled Markov decision process is defined as a four-tuple M=<S,A,R,γ>, where S is the state space, A is the action space, R is the reward space, and γ is the reward discount factor; the state space S={s 1 ,s 2 ...s t} Includes various statistics and indicators of network traffic, where each state s t is a feature vector that describes the current situation of network traffic; the agent is in each state s t According to the action strategy function π * (a t |s t ) Output the probability distribution of the action and sample it to get the specific action, forming the action space A = {a 1 ,a 2 ...a t}, action a t Labels representing network traffic behavior; each time the agent takes an action a t Each instant reward r is generated, and the reward value of the instant reward r is recorded to form a reward space R = {r 1 ,r2 ...r t}; The reward function assigns higher reward values ​​to the minority categories of abnormal traffic.

[0022] Preferably, the reward function is:

[0023]

[0024] in, represents the predicted action, a t represents the real action, and Total represents the total amount of data in the training set.

[0025] Preferably, the SAC reinforcement learning algorithm obtains experience of state transfer through interaction with the environment, and uses experience replay to update the strategy and value function, and adaptively learns the best strategy for network attack detection;

[0026] The method for building the experience replay pool is as follows: in network intrusion detection, the state of the current iteration t is s t , action a t Based on the probability distribution of the Actor network output π(a|s t )Select, action a t Input to state s t Get the reward r of the current iteration t t , the next state s t+1 and a Boolean value d indicating whether the interaction between the agent and the environment has reached a terminal state t , and each experience is associated with a weight parameter w calculated by time difference error t , each experience is in the form of a sextet <s t ,a t ,r t ,s t+1 ,d t ,w t > in the form of experience replay pool;

[0027] The method for constructing a reinforcement learning agent for network intrusion detection is as follows: a reinforcement learning agent based on a SAC algorithm is designed, and the reinforcement learning agent includes an Actor network, two V Critic networks and two Q Critic networks. Among the two V Critic networks, the V Critic network is responsible for estimating the value of each state in real time according to the current strategy, the target V Critic network serves as the target network and provides a stable value estimation as a learning goal. The target V Critic network is soft-updated through the V Critic network, and the two Q Critic networks are used to independently evaluate the value of action-state pairs.

[0028] Preferably, the weight parameter w t The calculation method is:

[0029]

[0030] Among them, δ t represents the time difference error of the current time step, Q(s t ,a t ) means in state s t Take action a t The expected return, γ represents the discount factor, and the state value function V(s t+1 ) represents the next state s t+1 Expected return;

[0031] The Actor network is responsible for the action output of the reinforcement learning agent, and the input is the state s of the current time step t , the probability distribution of the output action π(a|s); the loss function of the Actor network is

[0032]

[0033] Where e∈M represents the experience in the experience replay pool M, |M| represents the number of experiences in the experience replay pool; Q j (s t ,a t ) indicates that the jth Q Critic network is in state s t and action a t The output value under , takes the minimum value of the output of the two Q Critic networks; α represents the entropy regularization coefficient;

[0034] The V Critic network is used to estimate the value of the state, and the input is the state s of the current time step t , the output is the state value function V(s t ), indicating that in state s t The expected return that the agent can obtain by following the current strategy, the loss function of the V Critic network is

[0035]

[0036] in, Indicates the current strategy π in state s t The action a sampled from the action probability distribution under t 'Expectations;

[0037] The Q Critic network is used to evaluate the value of different actions, and the input is the state s of the current time step t , output action state value function Q(s t ,at ), indicating that in state s t The next agent chooses action a t The expected return that can be obtained by following the current strategy, the loss function of the Q Critic network is

[0038]

[0039] Among them, r t For state s t Execute action a t The immediate return obtained after , γ is the discount factor;

[0040] The calculation formula of the action strategy function is:

[0041]

[0042] H(π(·|s t ))=-logπ(·|s t );

[0043] Among them, π represents the strategy, E π [·] represents the mathematical expectation of the cumulative reward plus the entropy regularization term under strategy π, r(s t ,a t ) indicates state s t Next action a t The reward function is α, which represents the entropy regularization coefficient, H(π(·|s t )) is the policy π in state s t The entropy at , T represents the maximum number of iterations, π(·|s t ) means in state s t The strategy for selecting actions.

[0044] Preferably, an attack agent neural network and a defense agent neural network are constructed according to task requirements and data characteristics. Both the attack agent neural network and the defense agent neural network are composed of a multi-layer perceptron. The MLP weights and biases are dynamically adjusted through the reward signal of the reinforcement learning agent and the environment interaction training to maximize the cumulative reward. Both the attack agent neural network and the defense agent neural network take the characteristics of the current sample as input. The output of the attack agent neural network is to select an attack type from multiple functional attack types and extract a random sample corresponding to the output attack type. The output of the defense agent neural network is a prediction of the intrusion category.

[0045] The implementation method of the asynchronous update method is as follows: create multiple worker threads, each worker thread contains an independent attacking agent, a defending agent and a shared environment, and the environment is constructed by a feature subset obtained by preprocessing and feature selection; all worker threads execute training tasks in parallel, each worker thread interacts with the environment independently, each worker thread loads the same environment, and each maintains the parameters of its own thread agent; the data generated during the training process are used to calculate the losses of the policy network and the value network respectively, and the policy network and value network parameters of each agent are updated through the back propagation algorithm; after each update, the updated parameters are passed to the main network in an asynchronous manner after the training reaches a convergence state.

[0046] Preferably, the implementation method of step 6 is to create sub-agents representing the attacker and the defender in each worker thread, and perform dynamic and real-time iterative confrontation in the same simulated network environment. In each round of training, both the attacker and the defender will calculate the reward value according to the reward function and the current classification result obtained by the multi-layer perceptron of the defender;

[0047] The attacker's sub-agent attempts to extract data of various attack types and observe whether the attack type can successfully bypass the defender's detection: if successful, the attacker's agent receives a positive reward, otherwise it receives a negative reward; the defender's agent calculates the reward value based on whether the attack traffic is correctly classified: if the classification is accurate, the defender's agent receives a positive reward, otherwise it will be punished.

[0048] Compared with the prior art, the invention has the following beneficial effects: firstly, the original network traffic data set is preprocessed by one-hot encoding and other technologies, and the processed data is divided into a training set and a test set; secondly, the HSWOA (Hybrid Strategy Whale Optimization Algorithm) algorithm is used for feature selection to simulate the hunting behavior of whales, search for the optimal feature subset for the intrusion detection task, eliminate redundant and irrelevant features, effectively reduce the amount of calculation, improve the detection efficiency, and reduce the false positives and false negatives caused by the interference of irrelevant features; then, the SAC (Soft Actor-Critic) algorithm is improved, and the A2SAC (Adversarial Asynchronous Soft The Actor-Critic algorithm constructs a neural network model, initializes parameters, and maps preprocessed data into states, actions, and reward values. It combines the experience replay pool technology to stably and efficiently optimize network parameters. It uses asynchronous updates to efficiently utilize computing resources during training, accelerates the convergence of the model, and significantly improves the training efficiency of the network intrusion detection system. Finally, an adversarial training mechanism is introduced to define two intelligent agents, the attacker and the defender, to conduct iterative confrontation and simulate real attack and defense scenarios. The attacker and the defender continuously optimize the classification strategy through reinforcement learning to ensure that the generated model can quickly and accurately detect and respond to complex and changeable network attacks, maintain a high detection rate and a low false alarm rate, and comprehensively improve the security and reliability of the network intrusion detection system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 The overall architecture diagram of the HSWOA-A2SAC network intrusion detection model of the present invention.

[0051] Figure 2 A structural diagram of the A2SAC agent based on a multi-layer perceptron of the present invention.

[0052] Figure 3 The architecture diagram of the A2SAC network intrusion detection model of the present invention.

[0053] Figure 4 The neural network structure diagram of the attacking intelligent agent and the defending intelligent agent of the present invention, wherein (a) is the attacking intelligent agent neural network, and (b) is the defending intelligent agent neural network.

[0054] Figure 5 This is the confusion matrix of the HSWOA-A2SAC model of the present invention based on the CSE-CIC-IDS2018 dataset, where (a) is a binary confusion matrix and (b) is a multi-class confusion matrix. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] With the continuous development of attack technology, network attack methods are becoming increasingly diversified and covert. However, existing network intrusion detection systems have problems such as insufficient adaptability in dynamically changing network environments, high false alarm and missed alarm rates, and neglect of resource consumption during the detection process, resulting in reduced detection accuracy and efficiency and high deployment costs. In response to the above problems, the present invention proposes a security protection method for intelligent computing networks---(Hybrid Strategy Whale Optimization Algorithm-Adversarial Asynchronous Soft Actor-Critic, HSWOA-A2SAC). The overall architecture of HSWOA-A2SAC is as follows: Figure 1 HSWOM-A2SAC uses feature selection to eliminate redundant features, effectively controls resource consumption, and dynamically adjusts the detection strategy to adapt to changes in the network environment by simulating real attack and defense confrontation scenarios. It optimizes the detection algorithm to improve accuracy and efficiency. The specific workflow steps of the present invention are as follows:

[0057] Step 1: Preprocess the network traffic data set through one-hot encoding and other technologies, and divide the preprocessed data set into training set and test set for subsequent model training and performance evaluation. The specific contents are as follows:

[0058] The present invention selects 10 CSV files in the published network traffic dataset CSE-CIC-IDS2018 from the official website of the Canadian Institute for Cyber ​​Security (CIC). The dataset contains seven attack scenarios: brute force, heartbleed, botnet, DoS, DDoS, network attack and internal penetration network. The CSV files include the captured network traffic and system logs of each machine and 80 features extracted from the network traffic captured using the CICFlowMeter-V3 tool, mainly including network traffic information such as Dst Port, Protocol, Timestamp and Flow Duration.

[0059] The present invention first unifies CSV files into one CSV file through a merge operation to facilitate subsequent processing; secondly, missing value detection and deletion are performed to ensure that there are no missing records in the data set and data integrity is guaranteed; then, one-hot encoding technology is used to convert character data into numerical data to meet the requirements of the machine learning model for the input data format; finally, the numerical data is normalized and the data value is scaled to the interval of [0,1] to eliminate the influence of dimensional differences and uneven distribution.

[0060] In Python software, use the train_test_split method in the scikit-learn library and combine it with the StratifiedShuffleSplit cross-validation iterator to randomly sample the preprocessed data set, divide the preprocessed data set into training set, validation set and test set in a ratio of 70%, 15%, and 15%, and save the corresponding CSV files to ensure use in subsequent model training.

[0061] Step 2: Use the hybrid strategy whale optimization algorithm (HSWOA) to select features for the preprocessed data set. By simulating the hunting behavior of whales, explore the potential optimal solution area in the feature space, search for the optimal feature subset, and eliminate redundant and irrelevant features to improve the efficiency and accuracy of the model. The specific contents are as follows:

[0062] Whale Optimization Algorithm (WOA) is an optimization algorithm that simulates the predation behavior of whales. WOA consists of three actions: surrounding prey, spiral attack on prey, and random search for prey. HSWOA incorporates Gaussian mutation mechanism in the algorithm's surrounding prey phase, and introduces adaptive inertia weight and mirror selection strategy in the random search for prey phase to improve the algorithm's performance in feature selection tasks.

[0063] In the initialization phase of HSWOA, the preprocessed data set is first initialized to generate an initial population containing multiple potential feature combinations. The generated initial population (feature combination) is regarded as the initial position of the whale in the feature space. In order to increase the diversity of the population, that is, to cover a wider feature space during the search process, the present invention introduces a Gaussian mutation mechanism to adjust each feature dimension to a random number that obeys a Gaussian distribution.

[0064] For the i-th whale (i.e., the i-th solution) at the position X in the j-th dimension ij , first calculate the mean μ of the i-th whale in all dimensions i and the standard deviation σ in all dimensions i , the calculation formula is shown in formula (1) and formula (2).

[0065]

[0066]

[0067] Among them, d represents the number of dimensions, X ij (t) represents the position of the i-th whale in the j-dimension at time t. The position X of the whale in the j-dimension after the Gaussian mutation mechanism operation proposed by the present invention is i ' j (t+1) is as shown in formula (3).

[0068]

[0069] Position X i ' j The calculation result of (t+1) is a probability, that is, a random number that obeys the Gaussian distribution. The actual value of the final position is converted to 0 or 1 according to the probability.

[0070] In the iterative search phase of HSWOA, the whale adjusts its position according to the distance between the current position and the optimal position and the convergence factor. The present invention balances the capabilities of global search and local search by introducing adaptive inertia weights.

[0071] The adaptive inertia weight is a dynamically adjusted parameter, which is determined by the current number of iterations and the total number of iterations. In the early stage of iteration, a larger inertia weight enables the whale to explore the feature space extensively and find the potential global optimal solution. As the iteration progresses, the inertia weight gradually decreases, and the whale begins to conduct a detailed search in the discovered high-quality areas. The calculation formula of the adaptive inertia weight is shown in formula (4).

[0072]

[0073] In the iterative search phase, the whale position update formula combines the adaptive inertia weight, the whale position Xi The update formula of is shown in formula (5).

[0074]

[0075] Among them, X * (t) is the position of the best whale in the current iteration, A is the coefficient related to the convergence factor, D is the distance between the current whale and the best whale, X rand (t) is the randomly selected whale individual in the current iteration. i (t+1) is the position of the i-th whale in the next iteration.

[0076] In each iteration, a mirror selection strategy is used to update the position of the whale to further improve the algorithm's search efficiency and convergence speed. The present invention takes a uniform random number r∈(0,1) in each dimension and determines whether it exceeds a threshold (set to 0.5 in the present invention). The inverse transformation is performed on the feature dimension that exceeds the threshold to obtain a mirror solution, and the dimension that exceeds the threshold is transformed, and vice versa. The original solution and the mirror solution are merged to form an extended population, avoiding falling into a local optimal solution during the search process, improving the global search capability, and sorting the individuals in the extended population according to the fitness function, selecting individuals with higher fitness as the initial position of the next generation of whales, and continuously optimizing the population quality to improve the algorithm's convergence speed and solution accuracy.

[0077] In summary, the Gaussian mutation mechanism is used to increase the diversity of the population in the initialization phase, the adaptive inertia weight balances the ability of global search and local development in the iterative search phase, and the mirror selection strategy optimizes the whale's position update process in each iteration. HSWOA combines the three strategies to improve WOA, remove redundant and irrelevant features in network traffic data, effectively improve detection efficiency, and reduce false positives and false negatives caused by interference from irrelevant features, providing support for building an efficient and accurate network intrusion detection model.

[0078] The final result of HSWOA is a feature subset. For the CSE-CIC-IDS2018 dataset with 80 features, the feature subset is a one-element array. Assume that the feature subset is "[1,2,6,7,11,13]", which means that the 1st, 2nd, 6th, 7th, 11th and 13th features are finally selected. The subsequent steps directly change the dataset, ignore other feature columns, and only keep the columns corresponding to the feature subset. Then, the changed dataset is tested. The feature space refers to the entire space of all optional cases of n features (dimensions). For example, if there are 10 features and each feature takes only two values ​​(0 or 1), the size of the feature space is 2. 10 = 1024 possible points.

[0079] Step 3: Model the network intrusion detection process as a Markov decision process, define the state, action, reward, and discount factor quadruple, map the preprocessed data set, which is the data set after the feature subset processing of the original data set, to the state and action in the neural network, and design the reward function. The specific contents are:

[0080] The present invention models the detection process of network attack behavior as a Markov decision process (MDP). The modeled MDP can be defined as a four-tuple M=<S,A,R,γ>, where S is the state space, A is the action space, R is the reward space, and γ is the reward discount factor. Traditional MDP usually includes an explicit state transition probability P(s′|s,a), that is, the probability that the environment transfers to the next state s′ given the state s and action a. In the present invention, the state transition probability is not explicitly modeled. The SAC algorithm is a model-free method based on reinforcement learning. It obtains state transition experience through interaction with the environment and uses experience playback to update the strategy and value function. The SAC algorithm adaptively learns the optimal strategy for network attack detection without explicitly modeling the dynamic characteristics of the environment.

[0081] State space S = {s 1 ,s 2 ...s t}, including various statistics and indicators of network traffic, such as packet size, protocol type, traffic rate, etc., where each state s t is a feature vector that describes the current situation of network traffic.

[0082] The agent is in each state s t According to the action strategy function π * (a t |s t ) Output the probability distribution of the action and sample it to get the specific action, that is, the label of the network traffic behavior, forming the action space A = {a 1 ,a 2 ...a t}, action a t The label representing the network traffic behavior. The calculation formula of the action strategy function is shown in Equation (6) and Equation (7).

[0083]

[0084] H(π(·|s t ))=-logπ(·|s t ) (7)

[0085] Among them, π represents the strategy, E π[···] represents the mathematical expectation of the cumulative reward plus the entropy regularization term under strategy π, r(s t ,a t ) indicates state s t Next action a t The reward function is α, which represents the entropy regularization coefficient and determines the relative importance of the entropy term to the reward. t )) is the policy π in state s t The entropy at , T represents the maximum number of iterations. π(·|s t ) means in state s t The strategy for selecting actions.

[0086] Each time the agent takes an action a t will generate an immediate return r(s t ,a t ), the reward value of the record return can form a reward space R = {r 1 ,r 2 ...r t Due to the natural imbalance of network traffic, the proportion of normal traffic in the training set is much larger than that of abnormal traffic, and the proportion of traffic of different categories of abnormal traffic is also different. The present invention modifies the 0-1 reward function. In order to enable the classifier to identify the abnormal traffic of a minority category, a higher reward value is given to the abnormal traffic of a minority category, as shown in formula (8).

[0087]

[0088] in, represents the predicted action (label), a t Represents the actual action (label), and Total represents the total amount of data in the training set.

[0089] Figure 1 Mini-batch random sampling in the original dataset randomly samples a set of [s t ,a t ,s t+1 ] triplet samples to avoid sample duplication and ensure that the dataset is fully and balancedly explored during iterative training.

[0090] Step 4: Combine the four-tuple defined in step 3 with the SAC reinforcement learning algorithm, build an experience replay pool, and construct a reinforcement learning agent for network intrusion detection. Optimize the strategy and value estimation of the reinforcement learning agent by defining the reinforcement learning agent neural network and designing the loss function. The specific contents are as follows:

[0091] In network intrusion detection, the state at time t is s t , action a tBased on the probability distribution of the Actor network output π(a|s t )Select, action a t Input to state s t To obtain the current time step reward r t , the next state s t+1 and a Boolean value d indicating whether the interaction between the agent and the environment has reached a terminal state t , and each experience is associated with a weight parameter w calculated by temporal-difference error (TD Error) t , the calculation formula is shown in formula (9).

[0092]

[0093] Among them, δ t represents the TD Error of the current time step, Q(s t ,a t ) means in state s t Take action a t The expected return, γ represents the discount factor, and the state value function V(s t+1 ) represents the next state s t+1 The expected return of each experience is a sextuple <s t ,a t ,r t ,s t+1 ,d t ,w t > in the form of experience replay pool, adding an attribute / label to each experience, and giving different experiences different priority weight parameters w t To distinguish priorities, break the temporal correlation between samples, and improve the efficiency of sample utilization. During the agent training process, samples in the experience replay pool are randomly extracted according to priority to update the parameters of the neural network, which greatly improves the training efficiency, stability and convergence speed in reinforcement learning, ensuring that the agent can learn higher quality strategies in fewer training steps.

[0094] In the process of constructing the intelligent agent of the present invention, a reinforcement learning (A2SAC) intelligent agent based on the SAC algorithm is designed, aiming to optimize the decision-making strategy by interactive learning with network traffic data. Figure 2As shown in the figure, the A2SAC agent includes an Actor network, two V Critic networks and two Q Critic networks. The V Critic network in the two V Critic networks is responsible for estimating the value of each state in real time according to the current strategy, providing an important reference for the action selection of the agent. The target VCritic network, as the target network, is a delayed copy of the original network, providing a stable value estimate as a learning target, which helps to reduce fluctuations during training. The actual output content of the V Critic network and the target V Critic network are consistent. The target V Critic network is soft-updated through the V Critic network, and the strategy may be different from that of the V Critic network. The two Q Critic networks are used to independently evaluate the value of the action-state pair, reduce valuation bias, and improve the robustness of the algorithm.

[0095] The Actor network is responsible for the output of the agent's actions, and its input is the state s of the current time step. t , output the probability distribution of the action π(a|s), each action has a corresponding probability value. The agent usually chooses actions with higher probabilities, but because the SAC algorithm encourages exploration, the agent will occasionally choose actions with low probabilities to maintain the randomness and diversity of the strategy. The exploration balance is usually achieved by adding an entropy term to the loss function. The Actor network loss function in the present invention is shown in formula (10).

[0096]

[0097] Where |M| represents the number of experiences used in a training step, Q j (s t ,a t ) indicates that the jth Q Critic network is in state s t and action a t The output value under the condition is the minimum value of the output of the two Q Critic networks to reduce overestimation. α represents the entropy regularization coefficient, which is used to encourage the Actor network to output more uncertain actions. The value of the entropy regularization coefficient has no fixed range. A larger entropy regularization coefficient α will enhance exploration, while a smaller entropy regularization coefficient α will focus more on using currently known information. The value of the present invention is 0.2. e∈M represents each experience in a training step.

[0098] The V Critic network is used to estimate the value of the state, and its input is the state s of the current time step t , output state value function V(s t ), indicating that in state s tThe expected reward that the agent can obtain by following the current strategy. The V Critic network loss function in the present invention is shown in formula (11).

[0099]

[0100] in, Indicates the current strategy π in state s t The action a sampled from the action probability distribution under t The present invention considers adding a regularization term related to the policy entropy into the loss function of the V Critic network to enhance the generalization ability of the model.

[0101] The Q Critic network is used to evaluate the value of different actions, and its input is the state s at the current time step t , output action state value function Q(s t ,a t ), indicating that in state s t The next agent chooses action a t The expected return that can be obtained by following the current strategy is the Q Critic network loss function in the present invention as shown in formula (12).

[0102]

[0103] Among them, r t For state s t Execute action a t The loss function of the Q Critic network optimizes the network parameters by minimizing the difference between the predicted value and the actual return.

[0104] The loss value calculated by the loss function is used to monitor the training process. If the loss value continues to decrease, it means that the neural network is learning effectively; if the loss value fluctuates or increases, it indicates that the learning rate, network structure or other hyperparameters need to be adjusted.

[0105] Step 5: Based on the reinforcement learning agent obtained in step 4, define two adversarial agents, the attacker and the defender. The attacker is responsible for generating challenging network traffic data, while the defender is responsible for receiving and classifying data, initializing the weights and biases of the agent neural network, and synchronizing the policy network and value network parameters of the two adversarial agents, the attacker and the defender, with the main agent, i.e., the attacker and the defender, in an asynchronous update manner. The specific contents are as follows:

[0106] The architecture of the attacking agent and the defending agent is as follows Figure 4As shown in the figure, in both cases, the attack agent neural network and the defense agent neural network are constructed according to the task requirements and data characteristics, and the realization of the agent will be affected by the structure and performance of the neural network. The architecture of the attack agent neural network and the defense agent neural network are composed of a shallow neural network - multi-layer perceptron (MLP), which has 2 layers with 100 neurons per layer and 3 layers with 150 neurons per layer respectively. This architecture ensures a faster response speed, and dynamically adjusts the MLP weights and biases through the reward signal of the reinforcement learning agent's interaction training with the environment, that is, the reward value calculated by the reward function, to maximize the cumulative reward. The weights and biases of the agent neural network are randomly initialized and adjusted by the back-propagation algorithm during the training process. From Figure 4 As can be seen from the figure, both the attack agent and the defense agent take the features of the current sample as input, but produce different outputs depending on the network. For the attack agent, the output is to select one of the 15 attack types and extract a random sample corresponding to this output attack type; for the defense agent, the output is the prediction of the intrusion category. Therefore, the MLP of the attacker can be regarded as a generator of attack categories, and the MLP of the defender can be regarded as implementing a classifier.

[0107] The present invention uses an asynchronous update method to achieve efficient training of intelligent agents. Specifically, multiple worker threads are created, each of which contains an independent attacking intelligent agent, a defending intelligent agent, and a shared copy of the environment, and the environment is constructed by a feature subset obtained by preprocessing and feature selection. All worker threads execute training tasks in parallel, that is, each thread interacts with the environment independently, each worker thread loads the same copy of the environment, and each maintains the parameters of its own thread intelligent agent. The data collected during training are used to calculate the losses of the policy network and the value network respectively, and the policy network and value network parameters of each intelligent agent are updated through the back propagation algorithm. After each update, the intelligent agent parameters in the worker thread are not immediately synchronized to the main intelligent agent of the attacker / defender, but the updated parameters are passed to the main network in an asynchronous manner after the training reaches a convergence state, that is, the performance tends to be stable. The main intelligent agent of the attacker / defender and the intelligent agent of the worker thread are both constructed by the structure mentioned in the previous steps, with the same structure, do not participate in training, and only receive the parameters passed by the worker thread.

[0108] In the asynchronous update mode, each worker thread can independently explore different parts of the environment, thereby avoiding falling into a local optimal solution. The present invention realizes efficient synchronization of the attacking agent and the defending agent parameters in the above manner, ensures the stability and effectiveness of the entire training process, and fully utilizes the computing resources of the multi-core CPU to accelerate the training process.

[0109] A2SAC network intrusion detection model architecture Figure 3 As shown in the figure, the architecture integrates two adversarial agents, the attack agent and the defense agent, to perform adversarial training, thereby improving the detection accuracy and adaptability of the model in complex network environments. It adopts an asynchronous update strategy to maximize the use of computing resources and significantly shorten the training cycle.

[0110] Step 6: Start the asynchronous adversarial training process, create sub-agents for both the attacker and the defender for iterative confrontation. In each round of training, the attacker and the defender calculate the reward value based on the classification results of the MLP of the defender's agent, update their own sub-agent strategies, and pass the parameters to the attacker and defender agents. The specific content is as follows:

[0111] The present invention creates sub-agents representing the attacker and defender in each worker thread, and enables them to perform dynamic, real-time iterative confrontation in the same simulated network environment to simulate the attack and defense scenarios in a real network environment. In each round of training, both the attacker and defender will calculate the reward value based on the reward function shown in formula (8) and the current classification result obtained by the defender's MLP.

[0112] The attacker's sub-agent attempts to extract data of various attack types and observe whether the attack type can successfully bypass the defender's detection. If successful, the attacker's agent will receive a positive reward, otherwise it will receive a negative reward. Similarly, the defender's agent calculates the reward value based on whether the attack traffic is correctly classified. If the classification is accurate, the defender's agent will receive a positive reward, otherwise it will be punished. In addition, since the present invention assigns a higher reward value to attack categories that account for a small proportion of the total in the reward function, the agents of the attacker and defender focus on these key categories during the iterative confrontation process, so that the agents of both sides can achieve dynamic balance and common progress in a complex network environment. By simulating real attack and defense scenarios, the agents of both sides use the ASA2C reinforcement learning algorithm to iteratively implement reinforcement learning to continuously optimize the classification strategy, ensuring that the generated model can quickly and accurately detect and respond to complex and changeable network attacks, maintain a high detection rate and a low false alarm rate, and comprehensively improve the security and reliability of the network intrusion detection system.

[0113] Step 7: Use a test set independent of the training set to evaluate the performance of the defender’s agent. By calculating key indicators such as accuracy, false positive rate, and false negative rate, the classification ability of the defender’s agent is comprehensively evaluated. Classification ability refers to the ability of the agent to correctly classify the input data into the predetermined category. Use a test set independent of the training set to evaluate the classification ability of the defender’s agent to ensure that it performs well in practical applications. The specific content is as follows:

[0114] In network intrusion detection, true positive examples (TP) are observed when attack events are correctly predicted, true negative examples (TN) are observed when normal events are correctly predicted, false positive examples (FP) are observed when the model incorrectly predicts positive events, and false negative examples (FN) are observed when the model incorrectly predicts negative events. There are many indicators for evaluating network intrusion detection algorithms. The present invention uses evaluation indicators based on confusion matrix, including detection rate (DR), false alarm rate (FAR), accuracy (Acc), precision (Pre) and F1 Score. The confusion matrix is ​​shown in Table 1.

[0115] Table 1 Confusion matrix

[0116]

[0117] Detection Rate (DR) is also known as True Positives Rate (TPR), which indicates the ability to correctly identify all true positive samples, that is, the proportion of positive samples correctly detected by the model. In network intrusion detection, it indicates the proportion of intrusion events successfully detected by the model. A high detection rate means that the model can more accurately detect potential intrusion behaviors.

[0118]

[0119] False Alarm Rate (FAR) or False Positives Rate (FPR) refers to the proportion of negative samples that the model mistakenly identifies as positive samples. In network intrusion detection, it refers to the proportion of normal behaviors that the model mistakenly identifies as intrusion behaviors. A low false alarm rate means that the model can effectively reduce false alarms.

[0120]

[0121] Accuracy (Acc) refers to the proportion of samples correctly predicted by the model, indicating the overall prediction accuracy of the model.

[0122]

[0123] Precision (Pre) indicates the proportion of positive samples correctly identified by the model. In network intrusion detection, it indicates the correct rate of the model among all samples identified as intrusions. A high precision means that the model is more reliable in alarming and has fewer false alarms.

[0124]

[0125] F1-Score is the harmonic mean of precision and recall, which combines the accuracy and completeness of the model and helps evaluate the balance between recall and precision of the model.

[0126]

[0127] Among them, Pre is the precision rate and DR is the detection rate.

[0128] The experimental results of the present invention in the test set are as follows Figure 5 As shown, according to Figure 5 The binary classification results of (a) and equations (13)-(17) show that the detection rate of the present invention is about 97.99%, the false alarm rate is about 2.25%, the accuracy rate is about 97.95%, the precision rate is about 99.53%, the F1-Score is about 98.76%, and the prediction time in the test set with a sample size of 2434927 is 35.51s, that is, the prediction of a single sample is 14.58 microseconds. Figure 5 (b) is the classification of specific attack types, and the classification results of specific attack types can be seen.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A security protection method for intelligent computing networks, characterized in that: The steps are as follows: Step 1: Preprocess the network traffic data set and divide the preprocessed data set into a training set and a test set; Step 2: Use the mixed strategy whale optimization algorithm to select features for the preprocessed training set. By simulating the hunting behavior of whales, explore the potential optimal solution area in the feature space and search for the optimal feature subset; Step 3: Model the network intrusion detection process as a Markov decision process, define the state, action, reward, and discount factor quadruple, map the preprocessed data set to the state and action in the neural network, and design the reward function; Step 4: Combine the four-tuple defined in step 3 with the SAC reinforcement learning algorithm, build an experience replay pool, construct a reinforcement learning agent for network intrusion detection, define the reinforcement learning agent neural network and design a loss function to optimize the strategy and value estimation of the reinforcement learning agent; Step 5: Based on the obtained reinforcement learning agent, define the adversarial attacking agent and defending agent, initialize the parameters of the agent's neural network, and synchronize them with the agent's policy network and value network parameters using an asynchronous update method; Step 6: Create sub-agents for both the attacker and defender to perform iterative confrontation. In each round of training, the attacker and defender calculate the reward value based on the classification results of the defender's agent, update their own sub-agent strategies, and pass the parameters to the attacker and defender agents. Step 7: Use step 2 to process the test set and evaluate the performance of the defender’s agent. The classification ability of the defender’s agent is evaluated by calculating the accuracy, false positive rate, and false negative rate.

2. The security protection method for intelligent computing networks according to claim 1 is characterized in that: The preprocessing includes merging the CSV files in the network traffic data set into one CSV file; detecting and deleting missing values ​​in the merged CSV file; and converting the character data in the CSV file into numeric data using one-hot encoding technology. Normalize numerical data.

3. The security protection method for intelligent computing network according to claim 1 or 2, characterized in that: The implementation method of the hybrid strategy whale optimization algorithm for feature selection is as follows: initializing the training set to generate an initial population containing multiple potential feature combinations, the initial population is the initial position of the whale in the feature space, incorporating a Gaussian mutation mechanism into the prey encirclement stage, and introducing an adaptive inertia weight and a mirror selection strategy in the random prey search stage; the Gaussian mutation mechanism adjusts each feature dimension to a random number that obeys a Gaussian distribution, and the adaptive inertia weight is a dynamically adjusted parameter determined according to the current number of iterations and the total number of iterations.

4. The security protection method for intelligent computing networks according to claim 3 is characterized in that: The Gaussian mutation mechanism is implemented as follows: according to the position X of the i-th whale in the j-th dimension in the current iteration t ij (t), calculate the average μ of the i-th whale in all dimensions i and the standard deviation σ in all dimensions i ; The position of the whale in the jth dimension after the Gaussian mutation operation is The adaptive inertia weight is T is the total number of iterations, ω max and ω min are the maximum and minimum values ​​of the adaptive inertia weight respectively; The whale's position update formula is: Among them, X * (t) is the position of the best whale in the current iteration t, A is the coefficient related to the convergence factor, D is the distance between the current whale and the best whale, X rand (t) is the whale individual randomly selected in the current iteration, X i (t+1) is the position of the i-th whale in the next iteration; The implementation method of the mirror selection strategy is as follows: take a uniform random number r∈(0,1) in each feature dimension and determine whether it exceeds a threshold, perform an inverse transformation on the feature dimension that exceeds the threshold to obtain a mirror solution, merge the original solution and the mirror solution to form an extended population, and sort the individuals in the extended population according to the fitness function, and select individuals with higher fitness as the initial positions of the next generation of whales.

5. The security protection method for intelligent computing network according to claim 3 is characterized in that: The modeled Markov decision process is defined as a four-tuple M = <S, A, R, γ>, where S is the state space, A is the action space, R is the reward space, and γ is the reward discount factor; the state space S = {s1, s2...s t } Includes various statistics and indicators of network traffic, where each state s t is a feature vector that describes the current situation of network traffic; the agent is in each state s t According to the action strategy function π * (a t |s t ) Output the probability distribution of the action and sample it to get the specific action, forming the action space A = {a1, a2...a t }, action a t Labels representing network traffic behavior; each time the agent takes an action a t Each of them will generate an immediate reward r, and the reward value of the immediate reward r is recorded to form a reward space R = {r1, r2...r t }; The reward function assigns higher reward values ​​to abnormal traffic of minority categories.

6. The security protection method for intelligent computing network according to claim 5 is characterized in that: The reward function is: in, represents the predicted action, a t represents the real action, and Total represents the total amount of data in the training set.

7. The security protection method for intelligent computing network according to any one of claims 4 to 6, characterized in that: The SAC reinforcement learning algorithm obtains experience of state transfer through interaction with the environment, and uses experience replay to update the strategy and value function, adaptively learning the best strategy for network attack detection; The method for building the experience replay pool is as follows: in network intrusion detection, the state of the current iteration t is s t , action a t Based on the probability distribution of the Actor network output π(a|s t )Select, action a t Input to state s t Get the reward r of the current iteration t t , the next state s t+1 and a Boolean value d indicating whether the interaction between the agent and the environment has reached a terminal state t , and each experience is associated with a weight parameter w calculated by time difference error t , each experience is in the form of a sextet <s t ,a t ,r t ,s t+1 ,d t ,w t > in the form of experience replay pool; The method for constructing a reinforcement learning agent for network intrusion detection is as follows: a reinforcement learning agent based on a SAC algorithm is designed, the reinforcement learning agent includes an Actor network, two V Critic networks and two Q Critic networks, the V Critic network of the two V Critic networks is responsible for estimating the value of each state in real time according to the current strategy, the target V Critic network serves as the target network and provides a stable value estimation as a learning goal, the target V Critic network is soft-updated through the V Critic network, and the two Q Critic networks are used to independently evaluate the value of action-state pairs.

8. The security protection method for intelligent computing networks according to claim 7 is characterized in that: The weight parameter w t The calculation method is: Among them, δ t represents the time difference error of the current time step, Q(s t ,a t ) means in state s t Take action a t The expected return, γ represents the discount factor, and the state value function V(s t+1 ) represents the next state s t+1 Expected return; The Actor network is responsible for the action output of the reinforcement learning agent, and the input is the state s of the current time step t , the probability distribution of the output action π(a|s); the loss function of the Actor network is Where e∈M represents the experience in the experience replay pool M, |M| represents the number of experiences in the experience replay pool; Q j (s t ,a t ) indicates that the jth Q Critic network is in state s t and action a t The output value under , takes the minimum value of the output of the two Q Critic networks; α represents the entropy regularization coefficient; The V Critic network is used to estimate the value of the state, and the input is the state s of the current time step t , the output is the state value function V(s t ), indicating that in state s t The expected return that the agent can obtain by following the current strategy, the loss function of the V Critic network is in, Indicates the current strategy π in state s t The action a sampled from the action probability distribution under t 'Expectations; The Q Critic network is used to evaluate the value of different actions, and the input is the state s of the current time step t , output action state value function Q(s t ,a t ), indicating that in state s t The next agent chooses action a t The expected return that can be obtained by following the current strategy, the loss function of the Q Critic network is Among them, r t For state s t Execute action a t The immediate return obtained after , γ is the discount factor; The calculation formula of the action strategy function is: H(π(·|s t ))=-logπ(·|s t ); Among them, π represents the strategy, E π [·] represents the mathematical expectation of the cumulative reward plus the entropy regularization term under strategy π, r(s t ,a t ) indicates state s t Next action a t The reward function is α, which represents the entropy regularization coefficient, H(π(·|s t )) is the policy π in state s t The entropy at , T represents the maximum number of iterations, π(·|s t ) means in state s t The strategy for selecting actions.

9. The security protection method for intelligent computing networks according to claim 7 is characterized in that: According to the task requirements and data characteristics, an attack agent neural network and a defense agent neural network are constructed. Both the attack agent neural network and the defense agent neural network are composed of a multi-layer perceptron. The MLP weights and biases are dynamically adjusted through the reward signal of the interaction training between the reinforcement learning agent and the environment to maximize the cumulative reward. Both the attack agent neural network and the defense agent neural network take the characteristics of the current sample as input. The output of the attack agent neural network is to select an attack type from multiple functional attack types and extract a random sample corresponding to the output attack type. The output of the defense agent neural network is a prediction of the intrusion category. The implementation method of the asynchronous update method is as follows: create multiple worker threads, each worker thread contains an independent attacking agent, a defending agent and a shared environment, and the environment is constructed by a feature subset obtained by preprocessing and feature selection; all worker threads execute training tasks in parallel, each worker thread interacts with the environment independently, each worker thread loads the same environment, and each maintains the parameters of its own thread agent; the data generated during the training process are used to calculate the losses of the policy network and the value network respectively, and the policy network and value network parameters of each agent are updated through the back propagation algorithm; after each update, the updated parameters are passed to the main network in an asynchronous manner after the training reaches a convergence state.

10. The security protection method for intelligent computing network according to claim 9, characterized in that: The implementation method of step 6 is to create sub-agents representing the attacker and defender in each worker thread, and conduct dynamic and real-time iterative confrontation in the same simulated network environment. In each round of training, both the attacker and defender will calculate the reward value based on the reward function and the current classification result obtained by the multi-layer perceptron of the defender; The attacker's sub-agent attempts to extract data of various attack types and observe whether the attack type can successfully bypass the defender's detection: if successful, the attacker's agent receives a positive reward, otherwise it receives a negative reward; the defender's agent calculates the reward value based on whether the attack traffic is correctly classified: if the classification is accurate, the defender's agent receives a positive reward, otherwise it will be punished.

Images