A security protection method for a wisdom computing network

Abstract

The application provides a security protection method for a wisdom calculation network, and the steps are as follows: network flow data sets are preprocessed and divided into a training set and a test set; feature selection is performed, and an optimal feature subset is explored and searched in a feature space; a network intrusion detection process is modeled as a Markov decision process, a feature subset is mapped to a state, an action in a neural network, and a reward function is designed; the SAC reinforcement learning algorithm is combined and an experience replay pool is built, and a reinforcement learning intelligent agent for network intrusion detection is constructed; an antagonistic attack intelligent agent and a defense intelligent agent are defined, and parameter synchronization is performed in an asynchronous updating mode; sub-intelligent agents of the attack and defense sides are respectively created for iterative confrontation; and the classification ability of the defense intelligent agent is evaluated by using the test set. The application performs feature selection, reduces the calculation amount, and improves the detection efficiency; and an antagonistic training mechanism is introduced to ensure that complex network attacks can be quickly and accurately detected and responded.

Description

Technical Field

[0001] This invention relates to the technical field of network security protection, and in particular to a security protection method for intelligent computing networks. Background Technology

[0002] As a crucial component of the next-generation information infrastructure, intelligent computing networks are gradually evolving into a core force driving the deep development and transformation of the digital economy. These networks permeate numerous sectors, including public services, finance, energy, transportation, commerce, and industrial manufacturing, directly impacting national security, social stability, and personal privacy. With the continuous expansion of intelligent computing networks, the security threats they face are escalating. Cyberattacks, by penetrating these networks, cause incalculable losses and harm, seriously threatening national political stability and social security. Building upon existing communication networks, intelligent computing networks are characterized by deep integration of network and computing power, massive heterogeneous terminal access, and uneven terminal security protection capabilities. Traditional intrusion detection technologies suffer from incomplete extraction of network data features, reliance on data labels, and data imbalance, resulting in models exhibiting long detection times, high false positive and false negative rates, and insufficient ability to identify new types of attacks. This makes it difficult to meet the demands for high detection rates, dynamic detection, and automatic learning and recognition. Therefore, there is an urgent need to develop a more efficient and intelligent intrusion detection technology that optimizes data processing through feature selection, reduces the impact of redundant and irrelevant features, and improves the ability to identify complex and covert network attacks by introducing an improved reinforcement learning algorithm framework to collect and analyze historical attack data. This will enable rapid response and adaptive learning, effectively enhance the ability to detect intelligent network threats, and ensure the effectiveness and timeliness of network security protection.

[0003] Patent application number 202410931255.3 discloses a method, device, equipment, and medium for intelligent computing network protection of computing devices. The method includes: extracting features from acquired data packets to obtain target features while the operating system of the computing device is in kernel mode; calling a preset network security model to perform attack detection on the target features and obtaining detection results; using the preset network security model to determine whether the data packets corresponding to the target features exhibit attack behavior; compiling the preset program into eBPF bytecode and mounting it to an XDP Hook in kernel mode; and executing the target operation corresponding to the detection results on the data packets in the XDP Hook. This invention discloses the detection of data packets with potential attack behavior before they reach the application layer, resulting in early detection. The network security model enables real-time and accurate analysis of data packets, and the detection process runs directly in kernel space, allowing for rapid detection of attack behavior. However, the method does not address how to effectively filter out the truly critical features for the detection task, potentially leading to significant computational burden when dealing with a large number of data packet features. While emphasizing early detection and real-time accurate analysis, this method may lack sufficient adaptability. Summary of the Invention

[0004] To address the technical problems of existing network intrusion detection algorithms, such as long detection time, high false positive and false negative rates, and insufficient ability to identify complex attack types, this invention proposes a security protection method for intelligent computing networks. This method decomposes network intrusion detection into two steps: feature selection and intrusion detection. It optimizes the response speed and accuracy of network intrusion detection, significantly reduces false positive and false negative rates, enhances the security and defense capabilities of intelligent computing networks, and improves resource utilization efficiency, thereby increasing the intelligence and automation level of network security protection. This invention employs the HSWOA algorithm for feature selection, eliminating redundant and irrelevant features, reducing computational load, and improving detection efficiency. An adversarial training mechanism is introduced to simulate real attack and defense scenarios, ensuring rapid and accurate detection and response to complex network attacks.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows: a security protection method for intelligent computing networks, comprising the following steps:

[0006] Step 1: Preprocess the network traffic dataset and divide the preprocessed dataset into training and testing sets;

[0007] Step 2: Use the hybrid strategy whale optimization algorithm to perform feature selection on the preprocessed training set. By simulating the hunting behavior of whales, explore potential optimal solution regions in the feature space and search for the optimal feature subset.

[0008] Step 3: Model the network intrusion detection process as a Markov decision process, define the four-tuple of state, action, reward, and discount factor, map the preprocessed dataset to the state and action in the neural network, and design the reward function;

[0009] Step 4: Combine the quadruple defined in Step 3 with the SAC reinforcement learning algorithm, build an experience replay pool, construct a reinforcement learning agent for network intrusion detection, define the neural network of the reinforcement learning agent, and design a loss function to optimize the policy and value estimation of the reinforcement learning agent.

[0010] Step 5: Based on the obtained reinforcement learning agent, define adversarial attacker and defender agents, initialize the parameters of the agent's neural network, and use an asynchronous update method to synchronize with the agent's policy network and value network parameters;

[0011] Step 6: Create sub-agents for both the attacker and defender sides for iterative adversarial training. In each round of training, the attacker and defender sides calculate reward values ​​based on the classification results of the defender's agent, update their respective sub-agent policies, and pass the parameters to the attacker's agent and the defender's agent.

[0012] Step 7: Process the test set using Step 2 and evaluate the performance of the defending agent. Calculate the accuracy, false positive rate, and false negative rate to assess the classifying ability of the defending agent.

[0013] Preferably, the preprocessing involves merging the CSV files in the network traffic dataset into a single CSV file; detecting and deleting missing values ​​in the merged CSV file; converting the character data in the CSV file into numerical data using one-Hot encoding technology; and normalizing the numerical data.

[0014] Preferably, the feature selection method of the hybrid strategy whale optimization algorithm is as follows: the training set is initialized to generate an initial population containing multiple potential feature combinations. The initial population is the initial position of the whale in the feature space. A Gaussian mutation mechanism is incorporated in the prey encirclement stage, and an adaptive inertial weight and mirror selection strategy are introduced in the random prey search stage. The Gaussian mutation mechanism adjusts each feature dimension to a random number that follows a Gaussian distribution. The adaptive inertial weight is a dynamically adjusted parameter determined according to the current iteration number and the total iteration number.

[0015] Preferably, the Gaussian mutation mechanism is implemented as follows: based on the position X of the i-th whale in the j-th dimension in the current iteration t. ij (t), calculate the average μ of the i-th whale across all dimensions. i and standard deviation σ across all dimensions iThe position of the whale in the j-th dimension after Gaussian mutation is...

[0016] The adaptive inertia weight is T is the total number of iterations, ω max and ω min These are the maximum and minimum values ​​of the adaptive inertia weight, respectively.

[0017] The formula for updating the whale's position is:

[0018]

[0019] Among them, X * (t) represents the position of the optimal whale in the current iteration t, A is the coefficient related to the convergence factor, D is the distance between the current whale and the optimal whale, and X rand (t) is a randomly selected whale individual in the current iteration, X i (t+1) represents the position of the i-th whale in the next iteration;

[0020] The mirror selection strategy is implemented as follows: take a uniform random number r∈(0,1) in each feature dimension and determine whether it exceeds the threshold. Perform an inversion transformation on the feature dimension that exceeds the threshold to obtain the mirror solution. Merge the original solution and the mirror solution to form an extended population. Sort the individuals in the extended population according to the fitness function and select the individuals with higher fitness as the initial position of the next generation of whales.

[0021] Preferably, the modeled Markov decision process is defined as a quadruple M = <S, A, R, γ>, where S is the state space, A is the action space, R is the reward space, and γ is the reward discount factor; the state space S = {s1, s2, ..., s...} t This includes various statistics and metrics related to network traffic, where each state s t It is a feature vector describing the current state of network traffic; the agent in each state s t The following is based on the action strategy function π * (a t |s t The probability distribution of the output actions is sampled to obtain specific actions, forming the action space A = {a1, a2, ..., a...} t}, action a t Labels representing network traffic behavior; each action taken by the agent (a) t Each event generates an immediate reward r, and the reward value of the immediate reward r is recorded to form a reward space R = {r1, r2, ..., r}. t The reward function assigns higher reward values ​​to a few categories of abnormal traffic.

[0022] Preferably, the reward function is:

[0023]

[0024] in, Indicates the predicted action, a t Represents the actual action, and Total represents the total amount of data in the training set.

[0025] Preferably, the SAC reinforcement learning algorithm obtains experience in state transitions through interaction with the environment, and uses experience replay to update the policy and value function, adaptively learning the optimal policy for network attack detection.

[0026] The method for building the experience replay pool is as follows: In network intrusion detection, the state of the current iteration t is s. t Action a t The probability distribution π(a|s) based on the output of the Actor network t Choose action a t Input to state s t To obtain the reward r for the current iteration t. t Next state s t+1 The Boolean value d represents whether the interaction between the agent and the environment has reached a termination state. t At the same time, each piece of experience is associated with a weight parameter w calculated using time difference error. t Each experience is represented by a six-tuple <s t ,a t ,r t ,s t+1 ,d t ,w t Stored in the form of > in the experience replay pool;

[0027] The method for constructing a reinforcement learning agent for network intrusion detection is as follows: A reinforcement learning agent based on the SAC algorithm is designed. The reinforcement learning agent includes an Actor network, two V Critic networks, and two Q Critic networks. Among the two V Critic networks, the V Critic network is responsible for estimating the value of each state in real time according to the current policy. The target V Critic network serves as the target network, providing a stable value estimate as the learning objective. The target V Critic network is soft-updated through the V Critic network. The two Q Critic networks are used to independently evaluate the value of action-state pairs.

[0028] Preferably, the weighting parameter w t The calculation method is as follows:

[0029]

[0030] Where, δ t Q(s) represents the time difference error at the current time step. t ,a t ) indicates that in state s t Take action a t The expected return, γ represents the discount factor, and the state-value function V(s) t+1 ) represents the next state s t+1 Expected return;

[0031] The Actor network is responsible for the action output of the reinforcement learning agent, and the input is the state s of the current time step. t The probability distribution of the output action is π(a|s); the loss function of the Actor network is...

[0032]

[0033] Where e∈M represents the experience in the experience replay pool M, and |M| represents the number of experiences in the experience replay pool; Q j (s t ,a t ) indicates that the j-th Q-Critic network is in state s t and action a t The output value is taken as the minimum of the two Q Critic network outputs; α represents the entropy regularization coefficient;

[0034] The V Critic network is used to estimate the value of a state, with the input being the state s at the current time step. t The output is the state value function V(s) t ), indicating that in state s t The expected reward that the agent can obtain by following the current policy is given by the loss function of the V Critic network.

[0035]

[0036] in, This indicates that for the current policy π in state s t Action a is obtained by sampling from the following action probability distribution. t 'Expectations;

[0037] The Q Critic network is used to evaluate the value of different actions, and the input is the state s at the current time step. t Output action state value function Q(s) t ,a t ), indicating that in state s t The agent selects action a tThe expected return that can be obtained by following the current strategy is given by the loss function of the Q Critic network.

[0038]

[0039] Where, r t For state s t Perform action a t The immediate return obtained afterward, where γ is the discount factor;

[0040] The calculation formula for the action strategy function is as follows:

[0041]

[0042] H(π(·|s t ))=-logπ(·|s t );

[0043] Where π represents the strategy, E π [·] represents the mathematical expectation of the cumulative reward entropy regularization term under policy π, r(s) t ,a t ) represents state s t Next action a t The reward function, where α represents the entropy regularization coefficient, H(π(·|s t )) is the policy π in state s t The entropy at point T represents the maximum number of iterations, π(·|s t ) indicates that in state s t The strategy for selecting actions.

[0044] Preferably, an attack agent neural network and a defense agent neural network are constructed according to task requirements and data characteristics. Both the attack agent neural network and the defense agent neural network consist of a multilayer perceptron. The weights and biases of the MLP are dynamically adjusted by the reward signal trained through the interaction between the reinforcement learning agent and the environment to maximize the cumulative reward. Both the attack agent neural network and the defense agent neural network take the features of the current sample as input. The output of the attack agent neural network is to select an attack type from multiple functional attack types and extract a random sample corresponding to the output attack type. The output of the defense agent neural network is a prediction of the intrusion category.

[0045] The asynchronous update method is implemented as follows: multiple worker threads are created, each containing an independent attack agent, a defense agent, and a shared environment. The environment is constructed using a subset of features obtained through preprocessing and feature selection. All worker threads execute training tasks in parallel, with each worker thread interacting independently with the environment. Each worker thread loads the same environment and maintains the parameters of its own thread agent. The loss of the policy network and value network is calculated using the data generated during training, and the policy network and value network parameters of each agent are updated using the backpropagation algorithm. After each update, the updated parameters are asynchronously passed to the main network after training reaches convergence.

[0046] Preferably, the implementation method of step six is ​​to create sub-agents representing the attacker and defender in each worker thread, and conduct dynamic and real-time iterative adversarial confrontation in the same simulated network environment. In each round of training, both the attacker and the defender will calculate the reward value based on the reward function and the current classification result obtained by the defender's multilayer perceptron.

[0047] The attacker's sub-agent attempts to extract data on various attack types and observe whether the attack types can successfully bypass the defender's detection: if successful, the attacker's agent receives a positive reward, otherwise it receives a negative reward; the defender's agent calculates the reward value based on whether the attack traffic is correctly classified: if the classification is accurate, the defender's agent receives a positive reward, otherwise it will be penalized.

[0048] Compared with existing technologies, the beneficial effects of this invention are as follows: First, the original network traffic dataset is preprocessed using techniques such as one-hot encoding, and the processed data is divided into training and testing sets. Second, the HSWOA (Hybrid Strategy Whale Optimization Algorithm) algorithm is used for feature selection, simulating the hunting behavior of whales to search for the optimal feature subset for intrusion detection tasks, eliminating redundant and irrelevant features, effectively reducing computational load, improving detection efficiency, and reducing false positives and false negatives caused by interference from irrelevant features. Third, the SAC (Soft Actor-Critic) algorithm is improved, and A2SAC (Adversarial Asynchronous Soft Actor-Critic) is proposed. The Actor-Critic algorithm constructs a neural network model, initializes parameters, and maps preprocessed data to states, actions, and reward values. Combined with an experience replay pool technique, it stably and efficiently optimizes network parameters. Through asynchronous updates, it efficiently utilizes computational resources during training, accelerating model convergence and significantly improving the training efficiency of the network intrusion detection system. Finally, an adversarial training mechanism is introduced, defining two agents—an attacker and a defender—for iterative adversarial training to simulate real-world attack and defense scenarios. Both sides continuously optimize classification strategies through reinforcement learning, ensuring that the generated model can quickly and accurately detect and respond to complex and ever-changing network attacks, maintaining a high detection rate and low false positive rate, thus comprehensively improving the security and reliability of the network intrusion detection system. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 The overall architecture diagram of the HSWOA-A2SAC network intrusion detection model of the present invention.

[0051] Figure 2 The structural diagram of the A2SAC intelligent agent based on a multilayer perceptron of the present invention.

[0052] Figure 3 Architecture diagram of the A2SAC network intrusion detection model of the present invention.

[0053] Figure 4 The neural network structure diagrams of the attacking agent and the defending agent of the present invention are shown in the figure, wherein (a) is the neural network of the attacking agent and (b) is the neural network of the defending agent.

[0054] Figure 5 The HSWOA-A2SAC model of this invention is based on the confusion matrix of the CSE-CIC-IDS2018 dataset, where (a) is the binary confusion matrix and (b) is the multi-class confusion matrix. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] With the continuous development of attack techniques, network attack methods are becoming increasingly diversified and covert. However, existing network intrusion detection systems suffer from insufficient adaptability to dynamically changing network environments, high false positive and false negative rates, and neglect of resource consumption during the detection process, leading to decreased detection accuracy and efficiency as well as high deployment costs. To address these issues, this invention proposes a security protection method for intelligent computing networks—(Hybrid Strategy Whale Optimization Algorithm-Adversarial Asynchronous Soft Actor-Critic, HSWOA-A2SAC). The overall architecture of HSWOA-A2SAC is as follows: Figure 1 As shown. HSWOM-A2SAC utilizes feature selection to eliminate redundant features, effectively controlling resource consumption. By simulating real-world attack and defense scenarios, it dynamically adjusts the detection strategy to adapt to changes in the network environment, improving adaptability and optimizing the detection algorithm to enhance accuracy and efficiency. The specific workflow steps of this invention are as follows:

[0057] Step 1: The network traffic dataset undergoes comprehensive preprocessing using techniques such as one-hot encoding. The preprocessed dataset is then divided into training and testing sets for subsequent model training and performance evaluation. Specifically:

[0058] This invention selects 10 CSV files from the CSE-CIC-IDS2018 network traffic dataset obtained from the official website of the Canadian Cyber ​​Security Institute (CIC). The dataset contains seven attack scenarios: brute force, heartbleed, botnet, DoS, DDoS, network attack, and insider network penetration. The CSV files include network traffic and system logs captured for each machine, as well as 80 features extracted from the network traffic captured using the CICFlowMeter-V3 tool, mainly including network traffic information such as Dst Port, Protocol, Timestamp, and Flow Duration.

[0059] This invention first unifies CSV files into a single CSV file through a merging operation, facilitating subsequent processing. Secondly, it performs missing value detection and deletion to ensure no missing records in the dataset, guaranteeing data integrity. Then, it employs one-Hot encoding technology to convert character data into numerical data, meeting the input data format requirements of machine learning models. Finally, it normalizes the numerical data, shrinking the values ​​to the range [0,1] to eliminate the effects of dimensional differences and uneven distribution.

[0060] In Python software, the `train_test_split` method from the scikit-learn library, combined with the `StratifiedShuffleSplit` cross-validation iterator, is used to randomly sample the preprocessed dataset. The preprocessed dataset is then evenly divided into training, validation, and test sets at a ratio of 70%, 15%, and 15%, and the corresponding CSV files are saved to ensure its use in subsequent model training.

[0061] Step 2: The Hybrid Strategy Whale Optimization Algorithm (HSWOA) is used for feature selection on the preprocessed dataset. By simulating the hunting behavior of whales, it explores potential optimal solution regions in the feature space, searches for the optimal feature subset, and thus eliminates redundant and irrelevant features to improve the efficiency and accuracy of the model. Specifically:

[0062] The Whale Optimization Algorithm (WOA) is an optimization algorithm that simulates the predatory behavior of whales. WOA consists of three actions: surrounding the prey, spiraling attack on the prey, and randomly searching for prey. HSWOA incorporates a Gaussian mutation mechanism in the surrounding prey phase and introduces adaptive inertia weights and a mirror selection strategy in the random prey search phase to improve the algorithm's performance in feature selection tasks.

[0063] In the initialization phase of HSWOA, the preprocessed dataset is first initialized to generate an initial population containing multiple potential feature combinations. The generated initial population (feature combinations) is regarded as the initial position of the whale in the feature space. In order to increase the diversity of the population, that is, to cover a wider feature space during the search process, this invention introduces a Gaussian mutation mechanism to adjust each feature dimension to a random number that follows a Gaussian distribution.

[0064] For the i-th whale (i.e., the i-th solution), the position X in the j-th dimension is... ij First, calculate the average value μ of the i-th whale across all dimensions. i and standard deviation σ across all dimensions i The calculation formulas are shown in equations (1) and (2).

[0065]

[0066]

[0067] Where d represents the number of dimensions, X ij (t) represents the position of the i-th whale in the j-th dimension at time t. The position X of the whale in the j-th dimension, calculated using the Gaussian mutation mechanism proposed in this invention, is... i ′ j (t+1) is shown in equation (3).

[0068]

[0069] Position X i ′ j The result of (t+1) is a probability, which is a random number that follows a Gaussian distribution. The actual value of the final position is converted to 0 or 1 according to the probability.

[0070] During the iterative search phase of HSWOA, the whale adjusts its position based on the distance between its current position and the optimal position, as well as the convergence factor. This invention balances the global and local search capabilities by introducing adaptive inertial weights.

[0071] The adaptive inertia weight is a dynamically adjusted parameter determined based on the current iteration number and the total number of iterations. In the early stages of iteration, a larger inertia weight allows the whale to extensively explore the feature space and search for potential global optima. As the iteration progresses, the inertia weight gradually decreases, and the whale begins to perform a detailed search within the discovered high-quality regions. The formula for calculating the adaptive inertia weight is shown in equation (4).

[0072]

[0073] During the iterative search phase, the whale's position update formula incorporates adaptive inertia weights, where the whale's position X...i The update formula is shown in equation (5).

[0074]

[0075] Among them, X * (t) represents the position of the optimal whale in the current iteration, A is the coefficient related to the convergence factor, D is the distance between the current whale and the optimal whale, and X... rand (t) is a randomly selected whale individual in the current iteration. X i (t+1) represents the position of the i-th whale in the next iteration.

[0076] In each iteration, a mirror selection strategy is employed to update the whale's position, further improving the algorithm's search efficiency and convergence speed. This invention generates a uniformly random number r∈(0,1) in each dimension and determines whether it exceeds a threshold (set to 0.5 in this invention). For feature dimensions exceeding the threshold, a negation transformation is performed to obtain a mirror solution; dimensions exceeding the threshold are transformed, and vice versa. The original and mirror solutions are merged to form an expanded population. This avoids getting trapped in local optima during the search process, improving global search capability. Individuals in the expanded population are sorted according to a fitness function, and individuals with higher fitness are selected as the initial position for the next generation of whales. The population quality is continuously optimized to improve the algorithm's convergence speed and solution accuracy.

[0077] In summary, Gaussian mutation is used to increase population diversity during the initialization phase, adaptive inertia weights balance the ability of global search and local exploitation during the iterative search phase, and the mirror selection strategy optimizes the whale's position update process in each iteration. HSWOA improves WOA by combining these three strategies, removing redundant and irrelevant features from network traffic data, effectively improving detection efficiency, and reducing false positives and false negatives caused by irrelevant feature interference, thus providing support for building efficient and accurate network intrusion detection models.

[0078] The final result obtained from HSWOA is a feature subset. For the CSE-CIC-IDS2018 dataset with 80 features, the feature subset is a unigram array. Assuming the feature subset is "[1,2,6,7,11,13]", it means that features 1, 2, 6, 7, 11, and 13 are ultimately selected and retained. Subsequent steps directly modify the dataset, ignoring other feature columns and only retaining the columns corresponding to the feature subset, before performing detection on the modified dataset. The feature space refers to the entire space of all possible values ​​for n features (dimensions). For example, if there are 10 features, each taking only two values ​​(0 or 1), the feature space would be 2^n. 10 = 1024 possible points.

[0079] Step 3: Model the network intrusion detection process as a Markov decision process, defining a four-tuple of state, action, reward, and discount factor. Map the preprocessed dataset (i.e., the original dataset after feature subset processing) to the states and actions in the neural network, and design the reward function. Specific details are as follows:

[0080] This invention models the network attack detection process as a Markov Decision Process (MDP). The modeled MDP can be defined as a quadruple M = <S, A, R, γ>, where S is the state space, A is the action space, R is the reward space, and γ is the reward discount factor. Traditional MDPs typically include an explicit state transition probability P(s′|s,a), which is the probability that the environment transitions to the next state s′ given state s and action a. In this invention, the state transition probability is not explicitly modeled. The SAC algorithm is a model-free method based on reinforcement learning. It gains experience in state transitions through interaction with the environment and uses experience replay to update the policy and value function. The SAC algorithm adaptively learns the optimal policy for network attack detection without explicitly modeling the dynamic characteristics of the environment.

[0081] State space S = {s1, s2, ..., s} t This includes various statistical data and metrics of network traffic, such as packet size, protocol type, and traffic rate, where each state s t It is a feature vector that describes the current state of network traffic.

[0082] The agent in each state s t The following is based on the action strategy function π * (a t |s t The probability distribution of the output actions is sampled to obtain specific actions, i.e., labels for network traffic behavior, constituting the action space A = {a1, a2, ..., a...}. t}, action a t Labels representing network traffic behavior. The calculation formulas for the action policy function are shown in equations (6) and (7).

[0083]

[0084] H(π(·|s t ))=-logπ(·|s t (7)

[0085] Where π represents the strategy, E π [···] represents the mathematical expectation of the cumulative reward entropy regularization term under policy π, r(s t ,a t ) represents state s tNext action a t The reward function, where α represents the entropy regularization coefficient, determines the relative importance of the entropy term with respect to the reward, H(π(·|s t )) is the policy π in state s t The entropy at point T represents the maximum number of iterations. π(·|s t ) indicates that in state s t The strategy for selecting actions.

[0086] Each time the agent takes an action a t Both will generate an immediate reward r(s) t ,a t The recorded reward values ​​can form a reward space R = {r1, r2, ..., r}. t Due to the inherent imbalance of network traffic, normal traffic accounts for a much larger proportion of abnormal traffic in the training set, and the proportion of abnormal traffic varies among different categories. Therefore, this invention modifies the 0-1 reward function. In order for the classifier to identify a few categories of abnormal traffic, a higher reward value is assigned to a few categories of abnormal traffic, as shown in equation (8).

[0087]

[0088] in, Indicates the predicted action (label), a t The label represents the actual action, and Total represents the total amount of data in the training set.

[0089] Figure 1 Mini-batch random sampling in the dataset involves randomly sampling a group of data containing [s] from the original dataset. t ,a t ,s t+1 The use of triples avoids sample reuse and ensures that the dataset is explored comprehensively and in a balanced manner during iterative training.

[0090] Step 4: Combine the quadruple defined in Step 3 with the SAC reinforcement learning algorithm, build an experience replay pool, and construct a reinforcement learning agent for network intrusion detection. Optimize the agent's policy and value estimation by defining the neural network and designing a loss function. Specific details are as follows:

[0091] In network intrusion detection, the state at time t is s. t Action a t The probability distribution π(a|s) based on the output of the Actor network t Choose action a t Input to state s t To obtain the reward r for the current time step t Next state st+1 The Boolean value d represents whether the interaction between the agent and the environment has reached a termination state. t At the same time, each piece of experience is associated with a weight parameter w calculated using Temporal-Difference Error (TD Error). t The calculation formula is shown in equation (9).

[0092]

[0093] Where, δ t Q(s) represents the TD Error at the current time step. t ,a t ) indicates that in state s t Take action a t The expected return, γ represents the discount factor, and the state-value function V(s) t+1 ) represents the next state s t+1 Expected return. Each experience is represented by a six-tuple <s t ,a t ,r t ,s t+1 ,d t ,w t The experience is stored in the form of > in the experience replay pool. Each experience is given an attribute / tag, and different experience is assigned different priority weight parameters w. t This prioritizes samples to break down temporal correlations and improve sample utilization efficiency. During agent training, samples are randomly selected from the experience replay pool based on priority to update the neural network parameters, greatly improving training efficiency, stability, and convergence speed in reinforcement learning, ensuring that the agent can learn higher-quality policies in fewer training steps.

[0094] The present invention designs a reinforcement learning (A2SAC) agent based on the SAC algorithm during the agent construction process, aiming to learn and optimize decision-making strategies through interaction with network traffic data. Figure 2As shown, the A2SAC agent comprises one Actor network, two V Critic networks, and two Q Critic networks. The V Critic networks are responsible for estimating the value of each state in real-time according to the current policy, providing a crucial reference for the agent's action selection. The target V Critic network, serving as the objective network, is a delayed copy of the original network, providing a stable value estimate as a learning target, which helps reduce fluctuations during training. The actual outputs of the V Critic and target V Critic networks are identical; the target V Critic network is soft-updated through the V Critic network, and its policy may differ from the V Critic network. The two Q Critic networks are used to independently evaluate the value of action-state pairs, reducing estimation bias and improving the algorithm's robustness.

[0095] The Actor network is responsible for the agent's action output, and its input is the state s at the current time step. t The output action probability distribution π(a|s) is given, and each action has a corresponding probability value. The agent usually chooses actions with higher probabilities, but because the SAC algorithm encourages exploration, the agent may occasionally choose actions with lower probabilities to maintain the randomness and diversity of the strategy. This is usually achieved by adding an entropy term to the loss function. The loss function of the Actor network in this invention is shown in Equation (10).

[0096]

[0097] Where |M| represents the number of experiences used in a training step, Q j (s t ,a t ) indicates that the j-th Q-Critic network is in state s t and action a t The output value is taken as the minimum of the two Q-Critic network outputs to reduce overestimation. α represents the entropy regularization coefficient, used to encourage the Actor network to output more uncertain actions. The value of the entropy regularization coefficient has no fixed range; a larger entropy regularization coefficient α enhances exploratory behavior, while a smaller entropy regularization coefficient α focuses more on utilizing currently known information. In this invention, the value is 0.2. e∈M represents each experience in a training step.

[0098] The V Critic network is used to estimate the value of a state, and its input is the state s at the current time step. t Output state value function V(s) t ), indicating that in state s tThe expected reward that the agent can obtain by following the current strategy is given by the V Critic network loss function in this invention, as shown in equation (11).

[0099]

[0100] in, This indicates that for the current policy π in state s t Action a is obtained by sampling from the following action probability distribution. t The present invention considers adding a regularization term related to policy entropy to the loss function of the V Critic network to enhance the model's generalization ability.

[0101] The Q-Critic network is used to evaluate the value of different actions, and its input is the state s at the current time step. t Output action state value function Q(s) t ,a t ), indicating that in state s t The agent selects action a t The expected return that can be obtained by following the current strategy is given by the Q Critic network loss function in this invention, as shown in equation (12).

[0102]

[0103] Where, r t For state s t Perform action a t The immediate return obtained later, γ is a discount factor that controls the impact of future returns on the current value. The loss function of the Q Critic network optimizes the network parameters by minimizing the difference between the predicted value and the actual return.

[0104] The loss value calculated by the loss function is used to monitor the training process. If the loss value continues to decrease, it means that the neural network is learning effectively; if the loss value fluctuates or increases, it indicates that the learning rate, network structure or other hyperparameters need to be adjusted.

[0105] Step 5: Based on the reinforcement learning agent obtained in Step 4, define two adversarial agents: an attacker and a defender. The attacker is responsible for generating challenging network traffic data, while the defender is responsible for receiving and classifying the data. Initialize the weights and biases of the agent's neural network parameters, and use asynchronous updates to synchronize the policy and value network parameters with the main agent (i.e., the two adversarial agents). Specifically:

[0106] The architecture of attacking and defending agents is as follows: Figure 4As shown, in both cases, the implementation of the agent's neural network, whether for attack or defense, is influenced by the network structure and performance, depending on the task requirements and data characteristics. Both attack and defense agent neural networks are constructed from a shallow neural network—a multilayer perceptron (MLP)—with two layers of 100 neurons each and three layers of 150 neurons each, respectively. This architecture ensures a fast response time. The MLP weights and biases are dynamically adjusted based on the reward value calculated from the reward function, which is the reward signal generated during the reinforcement learning agent's interaction with the environment, to maximize the cumulative reward. The weights and biases of the agent neural network are randomly initialized and adjusted during training using the backpropagation algorithm. Figure 4 As can be seen, both the attacking and defending agents use the features of the current sample as input, but produce different outputs depending on the network. For the attacking agent, the output is selecting one of 15 attack types and drawing a random sample corresponding to that attack type; for the defending agent, the output is a prediction of the intrusion category. Therefore, the attacker's MLP can be viewed as a generator of attack categories, while the defender's MLP can be viewed as implementing a classifier.

[0107] This invention employs an asynchronous update approach to achieve efficient agent training. Specifically, multiple worker threads are created, each containing an independent attack agent, a defense agent, and a shared copy of the environment. The environment is constructed using a subset of features obtained through preprocessing and feature selection. All worker threads execute training tasks in parallel; each thread interacts with the environment independently, loads the same copy of the environment, and maintains the parameters of its own agent. The losses of the policy network and value network are calculated using the data collected during training, and the parameters of these networks are updated using the backpropagation algorithm. After each update, the agent parameters in the worker threads are not immediately synchronized to the main agent of the attacker / defender. Instead, the updated parameters are asynchronously passed to the main network after training reaches convergence and performance stabilizes. The main agent of the attacker / defender and the agents in the worker threads are constructed using the structure described in the preceding steps; they have the same structure, do not participate in training, and only receive parameters passed from the worker threads.

[0108] In asynchronous updates, each worker thread can independently explore different parts of the environment, thus avoiding getting trapped in local optima. This invention achieves efficient synchronization of parameters between the attacking and defending agents through this method, ensuring the stability and effectiveness of the entire training process, while fully utilizing the computing resources of multi-core CPUs to accelerate the training process.

[0109] The A2SAC network intrusion detection model architecture is as follows: Figure 3 As shown, the architecture integrates two adversarial agents, an attack agent and a defense agent, for adversarial training, which improves the model's detection accuracy and adaptability in complex network environments. It also adopts an asynchronous update strategy to maximize the use of computing resources and significantly shorten the training cycle.

[0110] Step Six: Initiate the asynchronous adversarial training process, creating sub-agents for both the attacker and defender sides for iterative adversarial training. In each round of training, both sides calculate reward values ​​based on the classification results of the defender's MLP, update their respective sub-agent policies, and pass the parameters to both the attacker and defender agents. Specifically:

[0111] This invention creates sub-agents representing the attacker and defender in each worker thread, and enables them to dynamically and in real-time iteratively engage in combat within the same simulated network environment to simulate attack and defense scenarios in a real network environment. In each round of training, both the attacker and defender calculate reward values ​​based on the reward function shown in Equation (8) and the current classification result obtained by the defender's MLP.

[0112] The attacker's sub-agent attempts to extract data on various attack types and observes whether each attack type can successfully bypass the defender's detection. If successful, the attacker's agent receives a positive reward; otherwise, it receives a negative reward. Similarly, the defender's agent calculates its reward based on whether it correctly classifies the attack traffic. If the classification is accurate, the defender's agent receives a positive reward; otherwise, it is penalized. Furthermore, because this invention assigns higher reward values ​​to a minority of attack categories in the reward function, both attacker and defender agents focus on these key categories during iterative adversarial processes, enabling dynamic balance and mutual progress for both agents in complex network environments. This invention simulates real-world attack and defense scenarios, with both agents iteratively optimizing their classification strategies using the ASA2C reinforcement learning algorithm. This ensures that the resulting model can quickly and accurately detect and respond to complex and ever-changing network attacks, maintaining a high detection rate and low false positive rate, thus comprehensively improving the security and reliability of the network intrusion detection system.

[0113] Step 7: Evaluate the performance of the defensive agent using a test set independent of the training set. Comprehensively assess the agent's classification ability by calculating key metrics such as accuracy, false positive rate, and false negative rate. Classification ability refers to the agent's ability to correctly categorize input data into predetermined classes. Using a test set independent of the training set to evaluate the agent's classification ability ensures good performance in real-world applications. Specifically:

[0114] In network intrusion detection, true positives (TP) are observed when attack events are correctly predicted, true negatives (TN) are observed when normal events are correctly predicted, false positives (FP) are observed when the model incorrectly predicts positive events, and false negatives (FN) are observed when the model incorrectly predicts negative events. There are various metrics for evaluating network intrusion detection algorithms. This invention uses evaluation metrics based on the confusion matrix, including Detection Rate (DR), False Alarm Rate (FAR), Accuracy (Acc), Precision (Pre), and F1 Score. The confusion matrix is ​​shown in Table 1.

[0115] Table 1 Confusion Matrix

[0116]

[0117] The detection rate (DR), also known as the true positives rate (TPR), represents the model's ability to correctly identify all true positive samples; in other words, it's the proportion of positive samples that the model correctly detects. In network intrusion detection, it represents the proportion of intrusion events that the model successfully detects. A high detection rate means the model can more accurately identify potential intrusion behaviors.

[0118]

[0119] The false alarm rate (FAR), also known as the false positive rate, refers to the proportion of negative samples that a model incorrectly identifies as positive samples. In network intrusion detection, it represents the proportion of normal behavior that the model mistakenly identifies as intrusion behavior. A low false alarm rate means that the model can effectively reduce false alarms.

[0120]

[0121] Accuracy (Acc) refers to the proportion of samples that a model correctly predicts, representing the overall prediction accuracy of the model.

[0122]

[0123] Precision (Pre) represents the proportion of positive samples correctly identified by a model. In network intrusion detection, it represents the model's accuracy among all samples identified as intrusions. High precision means the model is more reliable when issuing alerts and produces fewer false alarms.

[0124]

[0125] The F1-Score is the harmonic mean of precision and recall, which combines the accuracy and completeness of the model and helps to evaluate the balance between recall and precision.

[0126]

[0127] Where Pre represents precision and DR represents detection rate.

[0128] The experimental results of this invention on the test set are as follows: Figure 5 As shown, according to Figure 5 Based on the binary classification results of (a) and calculations using equations (13)-(17), the detection rate of this invention is approximately 97.99%, the false alarm rate is approximately 2.25%, the accuracy is approximately 97.95%, the precision is approximately 99.53%, the F1-Score is approximately 98.76%, and the prediction time in a test set with a sample size of 2,434,927 is 35.51 seconds, which is 14.58 microseconds for the prediction of a single sample. Figure 5 (b) is a classification of specific attack types, which shows the classification results for specific attack types.

[0129] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A security protection method for intelligent computing networks, characterized in that, The steps are as follows: Step 1: Preprocess the network traffic dataset and divide the preprocessed dataset into training and testing sets; Step 2: Use the hybrid strategy whale optimization algorithm to perform feature selection on the preprocessed training set. By simulating the hunting behavior of whales, explore potential optimal solution regions in the feature space and search for the optimal feature subset. Step 3: Model the network intrusion detection process as a Markov decision process, define the four-tuple of state, action, reward, and discount factor, map the preprocessed dataset to the state and action in the neural network, and design the reward function; Step 4: Combine the quadruple defined in Step 3 with the SAC reinforcement learning algorithm, build an experience replay pool, construct a reinforcement learning agent for network intrusion detection, define the neural network of the reinforcement learning agent, and design a loss function to optimize the policy and value estimation of the reinforcement learning agent. Step 5: Based on the obtained reinforcement learning agent, define adversarial attacker and defender agents, initialize the parameters of the agent's neural network, and use an asynchronous update method to synchronize with the agent's policy network and value network parameters; Step Six: Create sub-agents for both the attacker and defender sides for iterative adversarial training. In each round of training, the attacker and defender sides calculate reward values ​​based on the classification results of the defender's agent, update their respective sub-agent policies, and pass the parameters to the attacker's agent and the defender's agent. Step 7: Process the test set using Step 2 and evaluate the performance of the defending agent. Calculate the accuracy, false positive rate, and false negative rate to assess the classifying ability of the defending agent.

2. The security protection method for intelligent computing networks according to claim 1, characterized in that, The preprocessing involves merging the CSV files in the network traffic dataset into a single CSV file; detecting and deleting missing values ​​in the merged CSV file; and using one-Hot encoding technology to convert the character data in the CSV file into numerical data. Normalize numerical data.

3. The security protection method for intelligent computing networks according to claim 1 or 2, characterized in that, The hybrid strategy whale optimization algorithm implements feature selection as follows: the training set is initialized to generate an initial population containing multiple potential feature combinations. The initial population represents the initial position of the whale in the feature space. A Gaussian mutation mechanism is incorporated into the prey encirclement stage, and an adaptive inertial weight and mirror selection strategy are introduced into the random prey search stage. The Gaussian mutation mechanism adjusts each feature dimension to a random number that follows a Gaussian distribution. The adaptive inertial weight is a dynamically adjusted parameter determined based on the current iteration number and the total iteration number.

4. The security protection method for intelligent computing networks according to claim 3, characterized in that, The Gaussian mutation mechanism is implemented as follows: based on the first iteration t... The whale in the Dimensional position Calculate the first The average value of a whale across all dimensions and standard deviation across all dimensions ; The whale after the Gaussian mutation mechanism calculation is in the... The position of the dimension is ; The adaptive inertia weight is T is the total number of iterations. and These are the maximum and minimum values ​​of the adaptive inertia weight, respectively. The formula for updating the whale's position is: ； in, It is the position of the optimal whale in the current iteration t. These are coefficients related to the convergence factor. It is the distance between the current whale and the optimal whale. It is a randomly selected whale individual in the current iteration. For the next iteration The location of the whale; The mirror selection strategy is implemented by taking a uniformly random number for each feature dimension. It then determines whether the threshold is exceeded, performs an inverse transformation on the feature dimension that exceeds the threshold to obtain a mirror solution, merges the original solution and the mirror solution to form an extended population, and sorts the individuals in the extended population according to the fitness function, selecting individuals with higher fitness as the initial position of the next generation of whales.

5. The security protection method for intelligent computing networks according to claim 3, characterized in that, The modeled Markov decision process is defined as a quadruple. ,in It is a state space. It is the action space. It's a reward space. It is a reward discount factor; the state space This includes various statistics and metrics related to network traffic, with each state... It is a feature vector describing the current state of network traffic; the agent in each state Based on the action strategy function The probability distribution of the output action is sampled to obtain the specific action, which constitutes the action space. ,action Labels representing network traffic behavior; each action taken by the agent All of these will generate an immediate reward. Record instant returns The reward value forms the reward space. The reward function assigns higher reward values ​​to a few categories of abnormal traffic.

6. The security protection method for intelligent computing networks according to claim 5, characterized in that, The reward function is: ； in, Indicates the predicted action, To represent real actions, This represents the total amount of data in the training set.

7. The security protection method for intelligent computing networks according to any one of claims 4-6, characterized in that, The SAC reinforcement learning algorithm gains experience in state transitions through interaction with the environment and uses experience replay to update the policy and value function, adaptively learning the optimal policy for network attack detection. The method for building the experience replay pool is as follows: in the current iteration of network intrusion detection... The state is ,action Probability distribution based on Actor network output Select, Action Input to status Get the current iteration Rewards Next state The Boolean value indicating whether the interaction between the agent and the environment has reached a termination state. At the same time, each piece of experience is associated with a weight parameter calculated using time difference error. Each experience is represented by a six-tuple. Stored in the form of an experience replay pool; The method for constructing a reinforcement learning agent for network intrusion detection is as follows: A reinforcement learning agent based on the SAC algorithm is designed. The reinforcement learning agent includes an Actor network, two V Critic networks, and two Q Critic networks. Among the two V Critic networks, the V Critic network is responsible for estimating the value of each state in real time according to the current policy. The target V Critic network serves as the target network, providing a stable value estimate as the learning objective. The target V Critic network is soft-updated through the V Critic network. The two Q Critic networks are used to independently evaluate the value of action-state pairs.

8. The security protection method for intelligent computing networks according to claim 7, characterized in that, The weight parameters The calculation method is as follows: ； in, This represents the time difference error at the current time step. Indicates the state Take action Expected return Discount factor, state-value function Indicate the next state Expected return; The Actor network is responsible for the action output of the reinforcement learning agent, and the input is the state at the current time step. The probability distribution of output actions The loss function of the Actor network is: ； in, Indicates the experience replay pool The experience in This indicates the amount of experience in the experience replay pool; Indicates the first A Q-Critic network in state and actions The output value is the minimum of the two Q Critic network outputs. This represents the entropy regularization coefficient; The V Critic network is used to estimate the value of a state, with the state at the current time step as input. The output is a state value function. , indicating the state The expected reward that the agent can obtain by following the current policy is given by the loss function of the V Critic network. ； in, Indicates the current strategy In state Actions sampled from the following action probability distribution Expectations; The Q Critic network is used to evaluate the value of different actions, and the input is the state at the current time step. Output action state value function , indicating the state The agent selects actions The expected return that can be obtained by following the current strategy is given by the loss function of the Q Critic network. ； in, In the state Execute action The immediate reward obtained afterward Discount factor; The calculation formula for the action strategy function is as follows: ； ； in, Representation strategy, Representation strategy The expected value of the cumulative reward entropy regularization term. Representing state Next action The reward function, Represents the entropy regularization coefficient. It is a strategy In state Entropy at the location, Indicates the maximum number of iterations. Indicates the state The strategy for selecting actions.

9. The security protection method for intelligent computing networks according to claim 7, characterized in that, Based on task requirements and data characteristics, attack agent neural networks and defense agent neural networks are constructed. Both attack agent neural networks and defense agent neural networks consist of a multilayer perceptron (MLP). The weights and biases of the MLP are dynamically adjusted by the reward signal trained through the interaction between the reinforcement learning agent and the environment to maximize the cumulative reward. Both attack agent neural networks and defense agent neural networks use the features of the current sample as input. The output of the attack agent neural network is to select an attack type from multiple functional attack types and extract a random sample corresponding to the output attack type. The output of the defense agent neural network is a prediction of the intrusion category. The asynchronous update method is implemented as follows: multiple worker threads are created, each containing an independent attack agent, a defense agent, and a shared environment. The environment is constructed using a subset of features obtained through preprocessing and feature selection. All worker threads execute training tasks in parallel, with each worker thread interacting independently with the environment. Each worker thread loads the same environment and maintains the parameters of its own thread agent. The loss of the policy network and value network is calculated using the data generated during training, and the policy network and value network parameters of each agent are updated using the backpropagation algorithm. After each update, the updated parameters are asynchronously passed to the main network after training reaches convergence.

10. The security protection method for intelligent computing networks according to claim 9, characterized in that, The implementation method of step six is ​​to create sub-agents representing the attacker and defender in each worker thread, and conduct dynamic and real-time iterative adversarial battles in the same simulated network environment. In each round of training, both the attacker and defender will calculate the reward value based on the reward function and the current classification result obtained by the defender's multilayer perceptron. The attacker's sub-agent attempts to extract data on various attack types and observe whether the attack types can successfully bypass the defender's detection: if successful, the attacker's agent receives a positive reward, otherwise it receives a negative reward; the defender's agent calculates the reward value based on whether the attack traffic is correctly classified: if the classification is accurate, the defender's agent receives a positive reward, otherwise it will be penalized.

Images