A Network Adversarial Decision-Making Method Based on Reinforcement Learning

Abstract

This invention discloses a network adversarial decision-making method based on reinforcement learning, belonging to the technical field of network adversarial science. Based on the characteristics of the network adversarial process, it models the process as a partially observable Markov decision process and utilizes the state transition characteristics of the network environment to construct a network environment representation model through unsupervised and self-supervised graph tasks. Combining this network environment representation model with a reinforcement learning algorithm, a network adversarial decision-making model is constructed, forming corresponding agents for attack and defense scenarios, including attackers and defenders. Dynamic game theory is then conducted in a simulated network adversarial environment, continuously improving the decision-making quality by maximizing rewards. This invention transforms the network adversarial process into a partially observable Markov decision problem, combines graph neural networks with reinforcement learning algorithms to generate network adversarial strategies, and optimizes these strategies through dynamic game theory between agents.

Description

Technical Field

[0001] This invention relates to the technical field of network adversarial tactics, and in particular to a network adversarial decision-making method based on reinforcement learning. Background Technology

[0002] Applying reinforcement learning to cybersecurity is a current research hotspot, but several problems remain to be solved. First, current reinforcement learning policy optimization primarily employs an end-to-end training paradigm. Under this paradigm, the information used to improve decision-making ability is almost entirely limited to rewards, leading to a tendency for the model to overfit to the training environment and insufficient generalization ability to unknown environments. However, unknown environments and incomplete information are key characteristics of network adversarial systems, and stronger generalization ability allows decision-making models to make better decisions in unknown environments. Second, network adversarial systems involve combinations of multiple nodes and various technologies, and the node set can also change. Simply enumerating node and technology combinations in the policy generation part leads to an explosion in the action space dimension, making it difficult for the decision-making model to converge. Finally, in real-world adversarial scenarios, attackers and defenders need to deal with opponents with different action strategies. If attackers and defenders interact and train independently in the network environment, the network adversarial decision-making model struggles to adapt to unknown opponent strategies. In conclusion, improving the decision-making model's adaptability to unknown network environments, addressing the problem of high and dynamically changing action space dimensions, and enhancing the model's adaptability to different opponent strategies are crucial for building high-quality network adversarial decision-making models. Summary of the Invention

[0003] The purpose of this invention is to provide a network adversarial decision-making method based on reinforcement learning, which mainly includes four parts: network adversarial process modeling, network environment representation model, network adversarial decision-making model, dynamic game and strategy optimization. It transforms the network adversarial process into a partially observable Markov decision problem, combines graph neural networks and reinforcement learning algorithms to generate network adversarial strategies, and optimizes network adversarial strategies through dynamic game between attack / defense agents.

[0004] To achieve the above objectives, this invention provides a network adversarial decision-making method based on reinforcement learning, comprising the following steps:

[0005] S1: In the network environment, the attacker and defender are set as agents. Based on the assumption of Markov properties and the characteristics of network adversarial, the six-tuple (S,A,T,R,Ω,O) is used to define the state space, agent action space, agent observation space, state transition probability, conditional observation probability and reward function of the network environment.

[0006] S2: Establishing a network environment representation model based on graph neural networks;

[0007] S3: Combine the network environment representation model to construct a network adversarial decision model as the attacker and defender;

[0008] S4: The attacker and defender optimize the corresponding network adversarial decision-making model through a dynamic game process;

[0009] S5: Output the optimized attacker and defender models.

[0010] Preferably, the specific process of defining the six-tuple in step S1 is as follows:

[0011] In network adversarial scenarios, the six-tuple (S, A, T, R, Ω, O) is defined as follows:

[0012] S represents the state space of the network environment, and each state s t ∈S is described by network topology and node characteristics;

[0013] A is the action space of the agent, and each action a... t ∈A includes node identifier NID and technical identifier TID;

[0014] Ω represents the observation space of the agent, and each observation o t ∈Ω represents the feedback received from the environment after the agent performs an action;

[0015] T(s t+1 |s t ,a t ) represents the state transition probability, O(o t |s t+1 ,a t R(s) represents the conditional observation probability; t ,a t ) is the reward function, when s t+1 Compared to s t Positive rewards are given when it is more beneficial to the agent, and negative rewards are given when it is less beneficial.

[0016] Preferably, in step S2, the specific process of constructing the network environment representation model is as follows:

[0017] S21: Establish two auxiliary tasks, namely graph reconstruction and generation, to construct a network environment representation model;

[0018] Define the state of the network environment at time t. t The tuple (M) consisting of the network topology and node characteristics at that moment. t ,X t ), where N is the number of nodes, K is the feature dimension, and M∈{0,1} N×N Let be the adjacency matrix of the network. Given the feature matrix of all nodes, the agent's observation o at time t t Represented as a binary tuple (M) t ,X t The specific value depends on the predefined conditional observation probability O(o). t |s t+1 ,a t );

[0019] S22: The network environment representation model mainly includes graph autoencoder (GAE) and variational graph autoencoder (VGAE), which respectively achieve the parameter training of the neural network through unsupervised and self-supervised graph tasks.

[0020] S23: Unsupervised tasks reconstruct the state of the network environment, while self-supervised tasks generate the future network state based on the properties of Markov.

[0021] During training, the encoder in GAE and the encoder in VGAE responsible for generating the mean share the graph attention layer; when there are two shared encoders, the parameters of the two shared GAT layers in GAE and VGAE are defined as follows: And θ, the parameter of the GAT layer unique to VGAE is ρ;

[0022] The unsupervised task of GAE trains the parameters of the neural network by reconstructing the adjacency matrix and minimizing the reconstruction error, and generates the latent variable representation of the network environment through the encoder:

[0023]

[0024] In the above formula, For parameters GAT network, GAT θ For a GAT network with parameters θ, the reconstructed adjacency matrix is ​​obtained using a dot product decoder. σ is the activation function:

[0025]

[0026] It is Z GAE The transpose of the matrix is ​​used to calculate the true adjacency matrix M using the cross-entropy loss function. t With the reconstructed adjacency matrix The difference is used to obtain the loss function L of GAE. GAE :

[0027]

[0028] In the above formula, m ij For M t The element in the i-th row and j-th column, for The element in the i-th row and j-th column, the self-supervised task of VGAE generates the adjacency matrix for the next time step. Furthermore, the parameters of the neural network are trained by maximizing the lower bound of evidence for variational inference, and the distribution of latent variables is generated through the encoder:

[0029]

[0030] logσ=GAT ρ (GAT θ (M t ,X t ||D t ));

[0031]

[0032] In the above formula, D t Let μ be the embedding vector of the action. i The mean, Let z be the variance. i For the latent variables after sampling, GAT ρ For a GAT network with parameter ρ, It follows a normal distribution;

[0033] The latent variable Z is obtained by sampling from the distribution. VGAE The generated adjacency matrix is ​​obtained through the dot product decoder.

[0034]

[0035] Finally, the lower bound of evidence for VGAE is calculated, and the loss function L of VGAE is obtained. VGAE :

[0036]

[0037] In the above formula, β represents the weight. Combining GAE and VGAE, the overall loss function of the network environment representation model is:

[0038]

[0039] In the above formula, α represents the weight;

[0040] S24: Through the above steps, setting the loss function threshold yields the trained network environment representation model, where the GAE encoder accepts zero-padding vectors and observations o. t As input, after passing through parameters A graph neural network with θ generates a network representation of Z. GAE .

[0041] Preferably, in step S3, the specific process is as follows:

[0042] S31: The network adversarial decision-making model is constructed using the proximal policy optimization model (PPO), which includes a policy network (Actor) and a value network (Critic).

[0043] S32: For a value network, the input state s t and reward r t The front end of the value network is a graph neural network consisting of GAT and pooling operations, using the graph's adjacency matrix M. t and node features X t The graph representation vector u is calculated using the following formula when there are L GAT layers:

[0044]

[0045] In the above formula, Pool l For the l-th graph pooling layer, GAT l This is the l-th GAT layer;

[0046] The state value V(s) is obtained using a fully connected network based on the graph's representation vector u. t ):

[0047] V(s t ) = MLP(u);

[0048] Based on Bellman's formula and generalized advantage estimation, the TD error δ is obtained. t And dominance value A(s) t ,a t ):

[0049] δ t =r t +γ·V(s t+1 )-V(s t );

[0050]

[0051] In the above formula, γ is the discount factor, λ is an adjustable hyperparameter, and the value network outputs the advantage value A(s). t ,a t );

[0052] For the policy network, it is necessary to consider the trajectory τ = {o1, o2, ..., o} consisting of all observations from the current time and before. t}, observation o t The GAE encoder, which follows the network environment representation model in step S2, GAE Then, it is input into a GRU model to obtain the node feature matrix H with temporal information.t :

[0053] H t =GRU(Encoder) GAE (M t ,X t ));

[0054] Due to observation o t The number of nodes is variable, so an attention mechanism is used to calculate the probability P(v) of each node being selected. i |τ), introduce a learnable parameter vector a, which is used as the query for the attention mechanism, and then the node feature vector h i As the key for the attention mechanism, the feature h of each node is calculated. i The importance coefficient c of parameter vector a i Then, the normalized importance coefficient α is obtained through the Softmax function. i and the coefficient α i The probability P(v) of being selected as a node i |τ), the formula is as follows:

[0055]

[0056]

[0057] In the above formula, W is the parameter matrix of the attention mechanism, and τ is the trajectory;

[0058] If the selected node is v i The next step is to consider the probability P(t) of executing a specific technique at that node. j |v i ,τ), here node v i Features h i The input is fed into a fully connected network, and after passing through the softmax function at the end, the probability P(t) of each technique execution is obtained. j |v i ,τ), the final attacker targets node v i Execution technology t j The probability is P(v) i ,t j |τ), the formula is as follows:

[0059] P(v i ,t j |τ)=P(v i |τ)·P(t j |v i ,τ);

[0060]

[0061] In the above formula, The j-th element is the vector obtained by passing the feature of the i-th node through MLP;

[0062] S33: Before use, the network decision model needs to be trained. The optimization process is as follows:

[0063] For the value network, its optimization objective is to minimize the error of the time difference method, and the optimization objective is:

[0064]

[0065] In the above formula, φ and V φ For the parameters of the Critic network, Calculate the expected value;

[0066] For a policy network, its optimization objective is to maximize the reward; therefore, the optimization objective of the Actor is:

[0067]

[0068] In the above formula, θ represents the parameters of the Actor network, and π... old This is the old strategy;

[0069] In addition, to encourage reinforcement learning to explore new states, an entropy regularization term is introduced:

[0070]

[0071] In summary, the loss function of the network adversarial decision-making model is as follows:

[0072] L PPO (θ,φ)=c1·L Critic (φ)-c2·L Actor (θ)-c3·L Entripy (θ);

[0073] In the above formula, c1, c2, and c3 are weighting coefficients, and L PPO (θ,φ) is the total loss function;

[0074] S34: The trained network decision-making models are used as the attacker and defender, respectively.

[0075] Preferably, in step S4, the specific process is as follows:

[0076] S41: Establish a dynamic game environment, including the attacker and defender, and their respective checkpoint pools.

[0077] S42: Optimize strategies using a dynamic game environment;

[0078] S421: Randomly initialize the attacker's and defender's checkpoint pools, and set their respective checkpoint pools to empty;

[0079] S422: Allow the attacker and defender to engage in multiple rounds of dynamic game in a network environment and collect battle data; each time, K checkpoints are randomly sampled from the checkpoint pool, with the attacker sampled as RC and the defender sampled as BC. The attacker plays against the K BCs and the defender plays against the K RCs. Based on their own network adversarial decision-making model, and according to the observation information and rewards returned by the network environment, the attacker and defender make decisions and execute them in sequence, so that the attacker and defender take turns making decisions and acting until the network attack or defense goal is achieved. The network environment state, rewards, and the actions and observations of the agents generated during the dynamic game are the battle data required for model training.

[0080] S423: Use the battle data to update the network adversarial decision model of the agent, and store the updated agent as a checkpoint in the corresponding checkpoint pool;

[0081] S43: Repeat steps S421-S423 until the rewards obtained by the agent in the dynamic game tend to converge, at which point the training is complete.

[0082] Preferably, in step S423, the process of updating the network adversarial decision model with battle data is as follows:

[0083] First, initialize an empty experience replay pool. Simultaneously, K different network environments are initialized. Within each training cycle, a trajectory set is formed by interacting with all environments according to the current strategy. and each of them (o j ,a j ,o j+1 ) Tuples are stored in the experience replay pool Then, experience replay pools were used respectively. and trajectory set The data was used to train the network environment representation model and the network adversarial decision-making model.

[0084] Therefore, the network adversarial decision-making method based on reinforcement learning described above has the following advantages:

[0085] By analyzing the characteristics of network adversarial processes, this application models the network adversarial process as a Markov Decision Process (POMDP), which can be solved using reinforcement learning methods. Addressing the difficulty of traditional reinforcement learning methods in achieving good adaptability to unknown environments in network adversarial scenarios, this application proposes a network environment representation model based on graph neural networks. This model achieves efficient representation of the network environment through unsupervised graph reconstruction and self-supervised generation tasks, reducing dependence on reward functions and improving generalization ability to unknown environments. To address the issues of variable node sets and high-dimensional action spaces, this application proposes a network adversarial decision-making model based on reinforcement learning. On the one hand, it introduces the network environment representation model into the Actor part of the PPO to improve generalization ability; on the other hand, it decomposes the decision-making process into node selection and technique selection, and uses an attention mechanism to handle variable-length node sequences, improving the model's convergence performance. Finally, to enable the decision-making model to cope with various adversarial strategies, a dynamic game-based strategy optimization framework is proposed. This framework allows the agent to simulate adversarial interactions with multiple defenders and attackers at checkpoints. The decision-making model is trained based on collected adversarial data, thereby improving the agent's decision-making capabilities.

[0086] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0087] Figure 1 This is an overall flowchart of a network adversarial decision-making method based on reinforcement learning according to the present invention;

[0088] Figure 2 This is a schematic diagram of the decision-making process in a network adversarial decision-making method based on reinforcement learning according to the present invention;

[0089] Figure 3 This is a schematic diagram of the network environment representation model in a network adversarial decision-making method based on reinforcement learning according to the present invention.

[0090] Figure 4 This is a schematic diagram of the network adversarial decision-making model in a reinforcement learning-based network adversarial decision-making method of the present invention.

[0091] Figure 5 This is a schematic diagram of the strategy optimization method based on dynamic game theory in the network adversarial decision-making method based on reinforcement learning of the present invention. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Specific model specifications need to be selected and determined according to the actual specifications of the device, etc. The specific selection calculation method adopts existing technology in the art, and therefore will not be described in detail.

[0093] Example

[0094] like Figure 1 As shown, this invention provides a network adversarial decision-making method based on reinforcement learning, comprising the following steps:

[0095] S1: In the network environment, the attacker and defender are set as agents. Based on the assumption of Markov properties and the characteristics of network adversarial, the six-tuple (S,A,T,R,Ω,O) is used to define the state space, agent action space, agent observation space, state transition probability, conditional observation probability and reward function of the network environment.

[0096] In network adversarial scenarios, the six-tuple (S, A, T, R, Ω, O) is defined as follows:

[0097] S represents the state space of the network environment, and each state s t ∈S is described by network topology and node characteristics;

[0098] A is the action space of the agent, and each action a... t ∈A includes node identifier NID and technical identifier TID;

[0099] Ω represents the observation space of the agent, and each observation o t ∈Ω represents the feedback received from the environment after the Agent performs an action;

[0100] T(s t+1 |s t ,a t ) represents the state transition probability, O(o t |s t+1 ,a t R(s) represents the conditional observation probability; t ,a t ) is the reward function, when s t+1 Compared to s t Positive rewards are given when it is more beneficial to the agent, and negative rewards are given when it is less beneficial.

[0101] S2: Establish a network environment representation model based on graph neural networks, the process is as follows:

[0102] S21: Establish two auxiliary tasks: graph reconstruction and generation, and construct a network environment representation model;

[0103] Define the state of the network environment at time t. t The tuple (M) consisting of the network topology and node characteristics at that moment. t ,X t ), where N is the number of nodes, K is the feature dimension, and M∈{0,1} N×N Let be the adjacency matrix of the network. Given the feature matrix of all nodes, the agent's observation o at time t t Represented as a binary tuple (M) t ,X t The specific value depends on the predefined conditional observation probability O(o). t |s t+1 ,a t );

[0104] S22: The network environment representation model mainly includes graph autoencoder (GAE) and variational graph autoencoder (VGAE), which respectively achieve the parameter training of the neural network through unsupervised and self-supervised graph tasks.

[0105] S23: Unsupervised tasks reconstruct the state of the network environment, while self-supervised tasks generate the future network state based on the properties of Markov.

[0106] During training, the encoder in GAE and the encoder in VGAE responsible for generating the mean share the graph attention layer; when there are two shared encoders, the parameters of the two shared GAT layers in GAE and VGAE are defined as follows: And θ, the parameter of the GAT layer unique to VGAE is ρ;

[0107] The unsupervised task of GAE trains the parameters of the neural network by reconstructing the adjacency matrix and minimizing the reconstruction error, and generates the latent variable representation of the network environment through the encoder:

[0108]

[0109] In the above formula, For parameters GAT network, GAT θ For a GAT network with parameters θ, the reconstructed adjacency matrix is ​​obtained using a dot product decoder. σ is the activation function:

[0110]

[0111] It is Z GAEThe transpose of the matrix is ​​used to calculate the true adjacency matrix M using the cross-entropy loss function. t Reconstructing the adjacency matrix The difference is used to obtain the loss function L of GAE. GAE :

[0112]

[0113] In the above formula, m ij For M t The element in the i-th row and j-th column, for The element in the i-th row and j-th column, the self-supervised task of VGAE generates the adjacency matrix for the next time step. Furthermore, the parameters of the neural network are trained by maximizing the lower bound of evidence for variational inference, and the distribution of latent variables is generated through the encoder:

[0114]

[0115] logσ=GAT ρ (GAT θ (M t ,X t ||D t ))

[0116]

[0117] In the above formula, D t Let μ be the embedding vector of the action. i The mean, Let z be the variance. i For the latent variables after sampling, GAT ρ For a GAT network with parameter ρ, It follows a normal distribution;

[0118] The latent variable Z is obtained by sampling from the distribution. VGAE The generated adjacency matrix is ​​obtained through the dot product decoder.

[0119]

[0120] Finally, the lower bound of evidence for VGAE is calculated, and the loss function L of VGAE is obtained. VGAE :

[0121]

[0122] In the above formula, β represents the weight. Combining GAE and VGAE, the overall loss function of the network environment representation model is:

[0123]

[0124] In the above formula, α represents the weight;

[0125] S24: Through the above steps, setting the loss function threshold yields the trained network environment representation model, where the GAE encoder accepts zero-padding vectors and observations o. t As input, after passing through parameters A graph neural network with θ generates a network representation of Z. GAE .

[0126] S3: Combining the network environment representation model, a network adversarial decision-making model is constructed to represent both the attacker and the defender. The specific process is as follows:

[0127] S31: The network adversarial decision-making model is constructed using the proximal policy optimization model (PPO), which includes a policy network (Actor) and a value network (Critic).

[0128] S32: For a value network, the input state s t and reward r t The front end of the value network is a graph neural network consisting of GAT and pooling operations, using the graph's adjacency matrix M. t and node features X t The graph representation vector u is calculated. Assume there are L GAT layers in total:

[0129]

[0130]

[0131] In the above formula, Pool l For the l-th graph pooling layer, GAT l For the l-th GAT layer; use a fully connected network to obtain the state value V(s) based on the graph representation vector u. t ):

[0132] V(s t ) = MLP(u);

[0133] Based on Bellman's formula and generalized advantage estimation, the TD error δ is obtained. t And dominance value A(s) t ,a t ):

[0134] δ t =r t +γ·V(s t+1 )-V(s t );

[0135]

[0136] In the above formula, γ is the discount factor, λ is an adjustable hyperparameter, and the value network outputs the advantage value A(s). t ,a t );

[0137] For the policy network, it is necessary to consider the trajectory τ = {o1, o2, ..., o} consisting of all observations from the current time and before. t}, observation o t The GAE encoder, which follows the network environment representation model in step S2, GAE Then, it is input into a GRU model to obtain the node feature matrix H with temporal information. t :

[0138] H t =GRU(Encoder) GAE (M t ,X t ))

[0139] Due to observation o t The number of nodes is variable, so an attention mechanism is used to calculate the probability P(v) of each node being selected. i |τ), introduce a learnable parameter vector a, which is used as the query for the attention mechanism, and then the node feature vector h i As the key for the attention mechanism, the feature h of each node is calculated. i The importance coefficient c of parameter vector a i Then, the normalized importance coefficient α is obtained through the Softmax function. i and the coefficient α i The probability P(v) of being selected as a node i |τ), the formula is as follows:

[0140]

[0141]

[0142] In the above formula, W is the parameter matrix of the attention mechanism, and τ is the trajectory; if the selected node is v i The next step is to consider the probability P(t) of executing a specific technique at that node. j |v i ,τ), here node v i Features h i The input is fed into a fully connected network, and after passing through the softmax function at the end, the probability P(t) of each technique execution is obtained. j |v i ,τ), the final attacker targets node v i Execution technology t jThe probability is P(v) i ,t j |τ), the formula is as follows:

[0143] P(v i ,t j |τ)=P(v i |τ)·P(t j |v i ,τ);

[0144]

[0145] In the above formula, The j-th element is the vector obtained by passing the feature of the i-th node through MLP;

[0146] S33: Before use, the network decision model needs to be trained. The optimization process is as follows:

[0147] For the value network, its optimization objective is to minimize the error of the time difference method, and the optimization objective is:

[0148]

[0149] In the above formula, φ and V φ For the parameters of the Critic network, Calculate the expected value;

[0150] For a policy network, its optimization objective is to maximize the reward; therefore, the optimization objective of the Actor is:

[0151]

[0152] In the above formula, θ represents the parameters of the Actor network, and π... old This is the old strategy;

[0153] In addition, to encourage reinforcement learning to explore new states, an entropy regularization term is introduced:

[0154]

[0155] In summary, the loss function of the network adversarial decision-making model is as follows:

[0156] L PPO (θ,φ)=c1·L Critic (φ)-c2·L Actor (θ)-c3·L Entropy (θ)

[0157] In the above formula, c1, c2, and c3 are weighting coefficients, and L PPO (θ,φ) is the total loss function;

[0158] S34: The trained network decision-making models are used as the attacker and defender, respectively.

[0159] S4: The attacker and defender optimize the corresponding network adversarial decision-making model through a dynamic game process, as follows:

[0160] S41: Establish a dynamic game environment, including the attacker and defender, and their respective checkpoint pools.

[0161] S42: Optimize strategies using a dynamic game environment;

[0162] S421: Randomly initialize the attacker's and defender's checkpoint pools, and set their respective checkpoint pools to empty;

[0163] S422: Allow the attacker and defender to engage in multiple rounds of dynamic game in a network environment and collect battle data; each time, K checkpoints are randomly sampled from the checkpoint pool, with the attacker sampled as RC and the defender sampled as BC. The attacker plays against the K BCs and the defender plays against the K RCs. Based on their own network adversarial decision-making model, and according to the observation information and rewards returned by the network environment, the attacker and defender make decisions and execute them in sequence, so that the attacker and defender take turns making decisions and acting until the network attack or defense goal is achieved. The network environment state, rewards, and the actions and observations of the agents generated during the dynamic game are the battle data required for model training.

[0164] S423: Update the network adversarial decision-making model of the agent using the corresponding battle data. The process of updating the network adversarial decision-making model with battle data is as follows:

[0165] First, initialize an empty experience replay pool. Simultaneously, K different network environments are initialized. Within each training cycle, a trajectory set is formed by interacting with all environments according to the current strategy. and each of them (o j ,a j ,o j+1 ) Tuples are stored in the experience replay pool Then, experience replay pools were used respectively. and trajectory set The data is used to train the network environment representation model and the network adversarial decision-making model. The corresponding algorithms are executed as follows:

[0166] Input: K different network environments Learning rates α1, α2, loss function weights c1, c2, c3, executed as follows:

[0167] Initialize the experience replay pool Perform the following operations until convergence:

[0168] Initialize trajectory set For each environment With strategy π θ Interaction yields trajectory and trajectory All of (o) j ,a j ,o j+1 ) Tuples are stored in the experience replay pool

[0169] Within each training cycle of the network environment representation model, from the experience replay pool Medium sampling yields small-batch experience L is calculated based on small-batch experience. feat :

[0170]

[0171] During the reinforcement learning training cycle, from trajectory cache Medium sampling yields small batches of trajectories. L is calculated based on the mini-batch trajectory d. Actor L Critic and L Entropy :

[0172]

[0173] The updated agent is stored as a Checkpoint in the corresponding Checkpoint pool;

[0174] S43: Repeat steps S421-S423 until the rewards obtained by the agent in the dynamic game tend to converge, at which point the training is complete.

[0175] Therefore, this invention employs a network adversarial decision-making method based on reinforcement learning. By analyzing the characteristics of network adversarial processes, this application models the network adversarial process as a Markov Decision Process (POMDP), and the POMDP problem can be solved using reinforcement learning methods. Addressing the difficulty of traditional reinforcement learning methods in achieving good adaptability to unknown environments in network adversarial scenarios, this application proposes a network environment representation model based on graph neural networks. Through unsupervised graph reconstruction and self-supervised generation tasks, it achieves efficient representation of the network environment, reduces dependence on reward functions, and improves generalization ability to unknown environments. Addressing the issues of variable node sets and high-dimensional action spaces, this application proposes a network adversarial decision-making model based on reinforcement learning. On the one hand, it introduces a network environment representation model into the Actor part of the PPO to improve generalization ability; on the other hand, it decomposes the decision-making process into node selection and technique selection, and uses an attention mechanism to handle variable-length node sequences, improving the model's convergence performance. Finally, to enable the decision-making model to cope with various adversary action strategies, a strategy optimization framework based on dynamic game theory is proposed. The agent simulates adversarial confrontation with multiple defenders and attackers at checkpoints. The decision-making model is trained based on the collected adversarial data, thereby improving the agent's decision-making ability.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A network adversarial decision-making method based on reinforcement learning, characterized in that: Includes the following steps: S1: In a network environment, the attacker and defender are set as intelligent agents. Based on the Markov property assumption and the characteristics of network adversarial interactions, a six-tuple is used. Define the state space, agent action space, state transition probability, reward function, agent observation space, and conditional observation probability of the network environment; S2: A network environment representation model is established based on a graph neural network. The specific construction process is as follows: S21: Establish two auxiliary tasks: graph reconstruction and generation; Define time Network environment status The tuple consisting of the network topology and node characteristics at this moment. ,in For the number of nodes, For feature dimension, Let be the adjacency matrix of the network. Given the feature matrix of all nodes, the agent at time... Observations Represented as a pair The specific value depends on the predefined conditional observation probability. ; S22: The network environment representation model includes the graph autoencoder (GAE) and the variational graph autoencoder (VGAE), which respectively train the parameters of the neural network through unsupervised and self-supervised graph tasks. S23: Unsupervised tasks reconstruct the state of the network environment, while self-supervised tasks generate the future network state based on the properties of Markov. During training, the encoder in GAE and the encoder in VGAE responsible for generating the mean share the graph attention layer; when there are two shared encoders, the parameters of the two shared GAT layers in GAE and VGAE are defined as follows: and The parameters of VGAE's unique GAT layer are: ; The unsupervised task of GAE trains the parameters of the neural network by reconstructing the adjacency matrix and minimizing the reconstruction error, and generates the latent variable representation of the network environment through the encoder: ； In the above formula, For parameters GAT network, For parameters The GAT network is used to obtain the reconstructed adjacency matrix using a dot product decoder. , For activation function: ； yes The transpose of the matrix is ​​used to calculate the true adjacency matrix using the cross-entropy loss function. With the reconstructed adjacency matrix The difference is used to obtain the loss function of GAE. : ； In the above formula, for The Line 1 Column elements, for The Line 1 Column elements, VGAE's self-supervised task generates the adjacency matrix for the next time step. Furthermore, the parameters of the neural network are trained by maximizing the lower bound of evidence for variational inference, and the distribution of latent variables is generated through the encoder: ； ； ； ； In the above formula, The embedding vector for the action. The mean, For variance, These are latent variables after sampling. For parameters GAT network, It follows a normal distribution; Latent variables are obtained by sampling from the distribution. The generated adjacency matrix is ​​obtained through the dot product decoder. : ； Finally, the lower bound of evidence for VGAE is calculated, and the loss function of VGAE is obtained. : ； ； In the above formula, Using weights, combining GAE and VGAE, the overall loss function of the network environment representation model is: ； In the above formula, Indicates weight; S24: Through the above steps, setting the loss function threshold yields the trained network environment representation model. The GAE encoder accepts zero-padding vectors and observations. As input, after passing through parameters and Graph neural networks, generative network representation ; S3: Combine the network environment representation model to construct a network adversarial decision model as the attacker and defender; S4: The attacker and defender optimize the corresponding network adversarial decision-making model through a dynamic game process; S5: Output the optimized attacker and defender models.

2. The network adversarial decision-making method based on reinforcement learning according to claim 1, characterized in that: In step S1, the specific process of defining the six-tuple is as follows: hexagram In network adversarial scenarios, the specific definitions are as follows: For the state space of the network environment, each state Described through network topology and node characteristics; For the action space of the agent, each action Includes node identifier Technical Identification ; For the observation space of the intelligent agent, each observation It is the feedback received from the environment after the intelligent agent performs an action; Let be the state transition probability. For conditional observation probabilities; For the reward function, when compared to Positive rewards are given when it is more beneficial to the agent, and negative rewards are given when it is less beneficial.

3. The network adversarial decision-making method based on reinforcement learning according to claim 2, characterized in that: In step S3, the specific process is as follows: S31: The network adversarial decision-making model is constructed using the Proximal Policy Optimization (PPO) model, including the policy network. and value network ; S32: For a value network, the input state and rewards The front end of the value network is a graph neural network consisting of GAT and pooling operations, using the graph's adjacency matrix. and node features The representation vector of the graph is calculated. When set There are GAT layers, and the formula is as follows: ； ； In the above formula, For the first Individual pooling layers, For the first One GAT layer; Using a fully connected network based on the graph's representation vector Obtain the state value : ； Based on the Bellman formula and generalized advantage estimation, the TD error is obtained. and advantage value : ； ； In the above formula, As a discount factor, As adjustable hyperparameters, the value network outputs an advantage value. ; For policy networks, it is necessary to consider the trajectory composed of all observations from the current time step to the previous time step. , observation The GAE encoder after the network environment representation model in step S2 Then, it is input into a GRU model to obtain a node feature matrix with temporal information. : ； Due to observation Since the number of nodes is variable, an attention mechanism is used to calculate the probability of each node being selected. Introducing a learnable parameter vector This is used as the query for the attention mechanism, and then the node feature vectors are... The key for the attention mechanism is used to calculate the features of each node. With parameter vector Importance coefficient Then, the normalized importance coefficients are obtained through the Softmax function. and the coefficients The probability of being selected as a node The formula is as follows: ； ； In the above formula, Here is the parameter matrix for the attention mechanism. For trajectory; If the selected node is The next step is to consider the probability of executing a specific technique at that node. Here will be the nodes Features The input is fed into a fully connected network, and the probability of each technique being executed is obtained by passing through the softmax function at the end. Ultimately, the attacker targeted the node. Execution technology The probability is The formula is as follows: ； ； In the above formula, For the first The vector obtained by MLP from the features of the nth node is the first... Bit; S33: Before use, the network decision model needs to be trained. The optimization process is as follows: For the value network, its optimization objective is to minimize the error of the time difference method, and the optimization objective is: ； In the above formula, and For the parameters of the Critic network, Calculate the expected value; For a policy network, its optimization objective is to maximize the reward; therefore, the optimization objective of the Actor is: ； In the above formula, For the parameters of the Actor network, This is the old strategy; In addition, to encourage reinforcement learning to explore new states, an entropy regularization term is introduced: ； In summary, the loss function of the network adversarial decision-making model is as follows: ； In the above formula, , , These are the weighting coefficients. This is the total loss function; S34: The trained network decision-making models are used as the attacker and defender, respectively.

4. The network adversarial decision-making method based on reinforcement learning according to claim 2, characterized in that: In step S4, the specific process is as follows: S41: Establish a dynamic game environment, including the attacker and defender, and their respective checkpoint pools. S42: Optimize strategies using a dynamic game environment; S421: Randomly initialize the attacker's and defender's checkpoint pools, and set their respective checkpoint pools to empty; S422: Enables attackers and defenders to engage in multiple rounds of dynamic gameplay within a network environment and collects battle data; each round randomly samples data from the Checkpoint pool. There are several checkpoints, with the attacker sampling as RC and the defender sampling as BC. Let the attacker and defender... BC, defender and Each RC, based on its own network adversarial decision-making model, makes and executes decisions in sequence according to the observation information and rewards returned by the network environment, so that the attacker and defender take turns making decisions and acting until the network attack or defense goal is achieved. The network environment state, rewards, and the actions and observations of the agents generated in the dynamic game process are the battle data required for model training. S423: Use the battle data to update the network adversarial decision model of the agent, and store the updated agent as a checkpoint in the corresponding checkpoint pool; S43: Repeat steps S421-S423 until the rewards obtained by the agent in the dynamic game tend to converge, at which point the training is complete.

5. The network adversarial decision-making method based on reinforcement learning according to claim 4, characterized in that: In step S423, the process of updating the network adversarial decision-making model with battle data is as follows: First, initialize an empty experience replay pool. At the same time, initialize In each training cycle, a trajectory set is formed by interacting with all environments in different network environments according to the current strategy. and each of them Tuples are stored in the experience replay pool Then, the experience replay pool was used respectively. and trajectory set The data was used to train the network environment representation model and the network adversarial decision-making model.

Images