A cyberspace intelligent game decision-making method and system

Abstract

A method and system for making decisions on cyberspace intelligent games, including: setting up a game space of a cyberspace intelligent game decision model and initializing it; establishing a mapping relationship between an intelligent agent reinforcement learning model and a cyberspace intelligent game decision model, and the cyberspace intelligent agent interacting with the game space based on the mapping relationship; in a round of game, the attacking agent and the defending agent make actions in each round of game according to their own strategies, and the attacking agent and the defending agent play multiple games until the number of game steps reaches a set threshold; the attacking agent and the defending agent play multiple rounds of game, and when the game reaches the maximum number of rounds and the maximum number of game steps per round, the cyberspace intelligent game is terminated, and the winner is determined according to the results and rules of the game. The present invention can solve the problem of how the defending agent makes automated and intelligent defense decisions when facing automated and intelligent network penetration attacks.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a cyberspace intelligent game decision-making method and system. Background Art

[0002] With the vigorous development of artificial intelligence technology, a large number of high-level AIs have defeated human players in the field of man-machine confrontation. There are more and more AI-enabled cyber attack methods and tools, which aim to train attack agents using intelligent methods such as reinforcement learning to achieve automated and intelligent cyber attacks. This has greatly reduced the technical threshold for cyber attacks and improved the intelligence level of cyberspace attack capabilities. However, defenders have inevitable defects when playing against automated and intelligent machines, mainly manifested in the physiological and psychological limits of humans, and humans are more susceptible to various external factors. The time and material costs of training an expert are high, and people's successful experience is not easy to replicate on a large scale. At the same time, defenders are constrained by static reinforcement defense methods and cannot adapt to the system performance loss caused by high-intensity attacks and the delay in defense strategy adjustment caused by highly concealed attacks. In the face of intelligent cyber attacks, the serious consequences of human decision-making errors by defenders are unbearable. The current cyberspace game has developed to the stage of intelligent machine vs. machine confrontation. Therefore, future network attacks and defenses urgently need fast, automatic and autonomous decision-making, better defense strategies and intelligent network defense methods, extending the human brain through intelligent technology, and fighting against intelligent attacks based on intelligent game technology, thereby improving defense decision-making capabilities and adapting to high-speed, complex and changeable network attack and defense confrontation environments.

[0003] From the perspective of existing technologies, the method of implementing cyberspace attack and defense modeling based on game theory and making decisions by solving game equilibrium is relatively close to this patent. However, the cyberspace game model constructed by the existing game theory is subject to the traditional game equilibrium solution method. The game participants are "people" and the intelligent agents supported by intelligent technology are not considered to participate in the game, and their intelligence level is not high; the rational assumptions of existing game participants are too strong, which is far from the real cyberspace attack and defense game; the utility function design of the existing cyberspace game is relatively simple and cannot reflect the impact of the strategy selection of participants in the real cyberspace game process.

[0004] The following is a general process of cyberspace attack and defense game based on game theory, which is the most common cyberspace attack and defense game method currently used to implement defender strategy selection. Figure 1 The establishment of the network attack and defense game model is mainly based on game theory. Both participants in the game are people. First, according to the network structure and topology, the network attack and defense game model is constructed, and then the interaction process of the attack and defense strategies is analyzed. Finally, the profit values ​​of both parties are obtained by solving the game equilibrium, so as to obtain the optimal defense strategy of the defender.

[0005] Current cyber attacks are becoming more and more automated and intelligent, while the existing cyberspace game model has “human” game participants, and does not consider the participation of intelligent entities supported by intelligent technology. The current defense measures are not intelligent enough to achieve intelligent network security defense decisions. The existing game theory constructs the cyberspace game model with too strong rational assumptions about the game participants, which is quite different from the real cyberspace attack and defense game. The utility function design of the existing game model is relatively simple, and cannot reflect the impact of the strategy choices of the participants in the real cyberspace game process. Summary of the invention

[0006] In order to meet the intelligent needs of cyberspace game confrontation, the present invention proposes a method and system for intelligent game decision-making in cyberspace. In the face of automated and intelligent network attacks initiated by intelligent agents, the Deep Reinforcement Learning (DRL) paradigm is combined with game theory to map the attack and defense game in cyberspace from the traditional "people-to-people game" to the intelligent "machine-to-machine game". The present invention can be used to solve the problems of constructing and solving the intelligent game decision-making model in cyberspace, and then solve the problem of how the defensive intelligent agent can make automated and intelligent defense decisions when facing automated and intelligent network penetration attacks.

[0007] The present invention proposes a cyberspace intelligent game decision-making method, comprising:

[0008] Set up the game space of the cyberspace intelligent game decision model in which the attacking agent and the defending agent implement automated attack and automated defense, and initialize the number of game rounds and the number of game steps per round;

[0009] A mapping relationship between the agent reinforcement learning model and the cyberspace intelligent game decision model is established. Based on the mapping relationship, the agent reinforcement learning model and the cyberspace intelligent game decision model can interact with each other to initialize the attack agent and the defense agent respectively. Then, the cyberspace intelligent game is carried out. In one round of the game, the attack agent and the defense agent take actions in each round of the game according to their own strategies. The attack agent and the defense agent play multiple games until the number of game steps reaches a set threshold.

[0010] The attacking agent and the defending agent play multiple rounds of games. When the game reaches the maximum number of rounds and the maximum number of game steps in each round, the cyberspace intelligent game terminates, calculates the benefits of the attacking agent and the defending agent after taking actions in the current game round, and determines the winner based on the game benefits and game rules.

[0011] Furthermore, the game space of the cyberspace intelligent game decision model for setting the attack agent and the defense agent to implement automated attack and automated defense specifically includes:

[0012] The game space of the cyberspace intelligent game decision model is represented by a 6-tuple:

[0013] <N, S, τ, Π, A, R> (1)

[0014] Where N represents the set of attacking agents and defending agents participating in the intelligent game in cyberspace.

[0015] N={att1,att2,…,att i}∪{def1,def2,…,def j} (2)

[0016] Among them, att i represents the i-th attacking agent, def j represents the j-th defending agent, i and j are natural numbers, and the participants include at least one attacking agent and one defending agent;

[0017] S represents the state space set of cyberspace intelligent games, which is represented by a graph model:

[0018] S=<V,E,W> (3)

[0019] Among them, V represents the node set in the state space, E represents the link set in the state space, and W represents the connection relationship matrix between nodes in the state space;

[0020] τ represents the state transition probability of the cyberspace intelligent game, which means that the attacking agent takes action a in the current state s. 0 , the probability of the defensive agent transitioning to the next state s' after taking action ad;

[0021] Π represents the strategy set of the participants in the cyberspace intelligent game, which includes the attack agent strategy set and the defense agent strategy set, expressed as:

[0022]

[0023] in, Represents the strategy set of the i-th attacking agent, which contains n specific attack strategies Represents the strategy set of the jth defense agent, which contains m specific defense strategies m, n are natural numbers;

[0024] A represents the action set of the participants in the cyberspace intelligent game, including the attacking agent action set and the defending agent action set, which can be expressed as:

[0025]

[0026] in, Represents the action set of the i-th attacking agent, which contains n specific attack actions Represents the action set of the jth defense agent, including m specific defense actions

[0027] R represents the benefits of network intelligent game participants, including the benefits of attacking agents and defending agents, expressed as:

[0028]

[0029] Among them, R i att represents the profit of the ith attacking agent, R j def represents the payoff of the j-th defending agent.

[0030] Furthermore, the mapping relationship between the agent reinforcement learning model and the cyberspace game model includes:

[0031] Construct the attacking agent and the defending agent models respectively, and initialize the environment, state space, strategy set, action space and reward function of the attacking agent and the defending agent respectively;

[0032] Environment: the game space of the cyberspace intelligent game decision-making model;

[0033] State space: The state space of both the attacking agent and the defending agent contains the action space of the agent and the observed behavior of the opponent agent;

[0034] The strategy set includes: temporal strategy, which means that the agent makes a decision at a fixed time or random time; event strategy, which means that the agent makes a decision after observing the opponent's decision; mixed strategy, which means that the agent combines temporal strategy and event strategy to formulate a mixed strategy;

[0035] The action space includes the action set of the attacking agent and the action set of the defending agent: the action set of the attacking agent includes obtaining credentials to establish a connection, exploiting local vulnerabilities, exploiting remote vulnerabilities, and performing feint attacks and deception actions; the action set of the defending agent includes opening a firewall, IP hopping, port hopping, server redundancy, and setting up a honeypot;

[0036] The reward function is the feedback given by the environment after the attacking agent or the defending agent takes an action. The result of the reward function has positive and negative values. Positive values ​​represent reward returns, and negative values ​​represent penalties. They are used to evaluate the effectiveness of the current action.

[0037] Furthermore, the cyberspace intelligent game process is specifically as follows: the attacking and defending agents use the DQN algorithm and the opponent modeling to perceive the opponent's behavior, and store the observed opponent action sequence into the knowledge base k to form the attacking agent strategy set π o and the defensive agent strategy set π d ; In a round of cyberspace attack and defense game, first, the attacking agent launches an automated attack according to the strategy set; then, the defending agent perceives the behavior of the attacking agent and analyzes the attacking agent strategy set π o , based on the deep reinforcement learning algorithm, makes a defense decision and selects the corresponding action from its action set; next, the attacking agent perceives the behavior of the defending agent and analyzes the defense agent strategy set π d , make attack decisions based on the deep reinforcement learning algorithm and select corresponding actions from its action set; repeat the game many times until the number of game steps in this round reaches the threshold.

[0038] Furthermore, the specific method for determining the winner based on the rules and results of the game includes: calculating the benefits of the attacking agent and the defending agent after taking actions in the current game round based on a pre-set reward function; the benefit of the attacking agent or the defending agent is equal to the cumulative return value minus the cumulative cost; when the maximum number of game rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game is terminated and the party with the higher cumulative benefit wins.

[0039] The present invention also proposes a cyberspace intelligent game decision-making system, comprising:

[0040] An initialization module sets the game space of the cyberspace intelligent game decision model for the attacking agent and the defending agent to implement automated attack and automated defense, and initializes the number of game rounds and the number of game steps per round;

[0041] The mapping and game module establishes a mapping relationship between the agent reinforcement learning model and the cyberspace intelligent game decision model. Based on the mapping relationship, the agent reinforcement learning model and the cyberspace intelligent game decision model can interact with each other and initialize the attack agent and the defense agent respectively. Then, the cyberspace intelligent game is carried out. In one round of the game, the attack agent and the defense agent take actions in each round of the game according to their own strategies. The attack agent and the defense agent play multiple games until the number of game steps reaches the set threshold.

[0042] In the decision-making module, the attacking agent and the defending agent play multiple rounds of games. When the game reaches the maximum number of rounds and the maximum number of game steps per round, the cyberspace intelligent game terminates, calculates the benefits of the attacking agent and the defending agent after taking actions in the current game round, and determines the winner based on the game benefits and game rules.

[0043] Furthermore, the game space of the cyberspace intelligent game decision model in which the attack agent and the defense agent implement automated attack and automated defense is set by the initialization module specifically includes:

[0044] The game space of the cyberspace intelligent game decision model is represented by a 6-tuple:

[0045] <N, S, τ, Π, A, R> (1)

[0046] Where N represents the set of attacking agents and defending agents participating in the intelligent game in cyberspace.

[0047] N={att1,att2,…,att i}∪{def1,def2,…,def j} (2)

[0048] Among them, att i represents the i-th attacking agent, def j represents the j-th defending agent, i and j are natural numbers, and the participants include at least one attacking agent and one defending agent;

[0049] S represents the state space set of cyberspace intelligent games, which is represented by a graph model:

[0050] S=<V,E,W> (3)

[0051] Among them, V represents the node set in the state space, E represents the link set in the state space, and W represents the connection relationship matrix between nodes in the state space;

[0052] τ represents the state transition probability of the cyberspace intelligent game, which means that the attacking agent takes action a in the current state s. o , the defensive agent takes action a d After that, the probability of transferring to the next state s';

[0053] Π represents the strategy set of the participants in the cyberspace intelligent game, which includes the attack agent strategy set and the defense agent strategy set, expressed as:

[0054]

[0055] in, Represents the strategy set of the i-th attacking agent, which contains n specific attack strategies Represents the strategy set of the jth defense agent, which contains m specific defense strategies m, n are natural numbers;

[0056] A represents the action set of the participants in the cyberspace intelligent game, including the attacking agent action set and the defending agent action set, which can be expressed as:

[0057]

[0058] in, Represents the action set of the i-th attacking agent, which contains n specific attack actions Represents the action set of the jth defense agent, including m specific defense actions

[0059] R represents the benefits of network intelligent game participants, including the benefits of attacking agents and defending agents, expressed as:

[0060]

[0061] Among them, R i att represents the profit of the ith attacking agent, R j def represents the payoff of the j-th defending agent.

[0062] Furthermore, the mapping and game module establishes a mapping relationship between the agent reinforcement learning model and the cyberspace game model, including:

[0063] Construct the attacking agent and the defending agent models respectively, and initialize the environment, state space, strategy set, action space and reward function of the attacking agent and the defending agent respectively;

[0064] Environment: the game space of the cyberspace intelligent game decision-making model;

[0065] State space: The state space of both the attacking agent and the defending agent contains the action space of the agent and the observed behavior of the opponent agent;

[0066] The strategy set includes: temporal strategy, which means that the agent makes a decision at a fixed time or random time; event strategy, which means that the agent makes a decision after observing the opponent's decision; mixed strategy, which means that the agent combines temporal strategy and event strategy to formulate a mixed strategy;

[0067] The action space includes the action set of the attacking agent and the action set of the defending agent: the action set of the attacking agent includes obtaining credentials to establish a connection, exploiting local vulnerabilities, exploiting remote vulnerabilities, and performing feint attacks and deception actions; the action set of the defending agent includes opening a firewall, IP hopping, port hopping, server redundancy, and setting up a honeypot;

[0068] The reward function is the feedback given by the environment after the attacking agent or the defending agent takes an action. The result of the reward function has positive and negative values. Positive values ​​represent reward returns, and negative values ​​represent penalties. They are used to evaluate the effectiveness of the current action.

[0069] Furthermore, the mapping and game module executes the cyberspace intelligent game process specifically as follows: the attacking and defending agents use the DQN algorithm and opponent modeling to perceive the opponent's behavior, store the observed opponent action sequence into the knowledge base k, and form the attacking agent strategy set π respectively. o and the defensive agent strategy set π d ; In a round of cyberspace attack and defense game, first, the attacking agent launches an automated attack according to the strategy set; then, the defending agent perceives the behavior of the attacking agent and analyzes the attacking agent strategy set π o , based on the deep reinforcement learning algorithm, makes a defense decision and selects the corresponding action from its action set; next, the attacking agent perceives the behavior of the defending agent and analyzes the defense agent strategy set π d , make attack decisions based on the deep reinforcement learning algorithm and select corresponding actions from its action set; repeat the game many times until the number of game steps in this round reaches the threshold.

[0070] Furthermore, the specific method by which the decision module determines the winner according to the rules and results of the game includes:

[0071] According to the pre-set reward function, the benefits of the attacking agent and the defending agent after taking actions in the current game round are calculated; the benefit of the attacking agent or the defending agent is equal to the cumulative return value minus the cumulative cost; when the maximum number of game rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game terminates and the party with the higher cumulative benefit wins.

[0072] The present invention implements a cyberspace intelligent game decision-making method and system for automated and intelligent attacks in cyberspace, aiming to solve the problem of building and solving the cyberspace intelligent game decision-making model, and then solve the problem of how the defense agent can make automated and intelligent defense decisions when facing automated and intelligent network penetration attacks. The present invention applies intelligent game technology to the field of cyberspace, combines game theory and deep reinforcement learning paradigm to construct a cyberspace intelligent game decision-making model, and achieves the following effects:

[0073] (1) The present invention enables the attack and defense game in cyberspace to be mapped from the traditional "game between people" to the intelligent "game between machines", thereby improving the intelligence level of cyberspace defense.

[0074] (2) The present invention improves the efficiency of solving the equilibrium of cyberspace attack and defense games through a deep reinforcement learning algorithm, providing a new intelligent defense decision-making solution for countering automated and intelligent network penetration attacks.

[0075] (3) By using the present invention, the action space of the agents is expanded by modeling non-completely rational attack and defense agents in cyberspace, including dynamic defense actions such as feints by attacking agents and mutations by defending agents, which is closer to the real network game scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flowchart of cyberspace attack and defense game based on game theory;

[0077] Figure 2 It is the overall technical scheme diagram of the present invention;

[0078] Figure 3 It is a flow chart of the decision-making model of intelligent game in cyberspace;

[0079] Figure 4 Flowchart of decision-making model for competing for sensitive resources in cyberspace. DETAILED DESCRIPTION

[0080] The present invention provides a cyberspace intelligent game decision-making method and system. The specific implementation of the present invention is described below in conjunction with the accompanying drawings.

[0081] The basic concepts involved in the present invention are as follows:

[0082] Reinforcement Learning (RL): It is one of the paradigms and methodologies of machine learning, which is mainly used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals by learning strategies during its interaction with the environment.

[0083] Deep Reinforcement Learning (DRL): It is the product of the combination of deep learning and reinforcement learning. It integrates the powerful understanding ability of deep learning in perceptual problems such as vision and the decision-making ability of reinforcement learning to achieve end-to-end learning.

[0084] Markov Decision Process (MDP): It is a mathematical model of sequential decision making, which is mainly used to simulate the random strategies and rewards that can be achieved by intelligent agents in an environment where the system state has Markov properties.

[0085] Agent: refers to a computing entity trained by reinforcement learning that can continuously and autonomously make automated and intelligent decisions in the corresponding network information system environment.

[0086] Adversary modeling: refers to using the interaction information between agents to model the opponent's agent behavior, then inferring and predicting the opponent's agent's behavior, and formulating the strategy of one's own agent accordingly.

[0087] Intelligent attack: refers to the use of deep reinforcement learning (DRL) and opponent modeling methods to describe and train agents to implement automated and intelligent network attacks on behalf of attackers.

[0088] Intelligent defense: refers to the use of deep reinforcement learning (DRL) and adversary modeling methods to describe and train agents to implement automated and intelligent network security defense on behalf of defenders.

[0089] Intelligent network attack and defense game: In a highly confrontational, uncertain and highly dynamic network attack and defense environment, through interactive learning between attacking agents, defending agents and the environment, we study how the defending agents use defense strategies to defeat or weaken intelligent attacks.

[0090] FlipIt game: FlipIt is an infinitely repeated game in which the same one-time stage game is repeated in discrete time periods. The FlipIt game consists of defenders, attackers, and public resources to simulate APT attack scenarios. Over time, the network attackers and defenders compete for control of sensitive resources (e.g., passwords, encryption keys, computer systems, or networks). At each stage of the game, the participants decide what actions to take based on their respective strategies. At any time, the participants can control the public resources by preempting (flipping) at a certain action cost.

[0091] DQN (Deep Q-Network): The deep Q network algorithm replaces the Q-value table in the Q learning algorithm with a neural network, thereby transforming the original action-value function convergence problem into a neural network function fitting problem. It is a pioneering work in the field of deep reinforcement learning.

[0092] Combination Figure 2The overall technical solution of the present invention is described. Based on the cyberspace intelligent game decision-making method and system of the present invention, a cyberspace intelligent game decision-making model can be obtained.

[0093] The cyberspace intelligent game decision-making model mainly consists of three parts: the construction of the cyberspace intelligent game decision-making model, the mapping of the cyberspace intelligent game decision-making model, and the deep reinforcement learning algorithm to solve the game equilibrium solution.

[0094] (1) Construction of a cyberspace intelligent game decision-making model

[0095] Constructing a cyberspace intelligent game decision-making model mainly includes: building a cyberspace intelligent game space, attack agent modeling, and defense agent modeling.

[0096] ① Intelligent game space in cyberspace

[0097] The cyberspace intelligent game space is used to provide a network environment for intelligent algorithm game confrontation, matching the network topology environment with the elements required by the deep reinforcement learning intelligent algorithm, including state space, action space, state transition relationship and reward function. The state space is the network topology environment, including hosts, vulnerabilities, port information, etc.; the action space is all the optional actions of the attacking agent and the defending agent; the state transition relationship is the probability of the current state transferring to the next state; the reward function is the feedback obtained after the attacking agent and the defending agent make an action and interact with the environment and the opponent agent, which can be positive or negative.

[0098] ② Attacking Agent Modeling

[0099] The attack agent modeling includes state space, strategy set, action space and reward function. The state space of the attack agent consists of two parts: one is the network environment detected by the attack agent after interacting with the environment, and the other is the perceived behavior of the opponent agent; the strategy set of the attack agent guides the attack agent to take actions, mainly including time strategy, event strategy and hybrid strategy; the action space of the attack agent is the set of optional actions of the attack agent, including obtaining credentials to establish connections, local vulnerability exploitation and remote vulnerability exploitation; the reward function of the attack agent is the feedback given by the environment after the attack agent takes action, which is used to evaluate the effect of the current action.

[0100] ③Defense agent modeling

[0101] The modeling of defense agents includes state space, strategy set, action space and reward function. The state space of the defense agent consists of two parts, one is the known network environment, and the other is the perceived behavior of the opponent agent; the strategy set of the defense agent guides the defense agent to take actions, mainly including time strategy, event strategy and hybrid strategy; the action space of the defense agent is the set of optional actions of the defense agent, including opening the firewall, IP hopping, port hopping, server redundancy and setting up honeypots; the reward function of the defense agent is the feedback given by the environment after the defense agent takes action, which is used to evaluate the effect of the current action.

[0102] (2) Mapping of the Cyberspace Intelligent Game Decision Model

[0103] The mapping of the cyberspace intelligent game decision model mainly maps the intelligent game space to intelligent algorithms such as deep reinforcement learning. The cyberspace game space is mapped to the environment of the intelligent agent in the deep reinforcement learning algorithm, the participants in the cyberspace game space are mapped to the attacking agent and the defending agent in the deep reinforcement learning algorithm, the strategy in the cyberspace game space is mapped to the strategy of the attacking agent and the defending agent in the deep reinforcement learning algorithm, the action in the cyberspace game space is mapped to the optional action of the attacking agent and the defending agent, and the benefit in the cyberspace game space is mapped to the reward function in the deep reinforcement learning algorithm. By mapping the cyberspace game space to the deep reinforcement learning algorithm, the deep reinforcement learning algorithm can be used to solve the equilibrium solution of the cyberspace intelligent game decision model.

[0104] (3) Deep reinforcement learning algorithm to solve the game equilibrium solution

[0105] The attacking agent and the defending agent play the game repeatedly. In each round of the game, the attacking agent and the defending agent make actions according to their own strategies, and reach a state of game equilibrium through interaction with the game environment and the opponent agent. According to the pre-set reward function, the profit value of the attacking agent and the defending agent after taking actions in the current round of the game can be calculated. The profit value of the attacking agent and the defending agent is equal to the cumulative return value minus the cumulative cost. When the maximum number of rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game ends, and the party with the higher cumulative profit value wins.

[0106] like Figure 3 As shown, the implementation method of the cyberspace intelligent game decision-making model includes three steps.

[0107] Step 101: Construction of network attack and defense intelligent game decision model

[0108] Provide a game space for attacking and defending agents to implement automated attacks and defenses. Define network attack and defense scenarios, such as competing for sensitive resources on a server or finding attack paths from the network topology. Initialize the number of training rounds eps and the number of game steps per round steps, the resource seizure benefit value b, and the resource seizure cost value c.

[0109] The game space of the cyberspace intelligent game decision model can be expressed as a 6-tuple:

[0110] <N, S, τ, Π, A, R> (1)

[0111] 1) N represents the set of participants in the cyberspace intelligent game. The participants in the cyberspace intelligent game are attacking agents and defending agents, which can be expressed as:

[0112] N={att1,att2,…,att i}∪{def1,def2,…,def j} (2)

[0113] Among them, att i represents the i-th attacking agent, def j represents the j-th defending agent, i and j are natural numbers, and the participants include at least 1 attacking agent and 1 defending agent.

[0114] 2) S represents the state space set of cyberspace intelligent games, which is the environment for implementing network attack and defense confrontation and can be represented by a graph model:

[0115] S=<V,E,W>(3)

[0116] Among them, V represents the node set in the state space, E represents the link set in the state space, and W represents the connection relationship matrix between nodes in the state space.

[0117] 3)τ represents the state transition probability of the cyberspace intelligent game, which can be expressed as:

[0118] τ{s,a d , a o , s′}=P(s′|s, a d , a o ) (4)

[0119] Indicates that the attacking agent takes action a in the current state s o , the defensive agent takes action a d After that, the probability of transferring to the next state s'.

[0120] 4) Π represents the strategy set of the participants in the cyberspace intelligent game, which is divided into the attack agent strategy set and the defense agent strategy set, which can be expressed as:

[0121]

[0122] in, Represents the strategy set of the i-th attacking agent, which contains n specific attack strategies Represents the strategy set of the jth defense agent, which contains m specific defense strategies

[0123] 5) A represents the action set of the participants in the cyberspace intelligent game, including the action set of the attacking agent and the action set of the defending agent, which can be expressed as:

[0124]

[0125] in, Represents the action set of the i-th attacking agent, which contains n specific attack actions Represents the action set of the jth defense agent, including m specific defense actions

[0126] 6) R represents the benefits of network intelligent game participants, including the benefits of attacking agents and defending agents, which can be expressed as:

[0127]

[0128] Among them, R i att represents the profit of the ith attacking agent, R j def represents the payoff of the j-th defending agent.

[0129] Step 102: Cyberspace Intelligent Game Decision Model Mapping

[0130] The mapping relationship between cyberspace games and agent reinforcement learning is as follows:

[0131] Table 1 Mapping relationship between reinforcement learning and game theory

[0132]

[0133] Based on the above mapping relationship, the agent reinforcement learning model and the cyberspace intelligent game decision model can interact. In cyberspace, the attacking agent and the defending agent are opponents of each other, and the attacking agent and the defending agent models need to be constructed separately. The environment, state space, strategy set, action space and reward function of the attacking agent and the defending agent are initialized separately. In one case, both the attacking agent and the defending agent use the DQN algorithm to interact with the environment, and each has an independent state space, strategy set, action space and reward function, and its reward function is affected by the environment and the opponent agent.

[0134] 1) Agent environment: It is the game space of the cyberspace intelligent game decision-making model.

[0135] 2) State space of the agent: The state space of both the attacking agent and the defending agent contains the action space of the agent and the observed behavior of the opponent agent.

[0136] 3) Agent strategy set:

[0137] ① Temporal strategy: Make decisions at fixed or random times.

[0138] ② Event strategy: Make a decision after observing the opponent's decision.

[0139] ③ Mixed strategy: A mixed strategy is formulated by combining time-based strategy and event-based strategy.

[0140] The agent can choose the above strategies to take actions. At the same time, both the attacking agent and the defending agent are boundedly rational and may deceive the opponent agent.

[0141] 4) Action space of the agent: The action space of the attacking agent is the set of optional actions of the attacking agent, including obtaining credentials to establish a connection, local vulnerability exploitation and remote vulnerability exploitation, etc. The attacking agent may also launch deceptive actions such as feint attacks to consume the resources of the defending agent; the action space of the defending agent is the set of optional actions of the defending agent, including opening a firewall, IP hopping, port hopping, server redundancy and setting up a honeypot, etc.

[0142] 5) Reward function of the agent: It is the feedback given by the environment after the agent takes action. The result of the reward function has positive and negative values. Positive values ​​represent reward returns, and negative values ​​represent penalties. It is used to evaluate the effect of the current action. The reward functions of the attacking agent and the defending agent are independent of each other and will be affected by the actions selected by the opponent agent. The design of the reward function in reinforcement learning will have a great impact on the results of intelligent games. Therefore, in the process of the game, expert experience is introduced to correct and optimize the reward function of the agent.

[0143] In a round of cyberspace attack and defense game, first, the attacking agent launches an automated attack according to the strategy set. Then, the defending agent uses the opponent modeling to perceive the attacking agent's behavior, stores the observed opponent action sequence in the knowledge base k, and analyzes the opponent's strategy π. o , combining traditional and dynamic defense methods, and making intelligent defense decisions based on intelligent algorithms such as deep reinforcement learning. Next, the attacking agent uses the opponent modeling to perceive the behavior of the defending agent and analyze the opponent's strategy π d , adjust their own strategies based on intelligent algorithms such as deep reinforcement learning, and make intelligent attack decisions. This game is repeated many times until the number of game steps in this round reaches the threshold.

[0144] Specifically, in a round of cyberspace attack and defense game, the attacking agent and the defending agent observe the state space S through the DQN algorithm. t , constantly interacting with the environment and the opponent agent. The historical interaction data is stored in the experience buffer buffer, from which batch_size sample data is randomly taken to train the agent, and the mean square error loss function of the predicted value of the current Q network and the expected value of the target Q network is minimized by the gradient descent method, and the parameters of the current Q network are updated. The relevant parameters of the current Q network are assigned to the target Q network at regular intervals. The attacking and defending agents use the DQN algorithm and opponent modeling to perceive the opponent's behavior, store the observed opponent action sequence in the knowledge base k, and analyze the opponent's strategy. The defensive agent combines traditional and dynamic defense methods to make intelligent defense decisions π from the defense strategy set d , and then select action a according to the strategy d The attacking agent perceives the strategy of the defending agent, adjusts its own strategy based on the DQN algorithm, and makes intelligent attack decisions from the attack strategy set π o , select action a according to the strategy o The game is repeated many times until the number of steps in this round reaches the threshold. After both agents make an action, they will get feedback from the environment, which is the agent's reward function R t .

[0145] Step 103: Solving the cyberspace intelligent game decision model

[0146] The attacking agent and the defending agent play the game repeatedly. In each round of the game, the attacking and defending agents make actions according to their own strategies, and reach a state of game equilibrium through interaction with the game environment and the opponent agent. According to the pre-set reward function, the profit value of the attacking agent and the defending agent after taking actions in the current round of the game can be calculated. The profit value of the attacking agent and the defending agent is equal to the cumulative return value minus the cumulative cost. When the maximum number of rounds and the maximum number of steps per round are reached, the cyberspace intelligent game ends, and the party with the higher cumulative profit value wins.

[0147] When calculating the benefits of the attacking and defending agents, the Q value is the reward value of the current step and the value of the next state.

[0148] The calculation formula of Q value is as follows:

[0149]

[0150]

[0151] Among them, Q def is the Q value of the defending agent, Q att is the Q value of the attacking agent, γ is the discount factor, γ∈[0,1], s is the current state, and s' is the next state.

[0152] Minimize the predicted value Q of the current Q network and the expected value Q of the target Q network by gradient descent E The mean square error loss function is:

[0153]

[0154]

[0155] Calculate the Q value of the attacking agent and the defending agent in each game step to get the corresponding cumulative reward value. The profit value of the attacking agent and the defending agent is equal to the cumulative reward value minus the cumulative cost. When the maximum number of rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game ends, and the party with the higher cumulative profit value wins.

[0156] Example

[0157] Combination Figure 4 As shown in the figure, a specific application example of this model in competing for sensitive resources in cyberspace is explained in detail.

[0158] Step S1: Construction of intelligent game decision-making model for competing for sensitive resources in cyberspace. Provide a game space for attacking agents and defending agents to implement automated attacks and automated defenses. Define the cyberspace attack and defense scenario as the two parties competing for sensitive resources on the server. Sensitive resources include core server data, host control, passwords or keys, etc. Assume that there is one attacking agent and one defending agent. The goal of the attacking agent is to seize sensitive resources in a covert manner, and the goal of the defending agent is to identify the opponent's agent strategy and protect sensitive resources.

[0159] The game space of the intelligent game decision-making model for competing for sensitive resources in cyberspace can be expressed as a 6-tuple:<N,S,τ,Π,A,R> .

[0160] 1) N represents the set of participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace. There is an attacking agent and a defending agent, which can be expressed as:

[0161] N={att1}∪{def1} (12)

[0162] Among them, att1 represents the attacking agent that competes for sensitive resources, and def1 represents the defending agent that guards sensitive resources.

[0163] 2) S represents the state space set of the intelligent game decision-making model for competing for sensitive resources in cyberspace, which is the Fliplt game space in cyberspace and can be expressed as:

[0164]

[0165]

[0166] Among them, V is a single node, representing a sensitive resource node in the cyberspace, such as servers, routers, administrator account passwords, etc. containing sensitive data. Due to single-node control, the link set E is empty, and the node connection relationship matrix W is empty.

[0167] For the defending agent, V d TFull V indicates that the defense agent can observe all the time series of its own seizure of sensitive resources. o TLM It means that the defending agent can only observe the time series of the opponent agent's last seizure of sensitive resources.

[0168] 3)τ represents the state transition probability of the intelligent game decision-making model for competing for sensitive resources in cyberspace, which can be expressed as:

[0169] τ{s,a d , a o , s′}=P(s′|s, ad , a o ) (15)

[0170] Indicates that the attacking agent takes action a in the current state s o , the defensive agent takes action a d After that, the probability of transferring to the next state s'.

[0171] 4) Π represents the strategy set of participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace, which is divided into the attack agent strategy set and the defense agent strategy set. Represents the strategy set of the attacking agent, including 3 specific attack strategies; Represents the strategy set of the defense agent, which includes 3 specific defense strategies. The three strategies of the attack agent and the defense agent are as follows:

[0172] ① Temporal strategy. a. Periodic strategy. Preempt sensitive resources at a fixed time step of δ = 20. b. Exponential strategy. The time interval between two preemptions of sensitive resources follows an exponential distribution with a rate of λ = 0.02.

[0173] ② Event strategy: After observing the opponent’s intelligent agent seizing resources, it starts to seize sensitive resources.

[0174] ③ Hybrid strategy: Combining time-based strategy and event-based strategy to formulate a hybrid strategy to seize sensitive resources.

[0175] The attacking agent and the defending agent can choose one of the three specific strategy sets to take action.

[0176] 5) A represents the action set of the participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace, including the action set of the attacking agent and the action set of the defending agent. Since the game participants are all boundedly rational, the attacking agent may launch a feint to deceptive action, consuming the resources of the defending agent; in order to respond to the attack, the defending agent may initiate a mutation action to invalidate the information collected by the attacking agent in the early stage, which can be expressed as:

[0177] A att ={void, flip, feint} (16)

[0178] A def ={void, flip, mutation} (17)

[0179] Among them, A att A represents the action set of the attacking agent, which includes three specific attack actions: resource grabbing flip, non-resource grabbing void, and feint attack feint; defRepresents the action set of the defense agent, including three specific defense actions: resource seizure flip, non-resource seizure void, and mutation.

[0180] 6) R represents the benefits of participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace, which can be expressed as:

[0181]

[0182]

[0183] Among them, R att represents the profit of attacking the agent, R def represents the payoff of the defending agent.

[0184] Step S2: Mapping of intelligent game decision-making model for competing for sensitive resources in cyberspace. Map the Fliplt game scenario of competing for sensitive resources in cyberspace to the agent deep reinforcement learning algorithm, and set the environment, state space, strategy set, action space and reward function of the attacking agent and the defending agent. Set the number of training rounds eps = 10000 and the number of game steps per round steps = 200.

[0185] The agent's environment is the game space of the intelligent game decision-making model for competing for sensitive resources in cyberspace, which can be expressed as a 6-tuple:<N,S,τ,Π,A,R> .

[0186] The state space of the agent: It is the state space set S of the intelligent game decision-making model for competing for sensitive resources in cyberspace, including the time series of all resource seizures by the agent on one side and the time series of the last resource seizure by the agent on the other side.

[0187] Agent strategy set: It is the strategy set Π of the participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace, including time strategies, event strategies and comprehensive strategies.

[0188] Agent action space: It is the action set A of the participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace, including two actions: seizing resources and not seizing resources.

[0189] Agent reward function: It is the income R of the participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace. Let the income value of seizing resources be 2, the cost value of seizing resources be 5, the cost value of the feint action of the attacking agent be 3, the cost value of the mutation action of the defending agent be 7, and the income value of the mutation action of the defending agent be 1. Cumulative return value = cumulative return value - cumulative cost value, and the party with the higher cumulative return value wins.

[0190] The intelligent agent constantly interacts with the environment and the opponent intelligent agent. By observing the state space and using opponent modeling to perceive the rules of the opponent's resource-seizing timing sequence, it then makes decisions from the strategy set through a deep reinforcement learning algorithm, i.e., the timing sequence for seizing resources, and then selects actions based on the strategy. After the intelligent agent takes action, it will receive feedback from the environment, i.e., the agent's reward function.

[0191] In one embodiment, the agent observes the state space S through the DQN algorithm t , continuously interact with the environment and the opponent agent, store the historical interaction data in the experience buffer buffer = 2000, randomly take batch_size = 32 sample data from it to train the agent, and minimize the predicted value Q of the current Q network and the expected value Q of the target Q network by gradient descent method E The mean square error loss function is used to update the parameters of the current Q network, and the relevant parameters of the current Q network are assigned to the target Q network at regular intervals. The intelligent agent uses the opponent modeling to perceive the rules of the opponent's resource seizure timing sequence, and then makes a decision from the strategy set through the DQN algorithm. t , that is, the timing sequence of seizing resources, and then selecting action a according to the strategy t After the agent takes action, it will get feedback from the environment, which is the agent’s reward function R t .

[0192] Step S3: Solve the intelligent game decision model for competing for sensitive resources in cyberspace. The attacking agent and the defending agent continuously make automated and intelligent decisions, and play games multiple times to reach the game equilibrium point. After reaching the game equilibrium point, the reward values ​​of both parties gradually stabilize, fluctuating up and down within a small range. The profit values ​​of both parties are calculated based on the attack and defense interactions of each game round. The profit value = reward value - cost value. After reaching the maximum number of game rounds and the number of game steps per round, the game ends and the party with the higher cumulative profit value wins.

[0193] In one embodiment, the benefit value of each of the two intelligent agents is calculated by the DQN algorithm, where benefit value = reward value - cost value. The calculation formula of Q value is as follows:

[0194]

[0195]

[0196] Among them, Q def is the Q value of the defensive agent guarding sensitive resources, Q att is the Q value of the attacking agent competing for sensitive resources, the discount factor γ = 0.99, s is the current state, and s' is the next state.

[0197] Minimize the predicted value Q of the current Q network and the expected value Q of the target Q network by gradient descent E The mean square error loss function is:

[0198]

[0199]

[0200] After reaching the maximum number of game rounds eps = 10000 and the number of game steps per round steps = 200, the game ends and the party with the higher cumulative profit value wins.

[0201] The present invention also proposes a cyberspace intelligent game decision-making system, comprising:

[0202] The initialization module sets the game space of the cyberspace intelligent game decision model in which the attacking agent and the defending agent implement automated attack and automated defense, and initializes the number of game rounds and the number of game steps per round.

[0203] The mapping and game module establishes a mapping relationship between the agent reinforcement learning model and the cyberspace intelligent game decision model. Based on the mapping relationship, the agent reinforcement learning model and the cyberspace intelligent game decision model can interact and initialize the attack agent and the defense agent respectively; then the cyberspace intelligent game is carried out. In one round of the game, the attack agent and the defense agent take actions in each game round according to their own strategies. The attack agent and the defense agent play multiple games until the number of game steps reaches the set threshold.

[0204] In the decision-making module, the attacking agent and the defending agent play multiple rounds of games. When the game reaches the maximum number of rounds and the maximum number of game steps per round, the cyberspace intelligent game terminates and the winner is determined based on the results and rules of the game.

[0205] The initialization module sets the game space of the cyberspace intelligent game decision model for the attack agent and the defense agent to implement automated attack and automated defense, which specifically includes:

[0206] The game space of the cyberspace intelligent game decision model is represented by a 6-tuple:

[0207] <N,S,τ,Π,A,R>(1)

[0208] Where N represents the set of attacking agents and defending agents participating in the intelligent game in cyberspace.

[0209] N={att1,att2,...,att i}∪{def1,def2,…,def j} (2)

[0210] Among them, att i represents the i-th attacking agent, def j represents the j-th defending agent, i and j are natural numbers, and the participants include at least 1 attacking agent and 1 defending agent.

[0211] S represents the state space set of cyberspace intelligent games, which is represented by a graph model:

[0212] S=<V,E,W> (3)

[0213] Among them, V represents the node set in the state space, E represents the link set in the state space, and W represents the connection relationship matrix between nodes in the state space.

[0214] τ represents the state transition probability of the cyberspace intelligent game, which means that the attacking agent takes action a in the current state s. 0 , the defensive agent takes action a d After that, the probability of transferring to the next state s'.

[0215] ∏ represents the strategy set of the participants in the cyberspace intelligent game, which includes the attack agent strategy set and the defense agent strategy set, expressed as:

[0216]

[0217] in, Represents the strategy set of the i-th attacking agent, which contains n specific attack strategies Represents the strategy set of the jth defense agent, which contains m specific defense strategies m and n are natural numbers.

[0218] A represents the action set of the participants in the cyberspace intelligent game, including the attacking agent action set and the defending agent action set, which can be expressed as:

[0219]

[0220] in, Represents the action set of the i-th attacking agent, which contains n specific attack actions Represents the action set of the jth defense agent, including m specific defense actions

[0221] R represents the benefits of network intelligent game participants, including the benefits of attacking agents and defending agents, expressed as:

[0222]

[0223] Among them, R i att represents the profit of the ith attacking agent, R j def represents the payoff of the j-th defending agent.

[0224] The mapping and game module establishes the mapping relationship between the agent reinforcement learning model and the cyberspace game model, including:

[0225] The attack agent and defense agent models are constructed separately, and the environment, state space, strategy set, action space and reward function of the attack agent and defense agent are initialized separately.

[0226] Environment: Game space of the cyberspace intelligent game decision-making model.

[0227] State space: The state space of both the attacking agent and the defending agent contains the action space of the agent and the observed behavior of the opponent agent.

[0228] The strategy set includes: temporal strategy, which means that the agent makes decision actions at fixed time or random time; event strategy, which means that the agent makes decision actions after observing the opponent's decision actions; mixed strategy, which means that the agent combines temporal strategy and event strategy to formulate a mixed strategy.

[0229] The action space includes the action set of the attacking agent and the action set of the defending agent: the action set of the attacking agent includes obtaining credentials to establish a connection, exploiting local vulnerabilities, exploiting remote vulnerabilities, and performing feint attacks; the action set of the defending agent includes opening a firewall, IP hopping, port hopping, server redundancy, and setting up a honeypot.

[0230] The reward function is the feedback given by the environment after the attacking agent or the defending agent takes an action. The result of the reward function has positive and negative values. Positive values ​​represent reward returns, and negative values ​​represent penalties. They are used to evaluate the effectiveness of the current action.

[0231] The mapping and game module executes the cyberspace intelligent game process specifically as follows:

[0232] The attacking and defending agents use the DQN algorithm and opponent modeling to perceive the opponent's behavior, and store the observed opponent action sequence into the knowledge base k to form the attacking agent strategy set π o and the defensive agent strategy set π d ; In a round of cyberspace attack and defense game, first, the attacking agent launches an automated attack according to the strategy set; then, the defending agent perceives the behavior of the attacking agent and analyzes the attacking agent strategy set π o, based on the deep reinforcement learning algorithm, makes a defense decision and selects the corresponding action from its action set; next, the attacking agent perceives the behavior of the defending agent and analyzes the defense agent strategy set π d , make attack decisions based on the deep reinforcement learning algorithm and select corresponding actions from its action set; repeat the game many times until the number of game steps in this round reaches the threshold.

[0233] The specific methods by which the decision module determines the winner according to the rules and results of the game include:

[0234] According to the pre-set reward function, the benefits of the attacking agent and the defending agent after taking actions in the current game round are calculated; the benefit of the attacking agent or the defending agent is equal to the cumulative return value minus the cumulative cost; when the maximum number of game rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game terminates and the party with the higher cumulative benefit wins.

Claims

1. A cyberspace intelligent game decision-making method, characterized in that: include: Set up the game space of the cyberspace intelligent game decision model in which the attacking agent and the defending agent implement automated attack and automated defense, and initialize the number of game rounds and the number of game steps per round; A mapping relationship between the agent reinforcement learning model and the cyberspace intelligent game decision model is established, and the mapping relationship includes: Construct the attacking agent and the defending agent models respectively, and initialize the environment, state space, strategy set, action space and reward function of the attacking agent and the defending agent respectively; Environment: the game space of the cyberspace intelligent game decision-making model; State space: The state space of both the attacking agent and the defending agent contains the action space of the agent and the observed behavior of the opponent agent; The strategy set includes: temporal strategy, which means that the agent makes a decision at a fixed time or random time; event strategy, which means that the agent makes a decision after observing the opponent's decision; mixed strategy, which means that the agent combines temporal strategy and event strategy to formulate a mixed strategy; The action space includes the action set of the attacking agent and the action set of the defending agent: the action set of the attacking agent includes obtaining credentials to establish a connection, exploiting local vulnerabilities, exploiting remote vulnerabilities, and performing feint attacks and deception actions; the action set of the defending agent includes opening a firewall, IP hopping, port hopping, server redundancy, and setting up a honeypot; The reward function is the feedback given by the environment after the attacking agent or the defending agent takes an action. The result of the reward function has positive and negative values. Positive values ​​represent reward returns, and negative values ​​represent penalties. It is used to evaluate the effect of the current action. Based on the mapping relationship, the agent reinforcement learning model and the cyberspace intelligent game decision model can interact to initialize the attacking agent and the defending agent respectively; The attacking agent and the defending agent play multiple rounds of games. When the game reaches the maximum number of rounds and the maximum number of game steps per round, the cyberspace intelligent game ends, and the gains of the attacking agent and the defending agent after taking actions in the current game round are calculated. The winner is determined based on the gains and game rules of the game. The cyberspace attack and defense scenario is defined as the competition between two parties for sensitive server resources, including core server data, host control, passwords or keys; The attack agent action set and the defense agent action set are expressed as: A att ={void,flip,feint} A def ={void,flip,mutation} A att A represents the action set of the attacking agent, including resource grabbing flip, non-resource grabbing void, and feint attack feint; def Represents the action set of the defense agent, including resource preemption flip, non-resource preemption void, and mutation; The benefits of participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace are expressed as: R att represents the profit of attacking the agent, R def represents the payoff of the defending agent.

2. The method according to claim 1, characterized in that The game space of the cyberspace intelligent game decision model for setting the attack agent and the defense agent to implement automated attack and automated defense specifically includes: The game space of the cyberspace intelligent game decision model is represented by a 6-tuple: <N,S,τ,∏,A,R> Where N represents the set of attacking agents and defending agents participating in the intelligent game in cyberspace. N={att1,att2,…,att i }∪{def1,def2,…,def j } Among them, att i represents the i-th attacking agent, def j represents the j-th defending agent, i and j are natural numbers, and the participants include at least one attacking agent and one defending agent; S represents the state space set of cyberspace intelligent games, which is represented by a graph model: S=<V,E,W> Among them, V represents the node set in the state space, E represents the link set in the state space, and W represents the connection relationship matrix between nodes in the state space; τ represents the state transition probability of the cyberspace intelligent game, which means that the attacking agent takes action a in the current state s. 0 , the defensive agent takes action a d After that, the probability of transferring to the next state s'; ∏ represents the strategy set of the participants in the cyberspace intelligent game, which includes the attack agent strategy set and the defense agent strategy set, expressed as: in, Represents the strategy set of the i-th attacking agent, which contains n specific attack strategies π i n ; Represents the strategy set of the jth defense agent, including m specific defense strategies π j m , m and n are natural numbers; A represents the action set of the participants in the cyberspace intelligent game, including the attacking agent action set and the defending agent action set, which can be expressed as: in, Represents the action set of the i-th attacking agent, which contains n specific attack actions a i n ; represents the action set of the jth defense agent, including m specific defense actions a j m ; R represents the benefits of network intelligent game participants, including the benefits of attacking agents and defending agents, expressed as: Among them, R i att represents the profit of the ith attacking agent, R j def represents the payoff of the j-th defending agent.

3. The method according to claim 2, characterized in that The cyberspace intelligent game process is specifically as follows: The attacking and defending agents use the DQN algorithm and opponent modeling to perceive the opponent's behavior, and store the observed opponent action sequence into the knowledge base k to form the attacking agent strategy set π o and the defensive agent strategy set π d ; In a round of cyberspace attack and defense game, first, the attacking agent launches an automated attack according to the strategy set; then, the defending agent perceives the behavior of the attacking agent and analyzes the attacking agent strategy set π o , based on the deep reinforcement learning algorithm, makes a defense decision and selects the corresponding action from its action set; next, the attacking agent perceives the behavior of the defending agent and analyzes the defense agent strategy set π d , make attack decisions based on the deep reinforcement learning algorithm and select corresponding actions from its action set; repeat the game many times until the number of game steps in this round reaches the threshold.

4. The method according to claim 3, characterized in that: The specific method for determining the winner according to the rules and results of the game includes: According to the pre-set reward function, the benefits of the attacking agent and the defending agent after taking actions in the current game round are calculated; the benefit of the attacking agent or the defending agent is equal to the cumulative return value minus the cumulative cost; when the maximum number of game rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game terminates and the party with the higher cumulative benefit wins.

5. A cyberspace intelligent game decision-making system, characterized in that: include: An initialization module sets the game space of the cyberspace intelligent game decision model for the attacking agent and the defending agent to implement automated attack and automated defense, and initializes the number of game rounds and the number of game steps per round; The mapping and game module establishes a mapping relationship between the agent reinforcement learning model and the cyberspace intelligent game decision model. The specific method of establishing the mapping relationship includes: Construct the attacking agent and the defending agent models respectively, and initialize the environment, state space, strategy set, action space and reward function of the attacking agent and the defending agent respectively; Environment: the game space of the cyberspace intelligent game decision-making model; State space: The state space of both the attacking agent and the defending agent contains the action space of the agent and the observed behavior of the opponent agent; The strategy set includes: temporal strategy, which means that the agent makes a decision at a fixed time or random time; event strategy, which means that the agent makes a decision after observing the opponent's decision; mixed strategy, which means that the agent combines temporal strategy and event strategy to formulate a mixed strategy; The action space includes the action set of the attacking agent and the action set of the defending agent: the action set of the attacking agent includes obtaining credentials to establish a connection, exploiting local vulnerabilities, exploiting remote vulnerabilities, and performing feint attacks and deception actions; the action set of the defending agent includes opening a firewall, IP hopping, port hopping, server redundancy, and setting up a honeypot; The reward function is the feedback given by the environment after the attacking agent or the defending agent takes an action. The result of the reward function has positive and negative values. Positive values ​​represent reward returns, and negative values ​​represent penalties. It is used to evaluate the effect of the current action. Based on the mapping relationship, the agent reinforcement learning model and the cyberspace intelligent game decision model can interact to initialize the attacking agent and the defending agent respectively; In the decision-making module, the attacking agent and the defending agent play multiple rounds of games. When the game reaches the maximum number of rounds and the maximum number of game steps per round, the cyberspace intelligent game ends, and the gains of the attacking agent and the defending agent after taking actions in the current game round are calculated. The winner is determined based on the gains and game rules of the game. The cyberspace attack and defense scenario is defined as the competition between two parties for sensitive server resources, including core server data, host control, passwords or keys; The attack agent action set and the defense agent action set are expressed as: A att ={void,flip,feint} A def ={void,flip,mutation} A att A represents the action set of the attacking agent, including resource grabbing flip, non-resource grabbing void, and feint attack feint; def Represents the action set of the defense agent, including resource preemption flip, non-resource preemption void, and mutation; The benefits of participants in the intelligent game decision-making model for competing for sensitive resources in cyberspace are expressed as: R att represents the profit of attacking the agent, R def represents the payoff of the defending agent.

6. The system according to claim 5, characterized in that The game space of the cyberspace intelligent game decision model in which the attack agent and the defense agent implement automated attack and automated defense is set by the initialization module specifically includes: The game space of the cyberspace intelligent game decision model is represented by a 6-tuple: <N,S,τ,Π,A,R> Where N represents the set of attacking agents and defending agents participating in the intelligent game in cyberspace. N={att1,att2,…,att i }∪{def1,def2,…,def j } Among them, att i represents the i-th attacking agent, def j represents the j-th defending agent, i and j are natural numbers, and the participants include at least one attacking agent and one defending agent; S represents the state space set of cyberspace intelligent games, which is represented by a graph model: S=<V,E,W> Among them, V represents the node set in the state space, E represents the link set in the state space, and W represents the connection relationship matrix between nodes in the state space; τ represents the state transition probability of the cyberspace intelligent game, which means that the attacking agent takes action a in the current state s. o , the defensive agent takes action a d After that, the probability of transferring to the next state s'; Π represents the strategy set of the participants in the cyberspace intelligent game, which includes the attack agent strategy set and the defense agent strategy set, expressed as: in, Represents the strategy set of the i-th attacking agent, which contains n specific attack strategies π i n ; Represents the strategy set of the jth defense agent, including m specific defense strategies π j m , m and n are natural numbers; A represents the action set of the participants in the cyberspace intelligent game, including the attacking agent action set and the defending agent action set, which can be expressed as: in, Represents the action set of the i-th attacking agent, which contains n specific attack actions a i n ; represents the action set of the jth defense agent, including m specific defense actions a j m ; R represents the benefits of network intelligent game participants, including the benefits of attacking agents and defending agents, expressed as: Among them, R i att represents the profit of the ith attacking agent, R j def represents the payoff of the j-th defending agent.

7. The system according to claim 6, characterized in that The mapping and game module performs the cyberspace intelligent game process specifically as follows: The attacking and defending agents use the DQN algorithm and opponent modeling to perceive the opponent's behavior, and store the observed opponent action sequence into the knowledge base k to form the attacking agent strategy set π o and the defensive agent strategy set π d ; In a round of cyberspace attack and defense game, first, the attacking agent launches an automated attack according to the strategy set; then, the defending agent perceives the behavior of the attacking agent and analyzes the attacking agent strategy set π o , based on the deep reinforcement learning algorithm, makes a defense decision and selects the corresponding action from its action set; next, the attacking agent perceives the behavior of the defending agent and analyzes the defense agent strategy set π d , make attack decisions based on the deep reinforcement learning algorithm and select corresponding actions from its action set; repeat the game many times until the number of game steps in this round reaches the threshold.

8. The system according to claim 7, characterized in that The specific method for the decision module to determine the winner according to the rules and results of the game includes: According to the pre-set reward function, the benefits of the attacking agent and the defending agent after taking actions in the current game round are calculated; the benefit of the attacking agent or the defending agent is equal to the cumulative return value minus the cumulative cost; when the maximum number of game rounds and the maximum number of game steps per round are reached, the cyberspace intelligent game terminates and the party with the higher cumulative benefit wins.

Images