A decision-making method for network defense games based on sequential counterfactual regret minimization

Abstract

The present invention relates to the field of network security, and in particular to a network defense game decision method based on sequential counterfactual regret minimization, the method comprising: constructing a game tree with the initial network state of two forces, each node representing a different network state of the two forces, and corresponding to an information set and a legal action set; initializing the global optimization strategy of both parties, and calculating the node benefit value and the cumulative counterfactual regret value based on the global optimization strategy; calculating the current iteration strategy according to the cumulative counterfactual regret value, and introducing a minimum action probability restriction coefficient to optimize the global optimization strategy; repeating the iteration process until a preset number of iterations is reached; averaging the current iteration strategy of the defender in all iterations to obtain the final network defense strategy. By constructing a game tree and calculating the counterfactual regret value, the present invention can dynamically adjust the current iteration strategy to better adapt to the possible behavior of the opponent's forces.

Description

Technical Field

[0001] The present invention relates to the field of network security, and in particular to a network defense game decision-making method based on sequential counterfactual regret minimization. Background Art

[0002] Cyber ​​attack and defense refers to the confrontation between cyber attackers and defenders in cyberspace. Cyber ​​attackers attempt to use various means to invade or destroy the network system of the network defender, while the network defender takes various measures to resist the attack and protect its own network security. Cyber ​​defense game is a theoretical framework that uses game theory to study cyber attack and defense confrontation. In this framework, cyber attackers and network defenders are regarded as two opposing forces in the game, and their strategies and benefits depend on the behavior of both forces. The study of cyber defense game is of great significance for understanding the laws of cyber attack and defense confrontation and designing effective cyber defense strategies.

[0003] Existing network attack and defense strategy reasoning methods usually aim to solve the Nash equilibrium of the game. However, in actual network attack and defense, existing methods are difficult to cope with complex dynamic network environments and cannot provide network defenders with flexible and adaptable defense strategies. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a network defense game decision method based on sequential counterfactual regret minimization, the method is used to generate a network defense strategy for a network defender in a network attack and defense system, where there are two opposing forces, a network attacker and a network defender, the method comprising:

[0005] Step 1: Use the initial network states of the two forces as the root node to construct a game tree. Each node in the game tree represents a different network state of the two forces, and corresponds to an information set and multiple legal actions of one force. The information set includes nodes corresponding to all possible network states of the opponent force of the force.

[0006] Step 2: Generate global optimization strategies for the two forces respectively. The global optimization strategy represents the probability of one force performing each legal action based on the information set.

[0007] Step 3: Based on the global optimization strategy, calculate the cumulative counterfactual regret value of each information set, and calculate the current iteration strategy of the two forces according to the cumulative counterfactual regret value;

[0008] Step 4: Update the global optimization strategy of the two forces according to the probability of each information set being accessed;

[0009] Step 5: Using the updated global optimization strategy as the global optimization strategy, repeat steps 3 to 4 until a preset number of iterations is reached;

[0010] Step 6: Generate a network defense strategy based on the current iteration strategy of the network defender in all iterations.

[0011] Furthermore, the nodes of the game tree include random interference nodes, each legal action in each random interference node corresponds to a transfer probability and a transfer target node, the transfer target node is a child node of the random interference node, and each random interference node converts the network state of both forces into the network state corresponding to the corresponding transfer target node by executing the legal action corresponding to the transfer probability according to the transfer probability.

[0012] Furthermore, the steps of step three include:

[0013] Step 3A, based on the global optimization strategy, calculate the benefit value of each node;

[0014] Step 3B, based on the benefit value, calculate the cumulative counterfactual regret value corresponding to executing each legal action based on each information set;

[0015] Step 3C: Based on the cumulative counterfactual regret value, calculate the probability of each force performing all legal actions based on each information set to obtain the current iterative strategies of the two forces.

[0016] Furthermore, the steps of step 3A include:

[0017] Starting from the root node of the game tree, traverse all legal actions under each node and recursively calculate each node according to the following formula The revenue value :

[0018] ;

[0019] in, Indicates that at the node Perform legal actions under the corresponding network status The benefit value of the child node reached, Indicates that when the node When it is a leaf node, the predetermined node benefit value, Representation Node The corresponding set of all legal actions, Indicates that when the node When it is a random interference node, the node Corresponding to legal actions The transition probability, Indicated in At the iteration, the power At the node The corresponding network state corresponds to the legal action The global optimization strategy of Representation Node It is neither a leaf node nor a random interference node.

[0020] Furthermore, the steps of step 3B include:

[0021] Based on each node The revenue value , calculate the At the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node ;

[0022] Node-based counterfactual regret , calculate the At the iteration, the power Based on information set Perform legal actions The information set counterfactual regret value :

[0023] ;

[0024] Counterfactual regret value based on information set , calculate the At the iteration, the power Based on information set Perform legal actions The corresponding cumulative counterfactual regret value :

[0025] ;

[0026] in, Indicates The number of the iteration before iteration.

[0027] Furthermore, based on each node The revenue value , calculate the At the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node The steps include:

[0028] Calculate the first At the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node :

[0029] ;

[0030] in, Represents from the root node to the node The historical path A node in and a legal action executed under that node. Indicates that at the node Forces that perform legal actions or randomly interfere with nodes, Indicates The power of the iteration The rival forces At the node The corresponding network state corresponds to the legal action The global optimization strategy of Indicates that at the node Execute legal actions when the corresponding network status The benefit value of the child node reached, Representation Node The revenue value.

[0031] Furthermore, the steps of step 3C include:

[0032] Based on the cumulative counterfactual regret value , according to the following formula, calculate the At the iteration, the power Based on information set Perform every legal action Probability , get the current iteration strategy of each force:

[0033] ;

[0034] in, It means that when the value of the expression in the brackets is less than zero, it takes the value of zero. Otherwise, it takes the value of the expression in the brackets. Representation Information Set The set of all legal actions corresponding to all nodes in A legal action in express The number of legal actions in .

[0035] Furthermore, the steps of step 4 include:

[0036] Calculate the current Iterations, information set The total probability of being visited, and the cumulative arrival probability :

[0037] ;

[0038] in, Indicates The number of the iteration before the iteration, Represents from the root node to the node The historical path A node in and a legal action executed under that node. Indicated in At the iteration, the power At the node The corresponding network state corresponds to the legal action Global optimization strategy of

[0039] Calculate the minimum action probability limit coefficient :

[0040] ;

[0041] in, and are the preset hyperparameters;

[0042] According to the minimum action probability limit coefficient , update the global optimization strategy of the two forces.

[0043] Furthermore, according to the minimum action probability limit coefficient , the steps of updating the global optimization strategy of the two forces include:

[0044] Update the power according to the following formula In the global optimization strategy based on information set Perform legal actions The corresponding probability :

[0045] ;

[0046] in, Representation Information Set The set of all legal actions corresponding to all nodes in The number of legal actions in Indicates At the iteration, the power Based on information set Perform legal actions probability.

[0047] Furthermore, the steps of step six include:

[0048] Calculate the network defense strategy according to the following formula :

[0049] ;

[0050] in, represents the total number of iterations, Indicates that the network defender The current iteration strategy at iteration .

[0051] By constructing a game tree and calculating the counterfactual regret value, the present invention can dynamically adjust the strategy in each iteration, thereby better adapting to the possible behavior of the opponent's forces. In addition, by introducing the minimum action probability restriction coefficient, the present invention further optimizes the exploration ability of the strategy and avoids the problem of strategy rigidity that may occur in traditional methods. Therefore, the present invention significantly improves the flexibility and adaptability of the network defense strategy, enabling it to better cope with complex network attack and defense environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 It is a flow chart of a network defense game decision-making method based on sequential counterfactual regret minimization disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, or product end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, devices, or product ends.

[0056] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] In the network attack and defense system, there are two forces, the network attacker and the network defender, who are rival forces to each other. The present invention proposes a network defense game decision method based on sequential counterfactual regret minimization: firstly, a game tree including all possible network states and legal actions of a network attacker and a network defender is established, each node in the game tree represents the current network state of the two forces, each node corresponds to an information set and at least one legal action, at each node, only one force can execute the legal action corresponding to the node, and the information set includes nodes corresponding to all possible network states of the opponent force of the force executing the legal action; then, the global optimization strategy of the two forces is initialized, and the benefit value and counterfactual regret value of each node in the game tree are recursively calculated by traversing the game tree, and the current iterative strategy is generated by using the counterfactual regret value. In order to allow the current iterative strategy to be updated even when the information set with a probability of reaching 0 is reached, the present invention introduces a mechanism to limit the minimum action probability of the current iterative strategy, and gradually reduces the limit probability. Through multiple iterations, the current iterative strategy is continuously updated and optimized until a preset number of iterations is reached; finally, all current iterative strategies of the network defender are averaged as the final generated network defense strategy, which is deployed and tested in an actual network security system. The network state in the present invention refers to the network environment state of each party at the current node, including comprehensive information such as resource distribution, network topology, actions taken and their effects. The present invention can automatically learn and optimize effective network defense strategies in a complex network attack and defense game environment, and improve the intelligence level and defense capability of the network security system.

[0058] The steps of the present invention are as follows Figure 1 As shown, the steps of the present invention are explained below in conjunction with a specific embodiment:

[0059] 1. Constructing the game tree

[0060] A game tree is constructed with the initial network states of the network attacker and the network defender as the root node. Each node in the game tree represents the current network state of the two forces. The network state refers to the network environment state of one force at the current node, including comprehensive information such as resource distribution, network topology, actions taken and their effects. Each node should have an information set and at least one legal action. Under each node, only one force can perform the legal action corresponding to the node. The information set includes nodes corresponding to all possible network states of the opponent force of the force performing the legal action, indicating that the force performing the legal action cannot accurately judge the specific network state of the opponent force due to incomplete information. Legal actions represent all feasible attack or defense measures that a force can take in a network state, such as deploying firewalls, reinforcing systems, launching different types of attacks, etc. The execution of each legal action will lead to a change in the state of the network environment, thus forming a child node in the game tree.

[0061] In order to more accurately describe the uncertainty factors in the network attack and defense game and make the game tree closer to the actual situation, the present invention introduces random interference nodes in the game tree. The nodes in the game tree include random interference nodes and non-random interference nodes. Each leaf node is a non-random interference node. Each legal action in each random interference node corresponds to a transfer probability and a transfer target node, and the transfer target node is a child node of the random interference node. Corresponding to each legal action, the random interference node converts the network state of both forces into the network state corresponding to the corresponding transfer target node by executing the legal action with the transfer probability corresponding to the legal action. In the actual network attack and defense system, the random interference node represents the impact of sporadic network events on the network state of both forces, such as sporadic network events such as network shocks.

[0062] 2. Traverse the game tree to calculate node benefits

[0063] Initialize a uniform random strategy as the global optimization strategy of the network attacker in the first iteration , initialize a uniform random strategy as the global optimization strategy of the network defender in the first iteration ,The global optimization strategy represents the probability of one force to perform each legal action in a network state, and the uniform random strategy refers to taking all legal actions with equal probability in all possible network states.

[0064] Starting from the root node of the game tree, traverse all legal actions under each node and recursively calculate each node according to the following formula The revenue value :

[0065] ;

[0066] in, Indicates that at the node Perform legal actions under the corresponding network status The benefit value of the child node reached, Indicates that when the node When it is a leaf node, the predetermined node benefit value, Representation Node The corresponding set of all legal actions, Indicates that when the node When it is a random interference node, the node Corresponding to legal actions The transition probability, Indicated in At the iteration, the power At the node The corresponding network state corresponds to the legal action The global optimization strategy of Representation Node It is neither a leaf node nor a random interference node.

[0067] 3. Calculate counterfactual regret

[0068] The payoff value of each node in the game tree It reflects the expected benefits of the two forces corresponding to each node under the current iterative global optimization strategy. The present invention introduces the counterfactual regret value to evaluate whether the two forces can obtain higher benefits by choosing other legal actions under the current global optimization strategy.

[0069] The counterfactual regret value is an indicator used to measure the degree of regret a party has for choosing a legal action under a certain information set. Specifically, the counterfactual regret value indicates that under the current network state, if a party chooses a legal action , instead of choosing legal actions according to the current global optimization strategy, the additional benefits can be obtained. The calculation of the counterfactual regret value not only considers the benefits of the current node, but also takes into account the global optimization strategy of the opponent and the historical path of the game.

[0070] In the In the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node for:

[0071] ;

[0072] in, Represents from the root node to the node The historical path A node in and a legal action executed under that node. Indicates that at the node Forces that perform legal actions or randomly interfere with nodes, Indicates that at the node The power to carry out legal actions, Indicates The power of the iteration The rival forces At the node The corresponding network state corresponds to the legal action Global optimization strategy.

[0073] Since information may be incomplete during the network attack and defense game, each party needs to consider all possible network states of the opponent. At the iteration, the power Perform legal actions based on the information set The information set counterfactual regret value for:

[0074] ;

[0075] in, Indicates power The corresponding information set.

[0076] The power Based on information set Perform legal actions The corresponding cumulative counterfactual regret value for:

[0077] ;

[0078] in, Indicates The number of the iteration before iteration.

[0079] By calculating the counterfactual regret value, we can evaluate the pros and cons of the current iterative strategy and adjust the current iterative strategy in subsequent iterations so that the two forces can gradually reduce the regret and eventually approach the equilibrium solution of the game.

[0080] 4. Generate the current iteration strategy

[0081] The cumulative counterfactual regret value reflects whether the two forces can obtain higher benefits by choosing other legal actions under the current global optimization strategy. The present invention uses the cumulative counterfactual regret value to update the current iterative strategy and approach the equilibrium solution of the game. By dynamically adjusting the current iterative strategy, each force can approach the sequential equilibrium in the long-term game, that is, under each information set, the forces The strategies are optimal.

[0082] In the At the iteration, the current iteration strategy Middle Power Based on information set Perform legal actions Probability for:

[0083] ;

[0084] in, It means that when the value of the expression in the brackets is less than zero, it takes the value of zero. Otherwise, it takes the value of the expression in the brackets. Representation Information Set The set of all legal actions corresponding to all nodes in A legal action in express The number of legal actions in .

[0085] 5. Optimize global optimization strategy

[0086] In order to allow the current iterative strategy to be updated even when the information set has an arrival probability of 0, the present invention introduces a mechanism to limit the minimum action probability of the current iterative strategy, so that the current iterative strategy can eventually approach sequential equilibrium.

[0087] Calculate the current Iterations, information set The total probability of being visited, that is, the cumulative arrival probability :

[0088] ;

[0089] in, Indicated in At the iteration, the power At the node The corresponding network state corresponds to the legal action Global optimization strategy.

[0090] Calculate the minimum action probability limit coefficient :

[0091] ;

[0092] in, and is a preset hyperparameter. .

[0093] Update the global optimization strategy according to the following formula:

[0094] ;

[0095] By introducing the minimum action probability constraint coefficient The invention ensures that the current iterative strategy can be updated even when the probability of reaching the information set is 0, so that the current iterative strategy can approach the sequential equilibrium. This optimization mechanism effectively improves the robustness and adaptability of the network defense game decision-making method, and provides a better defense strategy for complex network attack and defense environments.

[0096] 6. Iterative Optimization

[0097] Recalculate the benefit value and counterfactual regret value of each node based on the updated global optimization strategy, and generate the current iteration strategy for the next iteration. Repeat this step until the preset number of iterations is reached to obtain a series of current iteration strategies. . All current iteration strategies of the network defender Take the average as the generated network defense strategy for the network defender:

[0098] ;

[0099] in, represents the total number of iterations, Indicates that the network defender The current iteration strategy at iteration .

[0100] In summary, the network defense game decision-making method based on sequential counterfactual regret minimization disclosed in the present invention can dynamically adjust the strategy by constructing a game tree and calculating the counterfactual regret value, so as to better adapt to the possible behavior of the opponent's forces. In addition, the present invention further optimizes the exploration ability of the strategy by introducing the minimum action probability restriction coefficient, avoiding the problem of strategy rigidity that may occur in traditional methods. Therefore, the present invention significantly improves the flexibility and adaptability of the network defense strategy, enabling it to better cope with complex network attack and defense environments.

[0101] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A network defense game decision-making method based on sequential counterfactual regret minimization, characterized in that: The method is used to generate a network defense strategy for a network defender in a network attack and defense system, wherein the network attack and defense system includes two opposing forces, a network attacker and a network defender, and the method includes: Step 1: Use the initial network states of the two forces as the root node to construct a game tree. Each node in the game tree represents a different network state of the two forces, and corresponds to an information set and multiple legal actions of one force. The information set includes nodes corresponding to all possible network states of the opponent force of the force. Step 2: Generate global optimization strategies for the two forces respectively. The global optimization strategy represents the probability of one force performing each legal action based on the information set. Step 3: Based on the global optimization strategy, the cumulative counterfactual regret value of each information set is calculated, and the current iteration strategy of the two forces is calculated according to the cumulative counterfactual regret value; Step 4: Update the global optimization strategy of the two forces according to the probability of each information set being accessed; Step 5: Using the updated global optimization strategy as the global optimization strategy, repeat steps 3 to 4 until a preset number of iterations is reached; Step 6: Generate a network defense strategy based on the current iteration strategy of the network defender in all iterations; And, the steps of step four include: Calculate the current Iterations, information set The total probability of being visited, and the cumulative arrival probability : ； in, Indicates The number of the iteration before the iteration, Represents from the root node to the node The historical path A node in and a legal action executed under that node. Indicated in At the iteration, the power At the node The corresponding network state corresponds to the legal action Global optimization strategy of Calculate the minimum action probability limit coefficient : ； in, and are the preset hyperparameters; According to the minimum action probability limit coefficient , update the global optimization strategy of the two forces.

2. According to claim 1, a network defense game decision-making method based on sequential counterfactual regret minimization is characterized in that: The nodes of the game tree include random interference nodes, each legal action in each random interference node corresponds to a transfer probability and a transfer target node, the transfer target node is a child node of the random interference node, and each random interference node converts the network state of both forces into the network state corresponding to the corresponding transfer target node by executing the legal action corresponding to the transfer probability according to the transfer probability.

3. According to claim 2, a network defense game decision-making method based on sequential counterfactual regret minimization is characterized in that: The steps in step three include: Step 3A, based on the global optimization strategy, calculate the benefit value of each node; Step 3B, based on the benefit value, calculate the cumulative counterfactual regret value corresponding to executing each legal action based on each information set; Step 3C: Based on the cumulative counterfactual regret value, calculate the probability of each force performing all legal actions based on each information set to obtain the current iterative strategies of the two forces.

4. According to claim 3, a network defense game decision-making method based on sequential counterfactual regret minimization is characterized in that: The steps of Step 3A include: Starting from the root node of the game tree, traverse all legal actions under each node and recursively calculate each node according to the following formula The revenue value : ； in, Indicates that at the node Perform legal actions under the corresponding network status The benefit value of the child node reached, Indicates that when the node When it is a leaf node, the predetermined node benefit value, Representation Node The corresponding set of all legal actions, Indicates that when the node When it is a random interference node, the node Corresponding to legal actions The transition probability, Indicated in At the iteration, the power At the node The corresponding network state corresponds to the legal action The global optimization strategy of Representation Node It is neither a leaf node nor a random interference node.

5. According to claim 4, a network defense game decision-making method based on sequential counterfactual regret minimization is characterized in that: The steps of step 3B include: Based on each node The revenue value , calculate the At the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node ; Node-based counterfactual regret , calculate the At the iteration, the power Based on information set Perform legal actions The information set counterfactual regret value : ； Counterfactual regret value based on information set , calculate the At the iteration, the power Based on information set Perform legal actions The corresponding cumulative counterfactual regret value : ； in, Indicates The number of the iteration before iteration.

6. A network defense game decision-making method based on sequential counterfactual regret minimization according to claim 5, characterized in that: Based on each node The revenue value , calculate the At the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node The steps include: Calculate the first At the iteration, the power At the node The corresponding network state corresponds to the legal action Counterfactual regret value of the node : ； in, Represents from the root node to the node The historical path A node in and a legal action executed under that node. Indicates that at the node Forces that perform legal actions or randomly interfere with nodes, Indicates Iteration time force The rival forces At the node The corresponding network state corresponds to the legal action The global optimization strategy of Indicates that at the node Execute legal actions when the corresponding network status The benefit value of the child node reached, Representation Node The revenue value.

7. A network defense game decision-making method based on sequential counterfactual regret minimization according to claim 5, characterized in that: The steps of step 3C include: Based on the cumulative counterfactual regret value , according to the following formula, calculate the At the iteration, the power Based on information set Perform every legal action Probability , get the current iteration strategy of each force: ； in, It means that when the value of the expression in the brackets is less than zero, it takes the value of zero. Otherwise, it takes the value of the expression in the brackets. Representation Information Set The set of all legal actions corresponding to all nodes in A legal action in express The number of legal actions in .

8. According to claim 1, a network defense game decision-making method based on sequential counterfactual regret minimization is characterized in that: According to the minimum action probability limit coefficient , the steps of updating the global optimization strategy of the two forces include: Update the power according to the following formula In the global optimization strategy based on information set Perform legal actions The corresponding probability : ； in, Representation Information Set The set of all legal actions corresponding to all nodes in The number of legal actions in Indicates At the iteration, the power Based on information set Perform legal actions probability.

9. A network defense game decision-making method based on sequential counterfactual regret minimization according to claim 1, characterized in that: Steps in step six include: Calculate the network defense strategy according to the following formula : ； in, represents the total number of iterations, Indicates that the network defender The current iteration strategy at iteration .

Images