An intelligent decision system and method based on an attack-defense game model
By constructing an intelligent decision-making system based on an attack-defense game model, improving the action state value function of a deep Q-network using a genetic algorithm, and introducing a third-party performance evaluation mechanism, the influence of random factors in the network attack-defense game model is resolved, thereby improving the accuracy and effectiveness of strategy selection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INST OF COMP TECH & APPL
- Filing Date
- 2023-05-25
- Publication Date
- 2026-06-05
AI Technical Summary
Existing network attack and defense game models fail to effectively consider random factors, resulting in insufficient accuracy and effectiveness of decision-making.
An intelligent decision-making system based on an attack-defense game model is constructed, including a situation analysis module, an attack-defense game model framework construction module, an attack-defense game model training and optimization module, and a third-party performance evaluation module. The action state value function is improved by using a genetic algorithm to train a deep Q-network, and the system is verified by the third-party performance evaluation module.
This improves the rationality and accuracy of network attack and defense game models, ensuring the effectiveness and accuracy of strategy selection.
Smart Images
Figure CN116846592B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of network security technology, specifically relating to an intelligent decision-making system and method based on an attack-defense game model. Background Technology
[0002] With the continuous evolution and transformation of information warfare, cyber warfare is playing an increasingly important role in the modern battlefield as a new type of combat force. Due to the complexity of the cyber battlefield environment, traditional human decision-making is increasingly unable to support combat needs, and it is necessary to rely on intelligent decision-making assistance to ensure the correctness and timeliness of decisions.
[0003] In cyber attack and defense, both sides always make rational decisions using limited resources to achieve the greatest benefit at the lowest cost. Therefore, the essence of cyber attack and defense can be abstracted as a game between the two sides. In other words, the decision-making problem in cyber attack and defense is suitable for study using game theory. However, traditional game theory is based on a replicative dynamic learning mechanism, where decision-makers adjust their strategies through learning to maximize their own gains, but it does not consider the interference of various random factors in the game process. In actual attack and defense processes, the choice of attack methods, changes in the system operating environment, and interference from other external factors all have a certain degree of randomness. Ignoring the consideration of random factors will reduce the effectiveness and accuracy of models and methods. Summary of the Invention
[0004] (a) Technical problems to be solved
[0005] The technical problem to be solved by this invention is to provide an intelligent decision-making system and method based on an attack-defense game model for the intelligent selection of strategies in common network attack-defense confrontations, thereby improving the rationality and accuracy of the network attack-defense game model.
[0006] (II) Technical Solution
[0007] To address the aforementioned technical problems, this invention provides an intelligent decision-making system based on an attack-defense game model, comprising: a situation analysis module, an attack-defense game model framework construction module, an attack-defense game model training and optimization module, a third-party performance evaluation module, and a decision-making module; wherein:
[0008] The situation analysis module is used to analyze the offensive and defensive situation of both sides, extract the environmental state elements and strategic action elements of both sides, as well as the effectiveness evaluation index elements of both sides. The strategic action elements include strategic elements and action elements.
[0009] The attack and defense game model framework construction module is used to abstract the attack and defense confrontation problem between the two sides into a network attack and defense game model based on the environmental state elements and strategy action elements of both sides.
[0010] The attack-defense game model training and optimization module is used to improve the deep Q network based on the genetic algorithm to approximate the action state value function, thereby realizing the training and optimization of the network attack-defense game model.
[0011] The third-party performance evaluation module is used to test the network attack and defense game model based on the performance evaluation indicators of both parties.
[0012] The decision-making module is used to input the current state and solve for the current optimal strategy in a network attack and defense game model that has been trained, optimized, and verified by a third party.
[0013] The present invention also provides an intelligent decision-making method utilizing the aforementioned intelligent decision-making system, comprising the following steps:
[0014] (1) The situation analysis module analyzes the current situation information of both parties. Through the threat feature library and attack feature library, it uses multi-dimensional data analysis methods and correlation analysis methods to extract the environmental state elements and strategic action elements of both parties and transmits them to the attack and defense game model framework construction module. It also extracts the performance evaluation index elements of both parties and transmits them to the third-party performance evaluation module.
[0015] (2) The attack and defense game model framework construction module is based on environmental state elements and strategy action elements. It abstracts the attack and defense confrontation problem between the two sides into a network attack and defense game model through the Markov game model.
[0016] (3) Training and optimization module for attack and defense game model: The attack and defense game model is trained and optimized by using a deep Q network to approximate the action state value function Q(s,a,b). In the process of approximation, the deep Q network is improved based on the genetic algorithm. That is, the weight parameter θ of the neural network is optimized and trained by a hybrid genetic algorithm combined with the gradient descent algorithm until the loss function reaches the preset minimum error value.
[0017] (4) The third-party performance evaluation module constructs a third-party performance evaluation model based on the performance evaluation indicators of both parties, and uses the third-party performance evaluation model to test the trained and optimized network attack and defense game model. If the test is passed, the trained and optimized network attack and defense game model is selected as the final model; if the test is not passed, the training samples are reselected, and the weight parameter θ of the network attack and defense game model trained and optimized in the previous round is selected as the initial weight to continue training the network attack and defense game model until the trained and optimized network attack and defense game model passes the test of the third-party performance evaluation model.
[0018] (5) In the network attack and defense game model that has been trained, optimized and verified by a third party, the decision module takes the current state s as input and solves the solution (a,b) that makes the action state value function Q(s,a,b) optimal, which is the current optimal strategy selection.
[0019] Preferably, the situational information in step (1) includes basic environment information, attack behavior information, and defense behavior information. The basic environment information includes open service information, open ports, memory, and CPU. The attack behavior information includes attacker IP, attack device fingerprint information, attack behavior, attack characteristics, attack strategy, and attack results. The defense behavior information includes protection strategy, security function information, interception information, and alarm information.
[0020] Preferably, in step (2), a network attack and defense game model is constructed using a Markov game model, where the enemy represents the attacker and our side represents the defender, and the tuples are determined. <n,S,A1,....,A n ,T,γ,R1,....R n > Represents a network attack and defense game model, where:
[0021] Number n: represents the number of participants in the attack and defense game, n=2, representing the attacker and the defender respectively;
[0022] State S: Represents the state space, which is the set of all states s. The state will change after the attacking and defending parties select strategies and take actions. The state space contains all environmental state elements extracted by the situation analysis module.
[0023] Action A: Represents the action space, which describes the strategic actions of the attacker and defender; A1 represents the set of actions of the defender, and A2 represents the set of actions of the attacker; the action space contains all strategic action elements extracted from the situation analysis module.
[0024] The transition function T is the probability of transitioning from the current state s to the next state s′ under the influence of the joint action strategy (a,b) of our side choosing action a according to the strategy and the enemy choosing action b according to the strategy.
[0025] Discount factor γ: is the decay of future rewards, γ∈[0,1];
[0026] The reward function R is the reward obtained by both the attacker and the defender at state s' after taking joint action (A1,A2) in state s. R1 represents the reward function of the defender and R2 represents the reward function of the attacker. The attacker and the defender have opposite reward functions, i.e., R1 = -R2. The action state value function Q(s,a,b) represents the expected reward of choosing the joint action strategy (a,b) starting from state s.
[0027] Preferably, in step (3), the implementation method of the attack-defense game model training and optimization module includes:
[0028] Let the optimal value function be given by the state s of the Markov game. Where, πa The strategy, i.e., the probability of choosing action 'a', is represented by PD(A), which represents the discrete probability distribution of the action. Q(s,a,b) represents the action state-value function, and the action state-value function in the t-th iteration is Q. t (s,a,b)=(1-α)*Q t-1 (s,a,b)+α*(r+γV(s′)), where Q t-1 (s,a,b) is the action state value function for the (t-1)th iteration, where α represents the learning efficiency and r represents the current reward.
[0029] Step (3) is as follows:
[0030] Building a neural network involves finding a set of parameters θ to represent the weights of each layer in the network. The process of updating the action state value function is to continuously train and update the parameters θ. A neural network consists of an input layer, a hidden layer, and an output layer. The input layer contains state features, and the output layer contains Q(s,a,b) corresponding to the possible actions of both players in the game.
[0031] Collect K (K≥1000) sets of training samples (s,a,b,r,s′) to start training. During training, the sample order needs to be shuffled and samples are randomly selected from them for training.
[0032] Define a loss function and use a hybrid genetic algorithm combining the genetic algorithm and gradient descent to optimize the network attack-defense game model: First, give the initial parameters of the genetic algorithm and the initial point of the gradient descent algorithm; then, compare the optimal solution generated by the genetic algorithm with the optimal solution generated by the gradient descent algorithm, and select the best set of solutions as the starting point of the next round of the gradient descent algorithm; second, use the solution generated by each iteration of the gradient descent algorithm as the superior individual to replace the worst individual in the genetic algorithm; continue to iterate until the loss function reaches the pre-set minimum error value, then stop training; finally, obtain the optimal set of parameters θ, thus approximating the action state value function Q(s,a,b).
[0033] Preferably, the loss function is defined as loss = (target_q - q). 2 In this context, target_q represents the target Q value, which is the target value updated through interaction with the environment and learning. q is the original Q value. The entire training process is the process of the Q value (q) approaching the target Q value (target_q).
[0034] Preferably, in step (4), the third-party performance evaluation module constructs a network confrontation performance evaluation model as the third-party performance evaluation model: the network confrontation performance evaluation index system framework is sorted out by extracting the performance evaluation index elements of both parties from the situation analysis module, the network confrontation performance evaluation index system framework is converted into a neural network model, and the weight parameters of the neural network are adjusted using a swarm intelligence algorithm. The network adversarial effectiveness evaluation model was optimized and trained, and then used to test the optimized network attack and defense game model.
[0035] Preferably, the method for constructing the network adversarial effectiveness evaluation model in step (4) specifically includes:
[0036] (1) Construct a network confrontation effectiveness index system framework: The effectiveness evaluation index elements of both sides extracted from the situation analysis module are sorted into a network confrontation effectiveness index system framework.
[0037] (2) The network adversarial effectiveness index system framework is transformed into a neural network model, with the input layer being index elements and the output layer being adversarial effectiveness. The weight parameters of the neural network model are:
[0038] (3) Input training samples and use swarm intelligence algorithms to adjust the weight parameters of the neural network. Training is performed to eventually obtain the optimal set of parameters. This leads to the final network adversarial effectiveness evaluation model, which serves as a third-party effectiveness evaluation model.
[0039] Preferably, the specific method for verifying the trained and optimized network attack and defense game model using the network adversarial effectiveness evaluation model in step (4) is as follows:
[0040] The current network attack-defense game model was tested using a third-party performance evaluation model: the action state value obtained after choosing an action in one state is positively proportional to the network adversarial effectiveness in the next state; input the current state. Select the first m solutions (a1, b1), (a2, b2), ..., (a...) that optimize the action state value Q function. m ,b m ), calculate in state respectively The following adopts a joint action strategy (a1,b1), (a2,b2),...,(a m ,b m The states s′1, s′2, ..., s′ after ) m Then, the efficiency of the system is calculated using a third-party performance evaluation model at states s′1, s′2, ..., s′. m Network adversarial effectiveness e′1,e′2,...,e′ mIf the ranking of network adversarial effectiveness is consistent with the ranking of optimal solutions, then the test is passed; if the ranking of network adversarial effectiveness is inconsistent with the ranking of optimal solutions, then K′ (K′≥1000) sets of training samples are reselected, and the weight parameters θ of the previous round of network attack and defense game model are selected as the initial weights to continue training the model; until the trained and optimized network attack and defense game model passes the third-party effectiveness evaluation model test.
[0041] The present invention also provides a network attack and defense countermeasure method implemented using the aforementioned method.
[0042] (III) Beneficial Effects
[0043] (1) In order to solve the problem that the state space is continuous and multidimensional and cannot be calculated by a simple Q table, when updating the action state value Q function in the Markov game, we propose to improve the deep Q network based on the genetic algorithm. That is, when constructing the neural network to approximate the Q function, we use a hybrid genetic algorithm combined with the gradient descent algorithm to train and optimize the model, so as to avoid getting trapped in local optima and make the construction of the attack and defense game model more accurate and efficient.
[0044] (2) Introduce a third-party performance evaluation mechanism and use the third-party performance evaluation model to test the attack and defense game model. Only the attack and defense game model that passes the test can be used as the final attack and defense game model, so as to select the optimal action in the current state and improve the accuracy and effectiveness of strategy selection. Attached Figure Description
[0045] Figure 1 This is a schematic diagram illustrating the overall architecture and implementation principle of the intelligent decision-making system based on the attack-defense game model of this invention. Detailed Implementation
[0046] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.
[0047] This invention presents an intelligent decision-making system and method based on an attack-defense game model, taking a network adversarial environment as a backdrop. Considering the complexity of the network adversarial environment, the problem is abstracted into a zero-sum game. By constructing a network game model and introducing a third-party performance evaluation mechanism to verify the attack-defense game model, this invention helps decision-makers select the optimal adversarial strategy more intuitively and intelligently. To improve the rationality and accuracy of the network attack-defense game model, this invention uses a genetic algorithm to improve the deep Q-network to approximate the action-state value function, solving the problem of continuous multidimensional state space. Simultaneously, the third-party performance evaluation mechanism verifies the attack-defense game model, making it more accurate and thus providing decision support for decision-makers.
[0048] like Figure 1 As shown, this invention provides an intelligent decision-making system based on an attack-defense game model, including a situation analysis module, an attack-defense game model framework construction module, an attack-defense game model training and optimization module, a third-party performance evaluation module, and a decision-making module. The situation analysis module analyzes the attack-defense situation of both sides, extracting environmental state elements, strategic action elements, and performance evaluation index elements for both sides. The strategic action elements include both strategy elements and action elements. The attack-defense game model framework construction module abstracts the attack-defense confrontation problem between the two sides into a network attack-defense game model based on the environmental state elements and strategic action elements. The attack-defense game model training and optimization module improves the deep Q-network using a genetic algorithm to approximate the action state value function, thereby optimizing the training of the network attack-defense game model. The third-party performance evaluation module verifies the network attack-defense game model based on the performance evaluation index elements for both sides. The decision-making module, in the trained and optimized network attack-defense game model that has passed third-party verification, takes the current state as input and solves to obtain the current optimal strategy selection.
[0049] The specific process of the intelligent decision-making method implemented using this system is as follows:
[0050] (1) The situation analysis module analyzes the current situation information, extracts the environmental state elements and strategic action elements of both parties through the threat feature library, attack feature library, etc., and transmits them to the attack and defense game model framework construction module. It also extracts the performance evaluation index elements of both parties and transmits them to the third-party performance evaluation module.
[0051] (2) The attack and defense game model framework construction module is based on environmental state elements and strategy action elements. It abstracts the attack and defense confrontation problem between the two sides into a network attack and defense game model through the Markov game model.
[0052] (3) Training and optimization module for attack and defense game model: The attack and defense game model is trained and optimized by using a deep Q network to approximate the action state value function Q(s,a,b). In the process of approximation, the deep Q network is improved based on the genetic algorithm. That is, a hybrid genetic algorithm combining gradient descent algorithm is used to optimize and train the weight parameter θ of the neural network until the loss function reaches the preset error minimum value, so that the algorithm can converge globally faster.
[0053] (4) The third-party performance evaluation module constructs a third-party performance evaluation model based on the performance evaluation indicators of both parties, and uses the third-party performance evaluation model to test the trained and optimized network attack and defense game model. If the test is passed, the trained and optimized network attack and defense game model is selected as the final model; if the test is not passed, the training samples are reselected, and the weight parameter θ of the network attack and defense game model trained and optimized in the previous round is selected as the initial weight to continue training the network attack and defense game model until the trained and optimized network attack and defense game model passes the test of the third-party performance evaluation model.
[0054] (5) In the network attack and defense game model that has been trained, optimized and verified by a third party, the decision module takes the current state s as input and solves the solution (a,b) that makes the action state value function Q(s,a,b) optimal, which is the current optimal strategy selection.
[0055] In step (1), the situation analysis module analyzes the offensive and defensive situation of both sides. The situation information includes basic environmental information, attack behavior information, and defense behavior information. Among them, the basic environmental information includes open service information, open ports, memory, CPU, etc.; the attack behavior information includes attacker IP, attack device fingerprint information, attack behavior, attack characteristics, attack strategy, attack results, etc.; and the defense behavior information includes protection strategy, security function information, interception information, alarm information, etc. At the same time, the environmental state elements, strategy action elements, and effectiveness evaluation index elements of both sides are extracted from the above situation information through multi-dimensional data analysis, correlation analysis, and other means, and passed to the offensive and defensive game model framework construction module; the effectiveness evaluation index elements of both sides are extracted and passed to the third-party effectiveness evaluation module.
[0056] In step (2), the implementation method of the attack-defense game model framework construction module includes:
[0057] Since an increase in the gain for one side inevitably leads to a decrease in the gain for the other, the confrontation between the attacker and defender is essentially a zero-sum game. A network attack-defense game model is constructed using a Markov game model, with the enemy representing the attacker and our side representing the defender, to determine the tuples. <n,S,A1,....,A n ,T,γ,R1,....R n > Represents a network attack and defense game model, where:
[0058] (1) Number n: represents the number of participants in the attack and defense game, n=2, which are the attacker and the defender respectively.
[0059] (2) State S: Represents the state space, which is the set of all states s. The state will change after the attacking and defending parties select strategies and take actions. The state space contains all environmental state elements extracted by the situation analysis module.
[0060] (3) Action A: Represents the action space, which describes the strategic actions of the attacker and defender. A1 represents the set of actions of the defender, and A2 represents the set of actions of the attacker. The action space contains all strategic action elements extracted from the situation analysis module.
[0061] (4) Transition function T: The probability of transitioning from the current state s to the next state s′ under the influence of the joint action strategy (a,b) of our side choosing action a according to the strategy and the enemy choosing action b according to the strategy.
[0062] (5) Discount factor γ: The discount factor is the decay of future rewards, γ∈[0,1].
[0063] (6) Reward function R: The reward obtained by both the attacker and defender at state s' after taking joint action (A1,A2) in state s. R1 represents the reward function of the defender, and R2 represents the reward function of the attacker. The attacker and defender have opposite reward functions, i.e., R1 = -R2. The action state value function Q(s,a,b) represents the expected reward of choosing the joint action strategy (a,b) starting from state s.
[0064] In step (3), the implementation method of the attack-defense game model training and optimization module includes:
[0065] In a Markov game with state s, the optimal value function is: Where, π a The strategy, i.e., the probability of choosing action 'a', is represented by PD(A), which represents the discrete probability distribution of the action. Q(s,a,b) represents the action state-value function, and the action state-value function for the t-th iteration (this iteration) is Q(s,a,b). t (s,a,b)=(1-α)*Q t-1 (s,a,b)+α*(r+γV(s′)), where Q t-1 (s,a,b) is the action state value function of the (t-1)th iteration (previous iteration), where α represents the learning efficiency, r represents the current reward, and γ∈[0,1].
[0066] Due to the complex and ever-changing nature of offensive and defensive game dynamics, and the continuous multidimensional state space, the action state value function Q cannot be calculated using a simple Q-table. Therefore, this invention proposes to approximate the action state value function Q(s,a,b) using a deep Q-network, and simultaneously improves the deep Q-network based on a genetic algorithm, enabling the algorithm to converge globally more quickly. Details are as follows:
[0067] (1) Construct a neural network to approximate the action state value function Q(s,a,b). That is, find a set of parameters θ to represent the weights of each layer in the neural network. The process of updating the action state value function is to continuously train and update the parameters θ. The neural network contains an input layer, a hidden layer, and an output layer. The input layer is the state features, and the output layer is Q(s,a,b) corresponding to the actions that the two players can choose.
[0068] (2) Collect K (K≥1000) sets of training samples (s,a,b,r,s′) to start training. During training, the order of the samples needs to be shuffled and samples are randomly selected for training to break the correlation between the samples and make the algorithm more likely to converge.
[0069] (3) Define the loss function: loss = (target_q - q) 2 Here, `target_q` represents the target Q-value, which is the updated target value after interaction with the environment and learning, while `q` is the original Q-value. A hybrid genetic algorithm combining genetic and gradient descent is used to replace the traditional gradient descent algorithm for optimization: First, the initial parameters of the genetic algorithm and the initial point of the gradient descent algorithm are given. Then, the optimal solution generated by the genetic algorithm is compared with the optimal solution generated by the gradient descent algorithm, and the best set of solutions is selected as the starting point for the next round of the gradient descent algorithm. Second, the solution generated by each iteration of the gradient descent algorithm is used as the superior individual to replace the worst individual in the genetic algorithm. This process is repeated until the loss function reaches the pre-set minimum error value, at which point training stops. Finally, the optimal set of parameters θ is obtained, thereby approximating the action state value function Q(s,a,b). The entire training process is actually the process of the Q-value (q) approaching the target Q-value (target_q).
[0070] In step (4), the third-party performance evaluation module constructs a network adversarial performance evaluation model as the third-party performance evaluation model. The network adversarial performance evaluation index framework is derived from the performance evaluation index elements extracted by the situation analysis module. This framework is then converted into a neural network model, and swarm intelligence algorithms (such as genetic algorithms, particle swarm optimization, ant colony optimization, and cuckoo algorithm) are used to adjust the weight parameters of the neural network. The network adversarial effectiveness evaluation model was optimized and trained, and then used to test the optimized network attack and defense game model.
[0071] Specifically, the implementation methods of the third-party performance evaluation module include:
[0072] First, a network adversarial effectiveness evaluation model is constructed. The specific method is as follows:
[0073] (1) Construct a network confrontation effectiveness index system framework: The effectiveness evaluation index elements of both sides extracted from the situation analysis module are sorted into a network confrontation effectiveness index system framework.
[0074] (2) The network adversarial effectiveness index system framework is transformed into a neural network model, with the input layer being index elements and the output layer being adversarial effectiveness. The weight parameters of the neural network model are:
[0075] (3) Input training samples and use swarm intelligence algorithms (e.g., genetic algorithm, particle swarm optimization, ant colony optimization, cuckoo algorithm, etc.) to replace the traditional gradient descent algorithm for the weight parameters of the neural network. Training is performed to avoid getting trapped in local optima. The ultimate goal is to obtain the optimal set of parameters. This leads to the final network adversarial effectiveness evaluation model, which serves as a third-party effectiveness evaluation model.
[0076] Secondly, the trained and optimized network attack-defense game model is tested using a network adversarial effectiveness evaluation model. The specific method is as follows:
[0077] The current network attack-defense game model is tested using a third-party performance evaluation model. In the network attack-defense game model, the action state value obtained after choosing an action in a certain state should be directly proportional to the network adversarial effectiveness in the next state. Input the current state. Select the first m solutions (a1, b1), (a2, b2), ..., (a...) that optimize the action state value Q function. m ,b m ), calculate in state respectively The following adopts a joint action strategy (a1,b1), (a2,b2),...,(a m ,b m The states s′1, s′2, ..., s′ after ) m Then, the performance evaluation model is used to calculate the efficiency of the system in states s′1, s′2, ..., s′. m Network adversarial effectiveness e′1,e′2,...,e′ m If the ranking of network adversarial effectiveness matches the ranking of optimal solutions, then the test is passed. If the ranking of network adversarial effectiveness does not match the ranking of optimal solutions, then K′ (K′≥1000) sets of training samples are selected again, and the weight parameters θ of the previous round of network attack and defense game model are used as the initial weights to continue training the model. This continues until the trained and optimized network attack and defense game model passes the third-party effectiveness evaluation model test.
[0078] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. An intelligent decision-making method implemented using an intelligent decision-making system based on an attack-defense game model, characterized in that, The system includes: a situation analysis module, an attack-defense game model framework construction module, an attack-defense game model training and optimization module, a third-party performance evaluation module, and a decision-making module; among which: The situation analysis module is used to analyze the offensive and defensive situation of both sides, extract the environmental state elements and strategic action elements of both sides, as well as the effectiveness evaluation index elements of both sides. The strategic action elements include strategic elements and action elements. The attack and defense game model framework construction module is used to abstract the attack and defense confrontation problem between the two sides into a network attack and defense game model based on the environmental state elements and strategy action elements of both sides. The attack-defense game model training and optimization module is used to improve the deep Q network based on the genetic algorithm to approximate the action state value function, thereby realizing the training and optimization of the network attack-defense game model. The third-party performance evaluation module is used to test the network attack and defense game model based on the performance evaluation indicators of both parties. The decision-making module is used to input the current state into a network attack and defense game model that has been trained, optimized, and verified by a third party, and solve for the current optimal strategy. The method includes the following steps: (1) The situation analysis module analyzes the current situation information of both parties. Through the threat feature library and attack feature library, it uses multi-dimensional data analysis methods and correlation analysis methods to extract the environmental state elements and strategic action elements of both parties and transmits them to the attack and defense game model framework construction module. It also extracts the performance evaluation index elements of both parties and transmits them to the third-party performance evaluation module. (2) The attack and defense game model framework construction module is based on environmental state elements and strategy action elements. It abstracts the attack and defense confrontation problem between the two sides into a network attack and defense game model through the Markov game model. (3) Training and optimization module for attack and defense game model: Training and optimizing the network attack and defense game model: using a deep Q network to approximate the action state value function. Furthermore, during the approximation process, the deep Q-network is improved based on a genetic algorithm, specifically by using a hybrid genetic algorithm combining gradient descent to adjust the weight parameters of the neural network. Optimize and train until the loss function reaches the pre-set minimum error value; (4) The third-party performance evaluation module constructs a third-party performance evaluation model based on the performance evaluation indicators of both parties, and uses the third-party performance evaluation model to test the trained and optimized network attack and defense game model. If the test is passed, the trained and optimized network attack and defense game model is selected as the final model; if the test is not passed, training samples are reselected, and the weight parameters of the network attack and defense game model trained and optimized in the previous round are selected. The network attack and defense game model is continued to be trained using the initial weights until the trained and optimized network attack and defense game model is verified by a third-party performance evaluation model. (5) The decision-making module inputs the current state into the network attack and defense game model that has been trained, optimized, and verified by a third party. Solve for the action state value function Optimal solution This is the current optimal strategy selection; In step (2), a network attack and defense game model is constructed using a Markov game model. The enemy represents the attacker, and our side represents the defender. The tuples are then determined. Represents a network attack and defense game model, in which: Number : Represents the number of participants in an offensive and defensive game. They are the attacker and the defender, respectively; state : Represents the state space, which contains all states. The set of states changes after both the attacker and defender choose strategies and take actions; the state space contains all environmental state elements extracted by the situation analysis module. action : Represents the action space, which describes the strategic actions of the attacker and defender; The set of actions representing the defending side. The action space represents the set of actions of the attacker; it contains all the strategic action elements extracted from the situation analysis module. Transfer function : From the current state Our side selects actions based on strategy Choose actions based on strategy with the enemy. Joint behavioral strategies Under the influence of this, it transitions to the next state. The probability of; Discount factor This refers to the decay of future rewards. ; Reward function : It refers to the state of both the offensive and defensive sides. Take joint action Later in state The returns obtained from that place The reward function representing the defender. The attacker's reward function represents the payoff function, while the attacker and defender have opposite reward functions, i.e. ; using action state value function Indicates from state Start selecting joint behavior strategies Expected rewards; In step (3), the implementation method of the attack-defense game model training and optimization module includes: Let the state of the Markov game be... Below, the optimal value function is ,in, Representation strategy, i.e., action selection The probability, Represents the discrete probability distribution of an action; The function representing the action state value, the first t The action state value function of the round iteration is ,in, No. t -1 rounds of iteration's action state value function, Represents learning efficiency. This represents the reward currently received; Step (3) is as follows: Constructing a neural network to approximate the action state value function That is, to find a set of parameters This represents the weights of each layer in the neural network. The process of updating the action state value function is to continuously train and update the parameters. A neural network consists of an input layer, hidden layers, and an output layer. The input layer contains state features, and the output layer contains the possible actions of both players. ; collect K Group training samples To begin training, the sample order needs to be shuffled, and samples need to be randomly selected for training. Define a loss function and use a hybrid genetic algorithm combining genetic descent and gradient descent to optimize the network attack-defense game model: First, give the initial parameters of the genetic algorithm and the initial point of the gradient descent algorithm; then, compare the optimal solution generated by the genetic algorithm with the optimal solution generated by the gradient descent algorithm, and select the best set of solutions as the starting point for the next round of the gradient descent algorithm; second, replace the worst individual in the genetic algorithm with the solution generated by each iteration of the gradient descent algorithm as the superior individual; continue this process until the loss function reaches a pre-set minimum error value, then stop training; finally, obtain the optimal set of parameters. This approximates the action state value function. .
2. The method as described in claim 1, characterized in that, In step (1), the situational information includes basic environment information, attack behavior information, and defense behavior information. The basic environment information includes open service information, open ports, memory, and CPU. The attack behavior information includes attacker IP, attack device fingerprint information, attack behavior, attack characteristics, attack strategy, and attack results. The defense behavior information includes protection strategy, security function information, interception information, and alarm information.
3. The method as described in claim 1, characterized in that, Define loss function ,in, target _ q This represents the target Q-value, which is the target value updated through interaction with and learning from the environment. q The original Q value is used; the entire training process is represented by the Q value. q Towards the target Q value ( target _ q The process of approaching.
4. The method as described in claim 1, characterized in that, In step (4), the third-party effectiveness evaluation module constructs a network confrontation effectiveness evaluation model as the third-party effectiveness evaluation model: the network confrontation effectiveness indicator system framework is sorted out by extracting the effectiveness evaluation index elements of both sides from the situation analysis module, the network confrontation effectiveness indicator system framework is converted into a neural network model, and the weight parameters of the neural network are adjusted using a swarm intelligence algorithm. The network adversarial effectiveness evaluation model was optimized and trained, and then used to test the optimized network attack and defense game model.
5. The method as described in claim 4, characterized in that, The specific methods for constructing the network adversarial effectiveness evaluation model in step (4) include: (1) Construct a framework for network confrontation effectiveness index system: The effectiveness evaluation index elements of both sides extracted from the situation analysis module are sorted into a framework for network confrontation effectiveness index system. (2) The network adversarial effectiveness index system framework is transformed into a neural network model, with the input layer being index elements and the output layer being adversarial effectiveness. The weight parameters of the neural network model are: ; (3) Input training samples and use swarm intelligence algorithms to adjust the weight parameters of the neural network. Training is performed to eventually obtain the optimal set of parameters. Thus, the final network adversarial effectiveness evaluation model is obtained as a third-party effectiveness evaluation model.