A defense strategy decision method and system based on stochastic game and reinforcement learning
Patent Information
- Application Number
- CN202311392429.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-25
- Publication Date
- 2026-08-18
- Estimated Expiration
- 2043-10-25
AI Technical Summary
[0006]为克服上述现有技术的不足，本发明提供了一种基于随机博弈与强化学习的防御策略决策方法及系统，改进防御者的学习方法，克服基于完全理性假设所得的防御策略实用性低的缺点，提升防御者的学习速度
本发明将强化学习算法与随机博弈模型相结合，在网络攻防过程的初始阶段，基于有限理性约束，防御者与攻击者进行大量交互产生训练样本存入经验池，防御者利用优先经验回放机制从经验池中抽取样本进行训练学习，逐步寻找到各个博弈状态所对应的最优防御策略，克服基于完全理性假设所得的防御策略实用性低的缺点，改进了防御者的学习方法，提升了防御者的学习速度。
Smart Images
Figure CN117499087B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of network security defense for industrial control systems, and particularly relates to a defense strategy decision-making method and system based on random game theory and reinforcement learning. Background Technology
[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.
[0003] In the wave of industry-wide digitalization, the informatization level of industrial control systems is increasing. On the one hand, this saves industries significant hardware and infrastructure costs and significantly improves productivity. On the other hand, due to the increasing connection with the Internet, industrial control systems are suffering from complex and ever-changing cyberattacks. Furthermore, in the process of cyber attack and defense, different defense strategies will produce different defense benefits for a given attack strategy. Therefore, how to select the optimal defense strategy for industrial control systems to obtain the best defense benefits has broad practical significance.
[0004] In recent years, much research addressing these issues has focused on novel methods, such as game theory and reinforcement learning. First, the adversarial objectives between attackers and defenders, and the mutual influence between attack and defense strategies, make the network defense decision-making process highly compatible with the fundamental processes involved in game theory. Furthermore, stochastic games, a combination of game theory and Markov decision-making, can characterize the randomness and dynamism of network attacks and defenses. Therefore, using stochastic games to build network attack and defense models to guide defense decisions has gradually become a research hotspot.
[0005] Existing methods for analyzing network attacks and defenses of industrial control systems based on stochastic game theory generally employ the assumption of perfect rationality. Perfect rationality includes the pursuit of maximum gain, analytical reasoning ability, and judgment ability, among other perfection requirements. Imperfection in any of these aspects constitutes bounded rationality. However, in actual network attack and defense processes, it is difficult to achieve the conditions of perfect rationality. As a result, existing defense strategies based on the assumption of perfect rationality still suffer from low practicality, poor accuracy, and poor robustness. Summary of the Invention
[0006] To overcome the shortcomings of the prior art, this invention provides a defense strategy decision-making method and system based on random game theory and reinforcement learning, which improves the learning method of the defender, overcomes the disadvantage of low practicality of defense strategies obtained based on the assumption of perfect rationality, and improves the learning speed of the defender.
[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions: The first aspect of this invention provides a defense strategy decision-making method based on random game theory and reinforcement learning.
[0008] A defense strategy decision-making method based on stochastic game theory and reinforcement learning includes: Under the constraint of bounded rationality, a network attack and defense game model is constructed to analyze the network attack and defense process of industrial control systems. By utilizing the priority experience replay mechanism, a PER-DDQN algorithm is constructed based on the DDQN algorithm; By combining the network attack and defense game model with the PER-DDQN algorithm as a learning method for defenders, the defenders can learn the optimal defense strategy corresponding to each game state under the constraint of bounded rationality. In the initial stage of the network attack and defense process, the learning involves a large number of interactions between the defender and the attacker to generate training samples that are stored in the experience pool. The defender uses a priority experience replay mechanism to extract samples from the experience pool for training and gradually finds the optimal defense strategy corresponding to each game state through learning.
[0009] Furthermore, the network attack-defense game model is a six-tuple, defined as NAD-SGM = (N,S,D,R,Q, );
[0010] Where N represents the participants in the game, S represents the state space of the random game, and D represents the set of defensive actions. Represents the immediate report after the defender takes a defensive action; This represents the expected benefit of the defender after taking a defensive action. The defensive strategy of the defender.
[0011] Furthermore, the aforementioned bounded rationality constraint means that in the constructed network attack and defense game model, the defender chooses the defense strategy based on the probability of choosing a defensive action. In the initial stage of the network attack and defense process, both sides adopt random probability selection strategies and find the optimal defense strategy through learning.
[0012] Furthermore, the PER-DDQN algorithm, based on the DDQN algorithm, incorporates a priority experience replay mechanism. This mechanism alters the probability of a sample being drawn, changing the probability of sample m being drawn in the PER-DDQN algorithm. Defined as:
[0013]
[0014] in, This represents the probability that the m-th sample is selected for learning. Represents priority parameters; This represents the priority of the m-th sample. k This represents the number of training samples.
[0015] Furthermore, the loss function of the PER-DDQN algorithm is:
[0016] in, Represents the target Q value. Represents the state of the agent at time t. Take action The predicted Q value, The importance sampling weights are used to correct errors when calculating the gradient of samples. The specific formula is as follows:
[0017]
[0018] Where N represents the size of the experience pool. Represents the degree of control correction. This represents the probability that the m-th sample is selected.
[0019] Furthermore, the combination of the network attack-defense game model with the PER-DDQN algorithm specifically involves: The environment of the PER-DDQN algorithm is modeled using the game state of a network attack-defense game model. The agent in the PER-DDQN algorithm corresponds to the defender in the network attack-defense game model. The agent's behavior adopts the defensive action of the network attack-defense game model. The agent's immediate reward corresponds to the immediate reward of the network attack-defense game model. The agent's policy corresponds to the defensive policy of the network attack-defense game model.
[0020] Furthermore, the specific process of learning under bounded rationality constraints is as follows: When an attack is detected, the optimal protection strategy is calculated based on the current state and the selected defense action is executed. This action is stored as an experience in the experience pool and given a higher priority than previous experiences. When the number of samples in the experience pool reaches the number of samples required for a training batch, samples are drawn for learning based on the probability of the samples being drawn.
[0021] A second aspect of the present invention provides a defense strategy decision system based on random game theory and reinforcement learning.
[0022] A defense strategy decision-making system based on stochastic game theory and reinforcement learning includes a model building module, an algorithm construction module, and a learning decision-making module. The model building module is configured to: construct a network attack and defense game model under bounded rationality constraints to analyze the network attack and defense process of industrial control systems; The algorithm construction module is configured to construct the PER-DDQN algorithm based on the DDQN algorithm by utilizing the priority experience replay mechanism. The learning decision module is configured to combine the network attack and defense game model with the PER-DDQN algorithm as a learning method for the defender, so that the defender can learn the optimal defense strategy corresponding to each game state under the constraint of bounded rationality. In the initial stage of the network attack and defense process, the learning involves a large number of interactions between the defender and the attacker to generate training samples that are stored in the experience pool. The defender uses a priority experience replay mechanism to extract samples from the experience pool for training and gradually finds the optimal defense strategy corresponding to each game state through learning. A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a defense strategy decision-making method based on random game theory and reinforcement learning as described in the first aspect of the present invention.
[0023] A fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in a defense strategy decision-making method based on random game theory and reinforcement learning as described in the first aspect of the present invention.
[0024] The above one or more technical solutions have the following beneficial effects: This invention combines reinforcement learning algorithms with a stochastic game model. In the initial stage of network attack and defense, based on bounded rationality constraints, the defender and attacker engage in a large number of interactions to generate training samples that are stored in an experience pool. The defender uses a priority experience replay mechanism to extract samples from the experience pool for training and learning, gradually finding the optimal defense strategy corresponding to each game state. This overcomes the shortcomings of defense strategies based on the assumption of perfect rationality, which has low practicality, improves the defender's learning method, and increases the defender's learning speed. This invention utilizes a priority experience replay mechanism to construct the PER-DDQN algorithm based on the DDQN algorithm. Based on the difference between the target Q value and the predicted Q value, the priority of the samples is adjusted, thereby changing the sampling probability of the samples. This enables the PER-DDQN algorithm to learn from the samples more efficiently and make optimal decisions.
[0025] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description
[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.
[0027] Figure 1 This is a flowchart of the method in the first embodiment.
[0028] Figure 2 This is a schematic diagram of the learning method in the first embodiment.
[0029] Figure 3 This is a schematic diagram of the industrial water distribution system used in the first embodiment experiment.
[0030] Figure 4 The diagram shows the game state transition relationship of the industrial water distribution system in the first embodiment.
[0031] Figure 5 This is a comparison chart of the convergence speeds of different algorithms in the first embodiment. Detailed Implementation
[0032] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0033] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0034] Given that real-world network attack and defense processes rarely achieve perfect rationality, defense strategy decisions can be made under bounded rationality constraints. Under these constraints, attackers and defenders do not initially adopt optimal strategies. This means that defenders need to learn and gradually find the optimal defense strategy during the attack and defense process. In this process, the defender's learning method is a key factor in finding the optimal defense strategy. Reinforcement learning is a classic learning method and has been widely studied as a learning method for defenders in stochastic game models. However, existing reinforcement learning algorithms combined with stochastic game models suffer from unstable training processes and slow convergence speeds. Therefore, there is an urgent need for a more efficient reinforcement learning algorithm combined with stochastic game models to enable defenders to learn the optimal defense strategy corresponding to each game state under bounded rationality constraints.
[0035] Example 1 In one or more embodiments, a defense strategy decision-making method based on stochastic game theory and reinforcement learning is disclosed, such as... Figure 1 As shown, it includes the following steps: Step S1: Under the constraint of bounded rationality, construct a network attack and defense game model to analyze the network attack and defense process of industrial control systems.
[0036] The network attack and defense game model is a six-tuple, defined as: NAD-SGM = (N,S,D,R,Q, (1)
[0037] Where N = ( , () represents the participants in the game. Representing the attacker and Represents the defender; S = ( , ,…, ) represents the state space of a random game, consisting of the system states; D = ( , ,…, ) represents the set of defensive actions, where, = ( , ,…, ) represents the state. A set of defensive actions; Represents the defender in status Take defensive action The system then transitions to state. Immediate returns; The state-action reward function represents the defender's state. Take defensive action Expected returns afterward; = ( ( , ), ( , ),…, ( , )) represents the defender in a certain state. The defense strategy, among which, ( , () represents the defender in the state Choose a defensive action The probability and .
[0038] In the initial stage of network attack and defense, the probability of choosing a defensive action is randomized. Subsequently, the defender and attacker interact extensively. As sample experience accumulates, a priority experience replay mechanism is used to extract samples from the experience pool for training and learning. The probability of the defensive action is updated based on the state-action payoff, thereby constructing the optimal defense strategy. This is a characteristic of bounded rationality constraints. In simple terms, under complete rationality constraints, both the attacker and defender will adopt the optimal strategy from the beginning. Under bounded rationality constraints, both the attacker and defender will not adopt the optimal strategy at the beginning, but will adopt random strategies, and then learn the optimal strategy through learning.
[0039] Step S2: Construct the PER-DDQN algorithm based on the DDQN algorithm by utilizing the priority experience replay mechanism.
[0040] The DDQN algorithm is an interaction model between an agent and its environment. The agent perceives the state of the environment and takes actions. The environment provides corresponding feedback to the agent based on the actions and transitions to the next state. The DDQN algorithm consists of two neural networks, a Q-network and a target Q-network. The two neural networks have the same structure but different parameters. The Q-network calculates the predicted Q-value corresponding to the current state action pair, using the following formula: (2)
[0041] in, Represents the state of the agent at time t. Take action The predicted Q value, Network parameters representing Q-network, This refers to an action in a general sense, which in this embodiment can be either a defensive action or an offensive action.
[0042] The target Q-network computes the current state. The corresponding next state The target Q value is calculated using the following formula:
[0043] (3)
[0044] in, Represents the target Q value; The agent represents the state. Transition to state Immediate returns; Represents the discount factor; Represents the state The action corresponding to the maximum Q value selected by the Q-network; The network parameters represent the target Q-network.
[0045] The loss function of the DDQN algorithm is calculated by comparing the predicted Q-value with the target Q-value. The calculation formula is as follows: (4)
[0046] in, Represents the target Q value. Represents the state of the agent at time t. Take action The predicted Q value, where E represents the expectation function.
[0047] The PER-DDQN algorithm adds a priority experience replay mechanism to the DDQN algorithm. This mechanism alters the probability of a sample being drawn from the experience pool for learning. In the PER-DDQN algorithm, the probability of sample m being drawn... Defined as:
[0048] (5)
[0049] in, This represents the probability that the m-th sample is selected. Represents priority parameters; This represents the number of training samples, which is the number of samples in the experience pool. The priority of the m-th sample is represented by the following formula:
[0050] (6)
[0051] in, This represents the absolute value of the TD error for the m-th sample, where the TD error is the difference between the target Q value and the predicted Q value. This is a small, non-zero positive number used to prevent the probability from being 0. Since the priority experience replay mechanism alters the sampling probability of a sample, an importance sampling weight needs to be introduced to correct for errors in gradient calculation. The formula for calculating the importance sampling weight is:
[0052] (7)
[0053] Where N represents the size of the experience pool. Represents the degree of control correction. This represents the probability that the m-th sample is selected.
[0054] Therefore, the loss function of the PER-DDQN algorithm is updated as follows: (8)
[0055] Step S3: Combine the network attack and defense game model with the PER-DDQN algorithm as a learning method for the defender, so that the defender can learn the optimal defense strategy corresponding to each game state under the constraint of bounded rationality.
[0056] Specifically, the combination of NAD-SGM and PER-DDQN algorithms constitutes the defender's learning method, such as... Figure 2 As shown, specifically, the environment of the PER-DDQN algorithm is modeled using the game states of NAD-SGM. The agent in the PER-DDQN algorithm corresponds to the defender in NAD-SGM, the agent's behavior adopts the defensive actions of NAD-SGM, the agent's immediate reward corresponds to the immediate reward of NAD-SGM, and the agent's policy corresponds to the defensive policy of NAD-SGM. In the initial stage of network attack and defense, the defender and the attacker interact extensively to generate training samples and store them in the experience pool. Afterward, the defender uses the priority experience replay mechanism to extract samples from the experience pool for training, and gradually finds the optimal protection policy corresponding to each game state through learning.
[0057] Based on the above, the specific defense strategy decision-making algorithm, especially the process by which the defender gradually converges to the optimal strategy through learning, is shown in Table 1, as follows: Table 1 Defense Strategy Decision Algorithm
[0058] (1) Set and initialize the relevant parameters of the network attack-defense game model NAD-SGM and PER-DDQN algorithm, i.e., steps 1-2 in Table 1. The parameters for setting initial values are experience pool size N, parameter update period K, batch size k, and learning rate. ,index and Discount Factor The initialization process is as follows: (The initialization process is:)
[0059] Initialize NAD-SGM, set initial parameter values, and establish the experience pool. Cumulative weight changes The weights of the Q-network The target Q-network weights are random values. .
[0060] (2) The defender's defense strategy decision-making process, namely steps 4-8 in Table 1, involves calculating the optimal defense strategy based on the current state and executing the selected defense action when an attack is detected. This action is stored as an experience in the experience pool and given a higher priority than previous experiences. Specifically: Q-network is based on the current state Choose a defensive action based on the probability of the defensive action. a Calculate the optimal protection strategy and execute the selected defense action 'a' to obtain the next state. And immediately return R, will Store this as an experience point in the experience pool. In the middle, and give it a higher priority than previous experience.
[0061] (3) The process of the defender learning through the priority experience replay mechanism, i.e., steps 9-22 in Table 1, is as follows: When the number of samples in the experience pool reaches the number of samples required for a training batch, samples are drawn according to the probability of being drawn for learning. Specifically: If the current size of the experience pool is divisible by the parameter update period, meaning the number of samples in the experience pool has reached the required number of samples for a training batch, then the experience pool is updated according to priority. Samples are drawn from the sample and the sampling probability of the sample is calculated. Then calculate the importance weights of the samples. Calculate the TD error of the sample. Priority of updating samples Cumulative weight changes Regularly update weights Reset weight changes The weights are copied to the target network at fixed intervals. Finally, the model's state is updated.
[0062] Through the attached Figure 3 The industrial water distribution system shown was used to conduct simulation experiments to verify the effectiveness of the method in this embodiment.
[0063] The simulation system mainly includes two management hosts, a human-machine interface, three PLCs (Programmable Logic Controllers), and their controlled physical actuators. Attackers discover and exploit vulnerabilities in the target system to gain system privileges and thus compromise it. The system's vulnerabilities are scanned using the Nessus tool, as shown in Table 2. Table 2 Vulnerability List
[0064] Based on the vulnerability information, Table 3 lists the possible attack actions that attackers might take, and the corresponding defensive actions are shown in Table 4. During network attack and defense, changes in system state are often caused by the attacker's attack actions; therefore, the different system privileges obtained by the attacker are considered as the set of game states in network attack and defense, as shown in Table 5. The game state transition relationships of the simulation system are as follows: Figure 4 As shown, the defensive action set for each game state is as follows: , , , , , , The game state changes when an attacker successfully exploits the vulnerability; otherwise, the system maintains its current state.
[0065] Table 3 List of Attack Actions
[0066] Table 4 List of Defensive Actions
[0067] Table 5. List of Game States
[0068] Throughout the game, the state The choice of defense strategy at this point is the most complex; therefore, this embodiment compares the various algorithms in terms of state. The convergence status at the point of attack is shown in Table 6, which lists the key parameter settings of the PER-DDQN algorithm. In the experiment, the attacker adopted a random attack strategy.
[0069] Table 6. List of key parameters for the PER-DDQN algorithm
[0070] Figure 5 The graph shows a comparison of the convergence speeds of various algorithms. The horizontal axis represents the number of experimental iterations, and the vertical axis represents the probability of a defensive action being selected. That is, when the optimal strategy is learned, the probability of a defensive action being selected is a constant, at which point convergence is achieved. Experimental results show that the PER-DDQN algorithm converges to the optimal strategy faster and more stably than the other comparative algorithms. Specifically, the PER-SAC algorithm converges to the optimal defense strategy after 116 iterations, while the Q-learning algorithm, DDQN algorithm, and DRQN algorithm converge after 285 iterations, 197 iterations, and 170 iterations, respectively. This indicates that the priority experience replay mechanism enables the PER-DDQN algorithm to more effectively utilize samples to learn and make optimal decisions.
[0071] The above description of the defense strategy decision-making method of the present invention, in conjunction with specific embodiments, shows that, compared with existing typical methods, the present invention enables the defender to obtain the optimal defense strategy corresponding to each game state more quickly and stably through learning under the constraint of bounded rationality.
[0072] Example 2 In one or more embodiments, a defense strategy decision-making system based on stochastic game theory and reinforcement learning is disclosed, including a model building module, an algorithm construction module, and a learning decision-making module: The model building module is configured to: construct a network attack and defense game model under bounded rationality constraints to analyze the network attack and defense process of industrial control systems; The algorithm construction module is configured to construct the PER-DDQN algorithm based on the DDQN algorithm by utilizing the priority experience replay mechanism. The learning decision module is configured to combine the network attack and defense game model with the PER-DDQN algorithm as a learning method for the defender, so that the defender can learn the optimal defense strategy corresponding to each game state under the constraint of bounded rationality. In the initial stage of the network attack and defense process, the learning involves a large number of interactions between the defender and the attacker to generate training samples that are stored in the experience pool. The defender uses a priority experience replay mechanism to extract samples from the experience pool for training and gradually finds the optimal defense strategy corresponding to each game state through learning.
[0073] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium.
[0074] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a defense strategy decision-making method based on random game theory and reinforcement learning as described in Embodiment 1 of this disclosure.
[0075] Example 4 The purpose of this embodiment is to provide an electronic device.
[0076] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in a defense strategy decision-making method based on random game theory and reinforcement learning as described in Embodiment 1 of this disclosure.
[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A defense strategy decision-making method based on stochastic game theory and reinforcement learning, characterized in that, include: Under the constraint of bounded rationality, a network attack and defense game model is constructed to analyze the network attack and defense process of industrial control systems. By utilizing the priority experience replay mechanism, a PER-DDQN algorithm is constructed based on the DDQN algorithm; The PER-DDQN algorithm, based on the DDQN algorithm, incorporates a priority experience replay mechanism. This mechanism alters the probability of a sample being drawn. In the PER-DDQN algorithm, the probability of sample m being drawn... Defined as: in, This represents the probability that the m-th sample is selected for learning. Represents priority parameters; This represents the priority of the m-th sample. k Represents the number of training samples; By combining the network attack and defense game model with the PER-DDQN algorithm as a learning method for defenders, the defenders can learn the optimal defense strategy corresponding to each game state under the constraint of bounded rationality. The combination of the network attack-defense game model with the PER-DDQN algorithm is specifically as follows: The environment of the PER-DDQN algorithm is modeled using the game state of the network attack and defense game model. The agent in the PER-DDQN algorithm corresponds to the defender in the network attack and defense game model. The agent's behavior adopts the defensive action of the network attack and defense game model. The agent's immediate reward corresponds to the immediate reward of the network attack and defense game model. The agent's policy corresponds to the defensive policy of the network attack and defense game model. The specific process of learning under the constraint of bounded rationality is as follows: When an attack is detected, the optimal protection strategy is calculated based on the current state and the selected defense action is executed. This action is stored as an experience in the experience pool and given a higher priority than previous experiences. When the number of samples in the experience pool reaches the number of samples required for a training batch, samples are drawn for learning based on the probability of the samples being drawn. In the initial stage of the network attack and defense process, the learning involves a large number of interactions between the defender and the attacker to generate training samples that are stored in the experience pool. The defender uses a priority experience replay mechanism to extract samples from the experience pool for training and gradually finds the optimal defense strategy corresponding to each game state through learning.
2. The defense strategy decision-making method based on stochastic game theory and reinforcement learning as described in claim 1, characterized in that, The network attack-defense game model is a six-tuple, defined as NAD-SGM = (N,S,D,R,Q, ); Where N represents the participants in the game, S represents the state space of the random game, and D represents the set of defensive actions. Represents the immediate report after the defender takes a defensive action; This represents the expected benefit of the defender after taking a defensive action. The defensive strategy of the defender.
3. The defense strategy decision-making method based on random game theory and reinforcement learning as described in claim 2, characterized in that, The aforementioned bounded rationality constraint, in the constructed network attack and defense game model, is that the defender chooses the defense strategy based on the probability of choosing a defensive action. In the initial stage of the network attack and defense process, both sides adopt random probability selection strategies and find the optimal defense strategy through learning.
4. The defense strategy decision-making method based on stochastic game theory and reinforcement learning as described in claim 1, characterized in that, The loss function of the PER-DDQN algorithm is: in, Represents the target Q value. Represents the state of the agent at time t. Take action The predicted Q value, The importance sampling weights are used to correct errors when calculating the gradient of samples. The specific formula is as follows: Where N represents the size of the experience pool. Represents the degree of control correction. This represents the probability that the m-th sample is selected.
5. A defense strategy decision-making system based on stochastic game theory and reinforcement learning, characterized in that, It includes a model building module, an algorithm construction module, and a learning decision module: The model building module is configured to: construct a network attack and defense game model under bounded rationality constraints to analyze the network attack and defense process of industrial control systems; The algorithm construction module is configured to construct the PER-DDQN algorithm based on the DDQN algorithm by utilizing the priority experience replay mechanism. The PER-DDQN algorithm, based on the DDQN algorithm, incorporates a priority experience replay mechanism. This mechanism alters the probability of a sample being drawn. In the PER-DDQN algorithm, the probability of sample m being drawn... Defined as: in, This represents the probability that the m-th sample is selected for learning. Represents priority parameters; This represents the priority of the m-th sample. k Represents the number of training samples; The learning decision module is configured to combine the network attack and defense game model with the PER-DDQN algorithm as a learning method for the defender, so that the defender can learn the optimal defense strategy corresponding to each game state under the constraint of bounded rationality. The combination of the network attack-defense game model with the PER-DDQN algorithm is specifically as follows: The environment of the PER-DDQN algorithm is modeled using the game state of the network attack and defense game model. The agent in the PER-DDQN algorithm corresponds to the defender in the network attack and defense game model. The agent's behavior adopts the defensive action of the network attack and defense game model. The agent's immediate reward corresponds to the immediate reward of the network attack and defense game model. The agent's policy corresponds to the defensive policy of the network attack and defense game model. The specific process of learning under the constraint of bounded rationality is as follows: When an attack is detected, the optimal protection strategy is calculated based on the current state and the selected defense action is executed. This action is stored as an experience in the experience pool and given a higher priority than previous experiences. When the number of samples in the experience pool reaches the number of samples required for a training batch, samples are drawn for learning based on the probability of the samples being drawn. In the initial stage of the network attack and defense process, the learning involves a large number of interactions between the defender and the attacker to generate training samples that are stored in the experience pool. The defender uses a priority experience replay mechanism to extract samples from the experience pool for training and gradually finds the optimal defense strategy corresponding to each game state through learning.
6. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-4.
7. A storage medium, characterized in that, The computer-readable instructions are stored non-temporarily, wherein when the computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-4 are executed.