A deep reinforcement learning strategy optimization defense method and device based on imitation learning
By using imitation learning and adversarial training methods in deep reinforcement learning, the deep reinforcement learning strategy is optimized, the potential safety hazards of reinforcement learning in the field of security decision-making are solved, and the robustness and security of the model are improved.
Patent Information
- Application Number
- CN202110282288.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-16
- Publication Date
- 2025-01-21
- Estimated Expiration
- 2041-03-16
AI Technical Summary
Existing reinforcement learning is vulnerable to attacks or automatic decision-making loopholes in the field of security decision-making, resulting in inaccurate decision-making results and poses security risks.
The defense method is optimized based on the deep reinforcement learning strategy based on imitation learning. By building the smart driving simulation environment of the intelligent body, using the deep Q network and the generative format against the network for countermeasure training, and optimizing the deep strengthening learning strategy to improve the robustness.
It improves the robustness of the deep reinforcement learning model, enhances the security in the field of security decision-making, and prevents inaccurate decisions caused by attacks or decision-making loopholes.
Smart Images
Figure CN112884131B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of defense for deep reinforcement learning, and specifically relates to a deep reinforcement learning strategy optimization defense method and device based on imitation learning. Background Art
[0002] Deep reinforcement learning is one of the areas of artificial intelligence that has received much attention in recent years. With the rapid development and application of reinforcement learning, reinforcement learning has been widely used in robot control, game gaming, computer vision, unmanned driving and other fields. In order to ensure the safe application of deep reinforcement learning in safety-critical fields, the key is to analyze and discover loopholes in deep reinforcement learning algorithms and models to prevent people with ulterior motives from exploiting these loopholes to make illegal profits. Unlike the single-step prediction task of traditional machine learning, the deep reinforcement learning system requires multi-step decision-making to complete a task, and there is a high correlation between consecutive decisions.
[0003] Reinforcement learning is usually a continuous decision-making process. Its basic idea is to learn the optimal strategy to achieve the learning purpose by maximizing the cumulative rewards obtained by the agent from the environment. Deep reinforcement learning makes full use of neural networks as parameter structures, combines the perception ability of deep learning and the decision-making ability of reinforcement learning to optimize deep reinforcement learning strategies, and finally realizes an end-to-end deep reinforcement learning framework from perception input to decision output. It has strong learning ability and is widely used. However, similar to machine learning models, reinforcement learning is also susceptible to interference from adversarial samples. According to the attack on the sample, it can be mainly divided into observation attack, reward attack, action attack and environment attack. At the same time, the training process of deep reinforcement learning will also be subject to poisoning attacks, which increases the training cost of the model and reduces the performance of the model. In addition, the strategy of deep reinforcement learning training also has security risks, especially in safety-critical fields, such as complex combat environments, combat command environments, drone armored vehicles, drone monitoring machines, intelligent robot control, etc. The existence of model strategy vulnerabilities will cause serious damage to the reinforcement learning system. Especially in safety-critical fields, this security risk brings great harm and will cause the decision-making of the reinforcement learning system to be wrong, which is a major challenge for the decision-making safety application field of reinforcement learning.
[0004] Since the strategies obtained by reinforcement learning training have security risks, improving the robustness of reinforcement learning models and strategies and making them effectively and safely applied in the field of security decision-making has become a focus of attention. At present, according to the existing defense mechanisms, common reinforcement learning defense methods can be divided into three categories: adversarial training, robust learning, and adversarial detection. Adversarial training refers to training the model by adding adversarial samples to training samples. Its main purpose is to improve the generalization ability of the strategy beyond normal samples. However, adversarial training can often only improve the strategy's ability to fit the samples involved in training. Robust learning is a learning mechanism that improves the robustness of the training model when facing attack methods from the training stage or the test stage. Adversarial detection refers to the model distinguishing between normal samples and adversarial samples, and processing adversarial samples without modifying the original model parameters to achieve defense effects. Summary of the invention
[0005] In order to solve the problem that existing reinforcement learning is attacked or has automatic decision-making loopholes in the field of security decision-making (such as autonomous driving scenarios), which leads to inaccurate decision-making results and potential safety hazards, the purpose of the present invention is to provide a deep reinforcement learning strategy optimization defense method and device based on imitation learning.
[0006] In order to achieve the above-mentioned invention object, the present invention provides the following technical solutions:
[0007] A deep reinforcement learning strategy optimization defense method based on imitation learning includes the following steps:
[0008] Build a deep reinforcement learning intelligent agent autonomous driving simulation environment, construct a target intelligent agent based on the deep Q network in reinforcement learning, and perform reinforcement learning on the target intelligent agent to optimize the parameters of the deep Q network;
[0009] A deep Q network with optimized parameters is used to generate a sequence of state-action pairs of the target agent driving at T moments as expert data, where the action value in the state-action pair corresponds to the action with the smallest Q value;
[0010] Construct an adversarial agent based on a generative adversarial network and perform imitation learning on the adversarial agent, that is, use the state in the expert data as the input of the generative adversarial network, and use the expert data as a label to supervise and optimize the parameters of the generative adversarial network;
[0011] Based on the state generated by the adversarial agent, the target agent is trained adversarially, and then the parameters of the deep Q network are optimized to achieve optimized defense of the deep reinforcement learning strategy.
[0012] A deep reinforcement learning strategy optimization defense device based on imitation learning includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. When the computer processor executes the computer program, the deep reinforcement learning strategy optimization defense method based on imitation learning is implemented.
[0013] Compared with the prior art, the present invention has the following beneficial effects:
[0014] 1) Imitation learning is used to train the adversarial agent to optimize the target agent's strategy and improve the robustness of the deep reinforcement learning model; 2) In the imitation learning process, the state-action pairs in the expert data are collected from the target agent's action with the smallest Q value. The output of the discriminator is used as the reward value to guide the adversarial agent's strategy learning to generate various strategy traps that put the target agent in a dangerous state; 3) The target agent and the adversarial agent are trained in alternating adversarial games. The target agent is trained for N steps, and the adversarial agent takes an action once to experience the dangerous environment state in advance, thereby improving the robustness of the target agent's strategy. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0016] Figure 1 is a flowchart of a deep reinforcement learning strategy optimization defense method based on imitation learning provided in an embodiment;
[0017] Figure 2 is a schematic diagram of adversarial agent training based on imitation learning provided by an embodiment;
[0018] Figure 3 It is a schematic diagram of the DQN algorithm structure in reinforcement learning provided by the embodiment. DETAILED DESCRIPTION
[0019] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.
[0020] Based on the defense mechanism of robust learning, the embodiment provides a deep reinforcement learning strategy optimization defense method based on imitation learning, which is mainly used in autonomous driving scenarios. The technical concept is: in the deep reinforcement learning training process of simulating the autonomous driving of a small car, the attack method based on strategy poisoning will cause the learner to learn a wrong strategy, thereby choosing a bad action, causing the learner to learn incorrectly. Based on this situation, the method uses the imitation learning mechanism to train the adversarial intelligent agent to produce adversarial erroneous actions, and uses the erroneous actions as strategy traps to disrupt the strategy of the target intelligent agent during the training process of the target intelligent agent, so that it can foresee the damage caused by the disturbed environmental state and erroneous behavior, thereby improving the strategy robustness of the target intelligent agent and enhancing the security of the deep reinforcement learning model.
[0021] Figure 1 : is a flowchart of a deep reinforcement learning strategy optimization defense method based on imitation learning provided by an embodiment. Figure 1 As shown, the deep reinforcement learning strategy optimization defense method based on imitation learning provided in the embodiment includes the following steps:
[0022] Step 1: Build a deep reinforcement learning intelligent agent autonomous driving simulation environment, construct a target intelligent agent based on the deep Q network (DQN) in reinforcement learning, and perform reinforcement learning on the target intelligent agent to optimize the parameters of the deep Q network.
[0023] In the embodiment, the intelligent agent can be a car in an autonomous driving environment, and the car is trained using a deep Q network in reinforcement learning, with the goal of making the car reach the destination as quickly as possible. The deep Q network combines Q learning and convolutional neural networks to construct a training model for deep reinforcement learning.
[0024] Among them, Figure 3 As shown in the figure, the deep Q network includes a main Q network and a target Q network, wherein the main Q network is used as the decision network of the target intelligent agent, and the target Q network is used as the value network of the target intelligent agent. Both the main Q network and the target Q network use convolutional neural networks. When the target intelligent agent is reinforced, the main Q network uses the decision Q value calculated according to the state as the action value, and generates the state value and reward value of the next moment according to the action value. The state value, action value and reward value are all stored. The target Q network takes the target Q value calculated by the environment at the next moment from the storage, and updates the main Q network with the decision Q value and target Q value output by the main Q network and the target Q network, as well as the accumulated reward value, and copies the current main Q network to the target Q network at regular intervals.
[0025] In the embodiment, the specific process of the DQN algorithm is:
[0026] (a) Current main Q θThe input of the network is the image data of the current scene, that is, the state of the environment, and the output is the value function corresponding to each action to estimate the accumulated long-term reward, that is, the decision Q value;
[0027] (b) DQN also uses a target network mechanism, that is, in the current main Q θ Based on the network, a target Q with exactly the same structure was built θ- The network constitutes the overall model framework of DQN. During the training process, the current main Q θ The decision Q value output by the network is used to select action a, and another target Q θ- The network is used to calculate the target Q value. The mean square error of the decision Q value and the target Q value including the cumulative reward value is used as the loss function to update the main Q network, where the loss function is:
[0028]
[0029] in, is the target Q value, and the current main Q is updated through the reverse gradient propagation of the neural network θ The network parameters θ, the serial number i represents the index of the input data, and m is the number of input data.
[0030] (c) During the training process, DQN uses the experience replay mechanism to convert the state transition process (state s i 、Action a i , Reward i , the next state s i ') is stored in the experience replay buffer Buff as the training data set of the network model, and batch learning is performed in the form of random sampling.
[0031] (d) Sample N training data sets from Buff and update the current main Q by minimizing the loss function θ The network parameters of the network are updated every once in a while. θ The parameters of the network are copied to the target Q θ- The network is updated with a delay before the next round of learning.
[0032] After the above steps (a) to (d), the parameters of the deep Q network of the target intelligent agent can be pre-trained. The pre-trained deep Q network is used to generate data pairs as sample data for adversarial intelligent agent learning.
[0033] Step 2: Use the parameter-optimized deep Q network to generate a sequence of state-action pairs of the target agent driving at T moments as expert data, where the action value in the state-action pair corresponds to the action with the smallest Q value.
[0034] The process of generating state-action pair sequences is a sampling process, using the T-time car driving sequence state-action pairs {(s1, a1), ..., (s T ,a T )} as expert data, where action a is the action with the smallest corresponding decision Q value, indicating that the action is a wrong action, where s represents the state value.
[0035] Step 3: construct an adversarial agent based on the generative adversarial network and perform imitation learning on the adversarial agent, that is, use the state in the expert data as the input of the generative adversarial network, and use the expert data as labels to supervise and optimize the parameters of the generative adversarial network.
[0036] In an embodiment, a generative adversarial network includes a generator and a discriminator, wherein the generator serves as a decision network of the adversarial agent, and the discriminator serves as a value network of the adversarial agent. When performing reinforcement learning on the adversarial agent, the state value of each state-action pair in the expert data is used as the input of the generator. The generator calculates and generates a predicted action value based on the input state value, and the state-action pair consisting of the predicted action value and the corresponding input state value is input to the discriminator. The state-action pairs with the same state value in the expert data are also input as labels to the discriminator. The discriminator is used to judge the action value generated by the generator, and use the judgment result as a reward value to guide the learning of the generator.
[0037] Among them, both the generator and the discriminator use convolutional neural networks. In order to achieve a role similar to a strategy trap, the adversarial agent learns to make the target agent fall into the wrong strategy by maximizing the expected reward value, and minimizes the loss function to update the network structure parameters. Specifically, Figure 2 As shown in Figure 1, the specific process of reinforcement learning for adversarial agents includes:
[0038] (a) In the generator, by inputting the current state data, the corresponding action strategy π(a|s; θ) is obtained. π(a|s; θ) represents the probability of selecting action a under the conditions of s and θ. The logarithm of the probability is multiplied by the total reward R of the action to obtain the gradient of θ, and θ is updated in a gradient ascent manner. The state s in the sampled expert data is input into the generator network (that is, the Actor network), and the corresponding action a' is obtained according to the network's initialization strategy parameters to generate the state-action pair (s, a') of the adversarial agent;
[0039] (b) The purpose of GAN is to make the generated data similar to the target data distribution (the distribution represented by the training set data). The discriminator D can be regarded as a binary classifier and represented by cross entropy:
[0040]
[0041] Among them, minmax represents the maximum and minimum objective function in Nash equilibrium, P data (x) is the distribution of sample data. The first item logD(x) represents the judgment of the discriminator on the real data, and the second item log(1-D(x)) represents the judgment on the generated data. Through such a maximum-minimum game process, G and D are optimized alternately to train the required generative network and discriminative network until the Nash equilibrium point is reached. In the training process of the adversarial agent, the Actor network is used to replace the generator G. Its output actions and states are input into the discriminator in pairs and compared with the expert data. The output of the discriminator D:S×A→(0,1) is used as a reward value to guide the strategy learning of the adversarial agent. Therefore, the discriminator loss function in imitation learning can be expressed as:
[0042]
[0043] Among them, E π [logD(s,a)] represents the ability value calculated by the discriminator based on the state action value generated by the generator, E πE [1-logD(s,a) represents the energy value calculated by the discriminator based on the state-action pair (s,a) in the expert data, D∈(0,1)S×A represents the output of the discriminator as a decimal between 0 and 1 with a size of S×A;
[0044] (c) During the training process, the adversarial agent minimizes the loss function by gradient derivation and reversely updates the discriminator and actor network parameters. Its loss function is as follows:
[0045]
[0046] in, is the entropy of the adversarial agent’s policy π, controlled by a constant λ (λ ≥ 0), which serves as a policy regularization term in the loss function.
[0047] Step 4: Perform adversarial training on the target agent based on the state generated by the adversarial agent, and then optimize the parameters of the deep Q network to achieve optimized defense for the deep reinforcement learning strategy.
[0048] In an embodiment, when adversarial training is performed on a target agent based on the state generated by the adversarial agent, when the target agent generates N actions based on the parameter-optimized main Q network, that is, after taking N steps of behavioral actions, the adversarial agent generates an action at time N+1 based on the stored state, and generates and stores the corresponding state value. The target Q network of the target agent calculates the target Q value based on the state value at time N+1 generated by the adversarial agent to optimize the parameters of the main Q network.
[0049] In this way, the deep Q network obtained by optimizing the defense through deep learning deep reinforcement learning strategy can be used to defend against attacks in an autonomous driving environment, that is, the target agent can take correct actions under attack.
[0050] In the embodiment, the adversarial game training process of the target agent is as follows:
[0051] (a) The target agent and the adversarial agent take actions alternately to perform adversarial game training on the pre-trained model. That is, after the target agent takes N steps of action, the adversarial agent takes one action and outputs the action according to the learned strategy to obtain the next state as the strategy perturbation result.
[0052] (b) The adversarial agent continuously learns through the state of the target agent and the action with the smallest Q value, which increases the probability of the target agent's wrong strategy action in response to adversarial perturbations, thereby optimizing the target agent's strategy and enhancing the robustness of the model.
[0053] (c) The target agent samples N training data sets from Buff, updates the network parameters of the current Q network by minimizing the mean square error between the predicted Q value of the current Q network and the target Q value of the target Q network, and copies the parameters of the current Q network to the target Q network at regular intervals.
[0054] An embodiment also provides a deep reinforcement learning strategy optimization defense device based on imitation learning, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. When the computer processor executes the computer program, the above-mentioned deep reinforcement learning strategy optimization defense method based on imitation learning is implemented.
[0055] In practical applications, the computer memory can be a volatile memory at the near end, such as RAM, or a non-volatile memory, such as ROM, FLASH, floppy disk, mechanical hard disk, etc., or a remote storage cloud. The computer processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), that is, these processors can be used to implement the deep reinforcement learning strategy optimization defense method steps based on imitation learning.
[0056] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A deep reinforcement learning strategy optimization defense method based on imitation learning, characterized in that: The following steps are involved: Build a deep reinforcement learning intelligent agent autonomous driving simulation environment, build a target intelligent agent based on the deep Q network in reinforcement learning, and perform reinforcement learning on the target intelligent agent to optimize the parameters of the deep Q network, wherein the deep Q network includes a main Q network and a target Q network, wherein the main Q network serves as the decision network of the target intelligent agent, and the target Q network serves as the value network of the target intelligent agent. When performing reinforcement learning on the target intelligent agent, the main Q network uses the decision Q value calculated according to the state as the action value, and generates the state value and reward value at the next moment according to the action value. The state value, action value and reward value are all stored, and the target Q network takes out the target Q value calculated by the environment at the next moment from the storage, and updates the main Q network with the decision Q value and target Q value output by the main Q network and the target Q network and the accumulated reward value. At the same time, the current main Q network is copied to the target Q network at regular intervals, and the mean square error of the decision Q value and the target Q value containing the accumulated reward value is used as the loss function to update the main Q network; A deep Q network with optimized parameters is used to generate a sequence of state-action pairs of the target agent driving at T moments as expert data, where the action value in the state-action pair corresponds to the action with the smallest Q value; An adversarial agent is constructed based on a generative adversarial network, and imitation learning is performed on the adversarial agent, that is, the state in the expert data is used as the input of the generative adversarial network, and the expert data is used as a label to supervise and optimize the parameters of the generative adversarial network, wherein the generative adversarial network includes a generator and a discriminator, wherein the generator serves as the decision network of the adversarial agent, and the discriminator serves as the value network of the adversarial agent. When reinforcement learning is performed on the adversarial agent, the state value of each state-action pair in the expert data is used as the input of the generator, and the generator calculates and generates a predicted action value based on the input state value. The state-action pair composed of the predicted action value and the corresponding input state value is input to the discriminator, and the state-action pair with the same state value in the expert data is also input to the discriminator as a label. The discriminator is used to judge the action value generated by the generator, and the judgment result is used as a reward value to guide the learning of the generator; Based on the state generated by the adversarial agent, the target agent is trained adversarially, and then the parameters of the deep Q network are optimized to achieve deep reinforcement learning strategy optimization defense. When the target agent is trained adversarially based on the state generated by the adversarial agent, when the target agent generates N actions based on the parameter-optimized main Q network, that is, after taking N steps of behavioral actions, the adversarial agent generates an action at time N+1 based on the stored state, and generates and stores the corresponding state value. The target Q network of the target agent calculates the target Q value based on the state value at time N+1 generated by the adversarial agent to optimize the parameters of the main Q network.
2. The deep reinforcement learning strategy optimization defense method based on imitation learning as claimed in claim 1, characterized in that: When performing reinforcement learning on an adversarial agent, the loss function shown in formula (1) is used to update the parameters of the discriminator: Among them, D represents the discriminator, E π [logD(s,a)] represents the ability value calculated by the discriminator based on the state action value generated by the generator. represents the energy value calculated by the discriminator based on the state-action pair (s, a) in the expert data, D∈(0,1)S×A represents the output of the discriminator as a decimal between 0 and 1 with a size of S×A, is the entropy of the adversarial agent’s policy π, controlled by a constant λ, λ ≥ 0, which serves as a policy regularization term in the loss function.
3. A deep reinforcement learning strategy optimization defense device based on imitation learning, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: When the computer processor executes the computer program, the deep reinforcement learning strategy optimization defense method based on imitation learning described in claim 1 or 2 is implemented.