A deep reinforcement learning data enhancement defense method and device based on SeqGAN

Abstract

The present invention discloses a deep reinforcement learning data enhancement defense method and device based on SeqGAN. Firstly, a target intelligent agent is pre-trained by using a DQN algorithm, and then a plurality of sequence state-action pairs at T moments are sampled as real sequence trajectory data; secondly, the initial state-action pairs are input into a generator, and the generator is used to generate sequence state strategy trajectory data, and a discriminator and a Monte Carlo method based on policy gradient are used to evaluate the rewards obtained by the generated sequence, which are used to guide the training of the generator, so as to generate state-action data close to the real state; finally, the training data is enhanced by comparing the cumulative reward value of the sequence obtained by the model strategy and the cumulative reward value obtained by the state strategy generated by SeqGAN, so as to optimize the strategy of the model and improve the robustness of the model.

Description

Technical Field

[0001] The present invention belongs to the field of defense for deep reinforcement learning, and specifically relates to a deep reinforcement learning data enhancement defense method and device based on SeqGAN. Background Art

[0002] Deep reinforcement learning is one of the areas of artificial intelligence that has received much attention in recent years. With the rapid development and application of reinforcement learning, reinforcement learning has been widely used in robot control, game gaming, computer vision, unmanned driving and other fields. In order to ensure the safe application of deep reinforcement learning in safety-critical fields, the key is to analyze and discover loopholes in deep reinforcement learning algorithms and models to prevent people with ulterior motives from exploiting these loopholes to make illegal profits. Unlike the single-step prediction task of traditional machine learning, the deep reinforcement learning system requires multi-step decision-making to complete a task, and there is a high correlation between consecutive decisions.

[0003] Reinforcement learning is usually a continuous decision-making process. Its basic idea is to learn the optimal strategy to achieve the learning purpose by maximizing the cumulative rewards obtained by the agent from the environment. Deep reinforcement learning makes full use of neural networks as parameter structures, combines the perception ability of deep learning and the decision-making ability of reinforcement learning to optimize deep reinforcement learning strategies, and finally realizes an end-to-end deep reinforcement learning framework from perception input to decision output. It has strong learning ability and is widely used. However, similar to machine learning models, reinforcement learning is also susceptible to interference from adversarial samples. According to the attack on the sample, it can be mainly divided into observation attack, reward attack, action attack and environment attack. At the same time, the training process of deep reinforcement learning will also be subject to poisoning attacks, which increases the training cost of the model and reduces the performance of the model. In addition, the strategy of deep reinforcement learning training also has security risks, especially in safety-critical fields, such as complex combat environments, combat command environments, drone armored vehicles, drone monitoring machines, intelligent robot control, etc. The existence of model strategy vulnerabilities will cause serious damage to the reinforcement learning system. Especially in safety-critical fields, this security risk brings great harm and will cause the decision-making of the reinforcement learning system to be wrong, which is a major challenge for the decision-making safety application field of reinforcement learning.

[0004] Existing studies have shown that policy poisoning attacks can change decisions by changing the data in the training set, so that the agent will never make mistakes in action selection, and the agent will not achieve the learning purpose in the end. This attack is very fatal to applications in the field of safety decision-making such as unmanned driving. At present, according to the existing defense mechanisms, common reinforcement learning defense methods can be divided into three categories: adversarial training, robust learning, and adversarial detection. Adversarial training refers to training the model by adding adversarial samples to the training samples. Its main purpose is to improve the generalization ability of the strategy beyond normal samples. However, adversarial training can often only improve the strategy's ability to fit the samples involved in the training. Robust learning is a learning mechanism that improves the robustness of the training model when facing attack methods from the training stage or the test stage. Adversarial detection refers to the model distinguishing between normal samples and adversarial samples, and processing adversarial samples without modifying the original model parameters to achieve a defensive effect. Summary of the invention

[0005] In view of the security threat problem caused by deep reinforcement learning in the field of safety decision-making (such as autonomous driving scenarios) due to its susceptibility to noise disturbance attacks, the purpose of the present invention is to provide a deep reinforcement learning data enhancement defense method and device based on SeqGAN. The deep reinforcement learning model is optimized by data enhancement to improve the robustness of the deep reinforcement learning model to defend against attacks.

[0006] In order to achieve the above-mentioned invention object, the present invention provides the following technical solutions:

[0007] In the first aspect, a deep reinforcement learning data enhancement defense method based on SeqGAN includes the following steps:

[0008] Build a deep reinforcement learning intelligent agent autonomous driving simulation environment, construct a target intelligent agent based on the deep Q network in reinforcement learning, and perform reinforcement learning on the target intelligent agent to optimize the parameters of the deep Q network;

[0009] A deep Q network with optimized parameters is used to generate a sequence of state-action pairs of the target agent driving at T moments as expert data, where the action value in the state-action pair corresponds to the action with the smallest Q value;

[0010] The SeqGAN, which consists of a generator and a discriminator, is trained using reinforcement learning. The state-action pairs in the expert data are used as the input of the generator to generate state-action pairs. At the same time, a policy gradient Monte Carlo search is used to simulate sampling. The state-action pairs obtained by sampling and the state-action pairs generated by the generator form a state-action pair sequence of fixed length and are input to the discriminator. The reward value is calculated and the network parameters of SeqGAN are updated according to the reward value.

[0011] Input the current state into the generator of the parameter-optimized SeqGAN to obtain a sequence of generated state-action pairs, use the parameter-optimized deep Q network to calculate the cumulative reward value of the generated state-action pair sequence, compare the cumulative reward value with the cumulative reward value obtained by the deep Q network strategy of the target agent, and use the state-action pair with a higher cumulative reward value as enhanced data for re-optimization of the deep Q network;

[0012] Select augmented data from storage to re-optimize parameters of the deep Q-network to achieve data augmentation defense for deep reinforcement learning.

[0013] In the second aspect, a deep reinforcement learning data enhancement defense device based on SeqGAN includes a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, and the computer processor implements the above-mentioned deep reinforcement learning data enhancement defense method based on SeqGAN when executing the computer program.

[0014] Compared with the prior art, the SeqGAN-based deep reinforcement learning data enhancement defense method and device of the present invention have at least the following beneficial effects:

[0015] 1) Generate sequence states and policy actions by training SeqGAN to enhance training data to optimize the target agent strategy and improve the robustness of the DRL model; 2) During the SeqGAN training process, the generator is used to generate sequence state and policy action trajectory data, and the real data input of the discriminator is the sequence state action pair data sampled from the pre-trained DQN model. During the training process, the parameters of the generator and discriminator are updated to generate more realistic sequence state policy data; 3) The training data is enhanced by comparing the cumulative reward value of the sequence obtained by the model strategy and the cumulative reward value obtained by the state strategy generated by SeqGAN to optimize the model's strategy and improve the robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 is a flowchart of a deep reinforcement learning data enhancement defense method based on SeqGAN provided in an embodiment;

[0018] Figure 2 Schematic diagram of a sequence data generation process based on SeqGAN training provided in an embodiment;

[0019] Figure 3 Schematic diagram of the DQN algorithm structure in the chemical learning provided by the embodiment. DETAILED DESCRIPTION

[0020] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0021] In the deep reinforcement learning training process of simulating the autonomous driving of a small car, the attack method based on poisoning of the training data set will cause the learner to learn a wrong strategy, thereby choosing a bad action, causing the learner to learn incorrectly. Based on this situation, the embodiment provides a deep reinforcement learning data enhancement defense method and device based on SeqGAN, which uses SeqGAN to generate sequence state action data for training data enhancement to achieve the purpose of optimizing the target agent strategy. The specific process is: first, the target agent is pre-trained using the DQN algorithm, and then multiple sequence state action pairs at time T are sampled as real sequence trajectory data; secondly, the initial state action pair is input into the generator and the generator is used to generate sequence state strategy trajectory data, and the discriminator and the Monte Carlo method based on the policy gradient are used to evaluate the reward obtained by the generated sequence, which is used to guide the training of the generator to generate state action data close to the real state; finally, the training data is enhanced by comparing the cumulative reward value of the sequence obtained by the model strategy and the cumulative reward value obtained by the state strategy generated by SeqGAN, so as to optimize the strategy of the model and improve the robustness of the model.

[0022] Figure 1 : is a flowchart of a deep reinforcement learning data enhancement defense method based on SeqGAN provided in an embodiment. Figure 1 As shown, the deep reinforcement learning data enhancement defense method provided by the embodiment includes the following steps:

[0023] Step 1: Build a deep reinforcement learning intelligent agent autonomous driving simulation environment, construct a target intelligent agent based on the deep Q network (DQN) in reinforcement learning, and perform reinforcement learning on the target intelligent agent to optimize the parameters of the deep Q network.

[0024] In the embodiment, the intelligent agent can be a car in an autonomous driving environment, and the car is trained using a deep Q network in reinforcement learning, with the goal of making the car reach the destination as quickly as possible. The deep Q network combines Q learning and convolutional neural networks to construct a training model for deep reinforcement learning.

[0025] Among them, Figure 3As shown in the figure, the deep Q network includes a main Q network and a target Q network, wherein the main Q network is used as the decision network of the target intelligent agent, and the target Q network is used as the value network of the target intelligent agent. Both the main Q network and the target Q network use convolutional neural networks. When the target intelligent agent is reinforced, the main Q network uses the decision Q value calculated according to the state as the action value, and generates the state value and reward value of the next moment according to the action value. The state value, action value and reward value are all stored. The target Q network takes the target Q value calculated by the environment at the next moment from the storage, and updates the main Q network with the decision Q value and target Q value output by the main Q network and the target Q network, as well as the accumulated reward value, and copies the current main Q network to the target Q network at regular intervals.

[0026] In the embodiment, DQN not only solves the problem of too large a state space to maintain by combining deep neural network with the Q learning algorithm of reinforcement learning, but also has a much greater potential than artificial feature representation due to the powerful feature extraction capability of neural network. The Q learning in reinforcement learning uses the Bellman equation to iteratively update the state-action value function Q by using the method of temporal difference:

[0027] Q i+1 (s t ,a t )=Q i (s t ,a t )+α(y i -Q i (s t ,a t ))

[0028] in, is the target Q value, s t+1 As an action t The next state that appears, a t+1 Yes t+1 The possible actions under the state. α is the learning rate and γ is the discount factor. According to the Bellman optimal equation theory, as long as the above formula is continuously updated iteratively, the Q function can be approximated to the true value Q * , thus finally obtaining the optimal strategy:

[0029] DQN also uses a target network mechanism, that is, in the current Q θ Based on the network structure, a target with exactly the same structure was built The network constitutes the overall model framework of DQN. During the training process, the current Q θ The predicted Q value output by the network is used to select action a, and another target The network is used to calculate the target Q value. The mean square error between the decision Q value and the target Q value including the cumulative reward value is used as the loss function to update the main Q network. Loss function:

[0030]

[0031] in, is the target Q value, and the current Q is updated through the reverse gradient propagation of the neural network θ The parameters of the network are θ.

[0032] During the training process, DQN uses the experience replay mechanism to convert the state transition process (state s i 、Action a i , Reward i , the next state s i ') is stored in the experience playback buffer Buff as the training data set of the network model, and batch learning is performed in the form of random sampling;

[0033] Sample N training data sets from Buff and update the current Q by minimizing the loss function θ The network parameters of the network, for the target Q θ- The network parameters do not need to be updated iteratively, but are updated from the current Q θ The network parameters are copied to the network, that is, delayed update, and then the next round of learning is carried out. This method reduces the impact of each Q value change on the policy parameters, that is, reduces the correlation between the target Q value and the predicted Q value, and increases the stability of policy training.

[0034] Step 2: Use the parameter-optimized deep Q network to generate a sequence of state-action pairs of the target agent driving at T moments as expert data, where the action value in the state-action pair corresponds to the action with the smallest Q value.

[0035] The process of generating state-action pair sequences is a sampling process, using the T-time car driving sequence state-action pairs {(s1, a1), ..., (s T ,a T )} as expert data, where action a is the action with the smallest corresponding decision Q value, indicating that the action is a wrong action, where s represents the state value.

[0036] Step 3: Use reinforcement learning methods to train SeqGAN, which includes the generator and the discriminator.

[0037] In an embodiment, when performing parameter optimization on a SeqGAN including multiple generators and discriminators, the state-action pairs in the expert data are used as inputs of the generator, and the generator is used to generate state-action pairs according to the input state-action pairs. At the same time, a Monte Carlo search based on a policy gradient is used to simulate sampling to obtain some state-action pairs. These state-action pairs and the state-action pairs generated by the generator form a generated state-action pair sequence of a fixed length. The generated state-action pair sequence is input to the discriminator, and the discriminator is used to calculate a reward value according to the input generated state-action pair sequence.

[0038] A fixed-length real state-action pair sequence is sampled online from the storage and input into the discriminator. The reward value of the real state-action pair sequence is calculated, and the cross entropy of the reward value and the reward value of the generated state-action pair sequence is used as the loss function to update the parameters of the discriminator and the generator.

[0039] Specifically, Figure 2 As shown, the training process of SeqGAN is:

[0040] The goal of the generator is to generate a sequence data Y 1:T =(y1,...,y t ,...,y T ), so that the discriminator D with parameter φ φ The sequence data is considered to be real sequence data, and its generation strategy is represented by G θ Among them, the sequence data Y 1：T It refers to the sequence state strategy action data, that is, ((s1',a1'),...,(s t ',a t '),...,(s' T ,a' T )). The reinforcement learning algorithm is used to guide the training of SeqGAN. The state state represents the generated state-action sequence ((s1',a1'),...,(s t ' -1 ,a t ' -1 ))=(y1,...,y t-1 ), action action represents the state action pair data y generated by the next selection t =(s t ',a t '), the reward value reward is used to evaluate the generated sequence;

[0041] When there is no intermediate reward value, the generator G θ (Y 1:t-1 ) aims to generate a sequence of state-action pairs with the highest reward value starting from the initial state-action pair:

[0042]

[0043] Among them, R T Represents the reward of the complete sequence. The Q function refers to the use of the generator G after selecting y1 in state (s0, a0). θ The cumulative reward obtained by the generation strategy is Used to evaluate the generated sequence data. However, since the discriminant can only get the reward of the complete sequence, in order to calculate the reward of the intermediate state, the Monte Carlo MC algorithm based on the policy gradient is used with the roll-out strategy G β (with the generator G θ The remaining Tt sequence state-action pairs are simulated and sampled N times. The purpose of sampling in roll-out is only to calculate the action value of the current state in order to enter the next state, and these estimated action values ​​will not be retained. The Q function can be obtained as follows:

[0044]

[0045] The purpose of GAN is to make the generated data close to the target data distribution (the distribution represented by the training set data), where the goal of the discriminator is to minimize the distribution of the two. The objective function can be expressed as:

[0046]

[0047] Among them, P data is the distribution of the sample sequence data, the logD in the first term φ (Y) represents the judgment of the discriminator on the real sequence data, and the second term log(1-D φ (Y)) represents the judgment of the generated sequence data, and the discriminator parameter φ is updated by reversely derivatizing the objective function;

[0048] During the training process, the discriminator is used to evaluate the generated sequence data to guide the training of the generator, and the parameters θ of the generator are updated by the policy gradient update method:

[0049]

[0050] Among them, α h ∈R + Represents the learning rate corresponding to the h-th step.

[0051] Step 4: Screen the enhanced data for re-optimization of the deep Q network based on the prediction results of the pre-trained deep Q network and the parameter-optimized SeqGAN.

[0052] Specifically, the current state is input into the generator G of the parameter-optimized SeqGAN to obtain a sequence of generated state-action pairs at T moments, which is then input into the DQN model. The cumulative reward value of the generated state-action pair sequence is calculated using the parameter-optimized deep Q network, and the cumulative reward value is compared with the cumulative reward value obtained by the deep Q network strategy of the target agent. The state-action pair with a higher cumulative reward value is used as the reinforcement data and stored in the experience buffer together with the obtained reward value as the training data of the model.

[0053] Among them, the process of obtaining the cumulative reward value obtained by the deep Q network strategy of the target intelligent agent is: taking the state-action pair sequence sampled online as the input of the deep Q network in sequence, and using the deep Q network to calculate the cumulative reward value obtained by the deep Q network strategy.

[0054] Step 5: Select enhanced data from the storage to re-optimize the parameters of the deep Q network to achieve deep reinforcement learning data enhancement defense.

[0055] Specifically, the target agent samples N training data sets from Buff, updates the network parameters of the current main Q network by minimizing the mean square error between the decision Q value of the current main Q network and the target Q value of the target Q network, and copies the parameters of the current main Q network to the target Q network at regular intervals. The model strategy is optimized through data enhancement, thereby improving the robustness of the model.

[0056] An embodiment also provides a deep reinforcement learning data enhancement defense device based on SeqGAN, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. When the computer processor executes the computer program, the above-mentioned deep reinforcement learning data enhancement defense method based on SeqGAN is implemented.

[0057] In practical applications, the computer memory can be a volatile memory at the near end, such as RAM, or a non-volatile memory, such as ROM, FLASH, floppy disk, mechanical hard disk, etc., or a remote storage cloud. The computer processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), that is, the deep reinforcement learning data enhancement defense method based on SeqGAN can be implemented through these processors.

[0058] The above-mentioned deep reinforcement learning data enhancement defense method and device based on SeqGAN are mainly used in scenarios where the reinforcement learning training process is attacked so that the training data set is changed. This method generates a training strategy based on a sequence adversarial generative network (SeqGAN), thereby optimizing the model using the generated sequence data to achieve the purpose of training data enhancement, thereby improving the robustness of the model.

[0059] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A deep reinforcement learning data enhancement defense method based on SeqGAN, The method comprises the following steps, characterized in that: The following steps are involved: Build a deep reinforcement learning intelligent agent autonomous driving simulation environment, construct a target intelligent agent based on the deep Q network in reinforcement learning, and perform reinforcement learning on the target intelligent agent to optimize the parameters of the deep Q network; A deep Q network with optimized parameters is used to generate a sequence of state-action pairs of the target agent driving at T moments as expert data, where the action value in the state-action pair corresponds to the action with the smallest Q value; The SeqGAN, which includes a generator and a discriminator, is trained using reinforcement learning. The state-action pairs in the expert data are used as the input of the generator to generate state-action pairs. At the same time, policy gradient Monte Carlo search is used to simulate sampling. The sampled state-action pairs and the state-action pairs generated by the generator form a state-action pair sequence of fixed length and are input to the discriminator to calculate the reward value. The fixed-length real state-action pair sequence is sampled online from the storage and input to the discriminator. The reward value of the real state-action pair sequence is calculated, and the cross entropy of the reward value and the reward value of the generated state-action pair sequence is used as the loss function to update the parameters of the discriminator and the generator. The current state is input into the generator of the parameter-optimized SeqGAN to obtain a sequence of generated state-action pairs, and the parameter-optimized deep Q network is used to calculate the cumulative reward value of the generated state-action pair sequence. The specific process is: the state-action pair sequence sampled online is used as the input of the deep Q network in sequence, and the deep Q network is used to calculate the cumulative reward value obtained by the deep Q network strategy, and the cumulative reward value is compared with the cumulative reward value obtained by the deep Q network strategy of the target agent. The state-action pair with a higher cumulative reward value is stored as enhanced data for re-optimization of the deep Q network, and the enhanced data is selected from the storage to re-optimize the parameters of the deep Q network to achieve deep reinforcement learning data enhancement defense.

2. The SeqGAN-based deep reinforcement learning data enhancement defense method according to claim 1, characterized in that: The deep Q network includes a main Q network and a target Q network, wherein the main Q network serves as the decision network of the target intelligent agent, and the target Q network serves as the value network of the target intelligent agent. When the target intelligent agent is subjected to reinforcement learning, the main Q network uses the decision Q value calculated according to the state as the action value, and generates the state value and reward value at the next moment according to the action value. The state value, action value and reward value are all stored, and the target Q network takes out the target Q value calculated by the environment at the next moment from the storage, and updates the main Q network with the decision Q value and target Q value output by the main Q network and the target Q network and the accumulated reward value, and at the same time copies the current main Q network to the target at regular intervals.

3. The SeqGAN-based deep reinforcement learning data enhancement defense method according to claim 2, characterized in that: The main Q network is updated using the mean square error of the decision Q value and the target Q value containing the cumulative reward value as the loss function.

4. The SeqGAN-based deep reinforcement learning data enhancement defense method according to claim 2, characterized in that: In the process of reinforcement learning, the deep Q network uses temporal difference to iteratively update the state-action value function Q: Q i+1 (s t ,a t )=Q i (s t ,a t )+α(y i -Q i (s t ,a t )) in, is the target Q value, s t+1 As an action t The next state that appears, i is the round index, a t+1 Yes t+1 The possible actions under the state, α is the learning rate, γ is the discount factor. According to the Bellman optimal equation theory, by continuously iteratively updating the above formula, the Q function can be approximated to the true value Q * , thus finally obtaining the optimal strategy: The loss function loss is defined as: in, is the target Q value, and the current Q is updated through the reverse gradient propagation of the neural network θ The parameters of the network are θ.

5. The SeqGAN-based deep reinforcement learning data enhancement defense method according to claim 1, characterized in that: When optimizing parameters of SeqGAN, which includes multiple generators and discriminators, the state-action pairs in the expert data are used as the input of the generator. The generator is used to generate state-action pairs based on the input state-action pairs. At the same time, policy gradient Monte Carlo search is used to simulate sampling to obtain some state-action pairs. These state-action pairs and the state-action pairs generated by the generator form a generated state-action pair sequence of fixed length. The generated state-action pair sequence is input to the discriminator. The discriminator is used to calculate the reward value based on the input generated state-action pair sequence.

6. The SeqGAN-based deep reinforcement learning data enhancement defense method according to claim 1, characterized in that: When training SeqGAN, the objective function used is: Among them, P data is the distribution of the sample sequence data, the logD in the first term φ (Y) represents the judgment of the discriminator on the real sequence data, and the second term log(1-D φ (Y)) represents the judgment of the generated sequence data, and the discriminator parameter φ is updated by reversely derivatizing the objective function; At the same time, the parameters θ of the generator are updated by the policy gradient update method: Among them, α h ∈R + represents the learning rate corresponding to the h-th step.

7. A deep reinforcement learning data enhancement defense device based on SeqGAN, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor, characterized in that: When the computer processor executes the computer program, the SeqGAN-based deep reinforcement learning data enhancement defense method according to any one of claims 1 to 6 is implemented.

Images