Deep reinforcement learning agent selection attack method based on trajectory approximation

Abstract

The present invention discloses a deep reinforcement learning agent selection attack method based on trajectory approximation. t by K The attack trajectory formed by all attacks in the first step is the best trajectory. K The attack trajectory formed by no attack is the worst trajectory. By approaching the best trajectory while avoiding the worst trajectory, N The attack strategy with the best attack perturbation distribution is selected from the random attack strategies for subsequent K The present invention achieves the approximation of the attack trajectory of the entire round in the reinforcement learning environment through the attack trajectory planning of each step, maximizes the probability of the attack trajectory with the minimum cumulative reward, and effectively improves the attack effect at a lower attack frequency while maintaining the attack effect at a higher attack frequency.

Description

Technical Field

[0001] The invention belongs to the field of artificial intelligence security technology, and specifically relates to a deep reinforcement learning agent selection attack method based on trajectory approximation. Background Art

[0002] Deep reinforcement learning is one of the emerging technologies in the field of artificial intelligence. It combines the powerful feature extraction capabilities of deep learning with the decision-making capabilities of reinforcement learning to achieve an end-to-end framework from perception input to decision output. It has strong learning capabilities and is widely used. However, studies have shown that deep reinforcement learning has security vulnerabilities and is vulnerable to adversarial sample attacks. Therefore, studying the weaknesses of deep reinforcement learning systems through adversarial attack techniques has become a necessary prerequisite for building robust deep reinforcement learning systems.

[0003] Compared with traditional deep learning systems, deep reinforcement learning systems are characterized by long sequence decisions rather than one-step decisions, so attackers must launch multi-step attacks on them. However, too many attacks can easily arouse suspicion from the target intelligent agent and expose the attacker's identity. The attacker should choose some specific key time steps to launch adversarial attacks.

[0004] There are some selective attack methods, which attack by selecting some key time steps and achieve the same attack effect as general attack methods, while reducing the possibility of being discovered by the target agent. However, some of them lack versatility and cannot attack agents in reinforcement learning environments with continuous action spaces or have poor attack effects at low attack frequencies. Summary of the invention

[0005] The technical problem solved by the present invention is to provide a deep reinforcement learning agent selection attack method based on trajectory approximation with good attack effect.

[0006] Technical solution: In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] A deep reinforcement learning agent selection attack method based on trajectory approximation, in time step t by K The attack trajectory formed by all attacks in the first step is the best trajectory. K The attack trajectory formed by no attack is the worst trajectory. By approaching the best trajectory while avoiding the worst trajectory, N The attack strategy with the best attack perturbation distribution is selected from the random attack strategies for subsequent K The specific steps are as follows:

[0008] Step 1: According to the time step t hour, t%KIf it is 0, it determines whether attack trajectory planning is required at the current time step. If attack trajectory planning is required, predict the best attack trajectory and the worst attack trajectory respectively. t+K The state value at , and then proceed to the subsequent steps 2, 3, 4, otherwise, jump to step 43, t%K yes t Divide by K The remainder of

[0009] Step 2: Generate N random attack strategies, and predict for each attack strategy that under the attack of this attack strategy, the environment at the time step t+K The status value of

[0010] Step 3: Calculate each attack strategy at time step t+K The trajectory approximation value at ;

[0011] Step 4: Use the calculation result of step 3 as the screening index, and select N Select the best attack strategy from the attack strategies and use this best attack strategy to perform time step t Follow-up K Step attack.

[0012] Preferably, the specific implementation steps of step 1 are as follows:

[0013] Step 11: First determine the time step t hour, t%K Is it 0? t%K A value of 0 means that at the time step t Need to follow up K Step 12 and 13 are performed to plan the attack trajectory of the next step, otherwise jump to step 43. t%K yes t Divide by K The remainder of

[0014] Step 12: If planning is required, predict the next K When the target agent is attacked at each step of the test environment, t+K Status value State_all ;

[0015] Step 13: Predictions in the future K When the target agent is not attacked at each step of the step, the environment is tested at time step t+K Status value State_no .

[0016] Preferably, the specific implementation steps of step 2 are as follows:

[0017] Step 21: Generate using random method N*20 A random list is randomly sampled from it N A list containing only 0 or 1. N A random list as N Attack strategies;

[0018] Step 22: Predict the attack strategy for each attack strategy by accessing the test environment and the target agent’s policy network. Strategy_i Under the attack of t+K Status value State_i, 0≤i ≤N-1；

[0019] As a preferred method, in step 3, the time step of each attack strategy is calculated. t+K Status value State_i and State_ all Distance dis_all_i ,as well as State_i and State_no Distance dis_no_i , and calculate the trajectory approximation value dis_ relative_i .

[0020] As a preferred method, the calculation process is as follows: State_i - State_all Calculate the vector two norm and get dis_ all_i ,right State_i - State_no Calculate the vector two norm and get dis_no_i , in the calculation dis_relative_i Before, for all dis_no_i Add 0.00000001, and then calculate dis_relative_i = dis_all_i / dis_no_i .

[0021] Preferably, the specific implementation steps of step 4 are as follows:

[0022] Step 41: dis_relative_i To filter the indicators, N Select the trajectory approximation value among the attack strategies dis_ relative_i The minimum attack strategy is used as the optimal attack strategy;

[0023] Step 42: Overwrite the old best attack strategy with the best attack strategy obtained by planning;

[0024] Step 43: Take the current best attack strategy t%K The decision value at attack_or_not If it is 0, do not attack, if it is 1, attack. Time step tAdd 1. If the end of this round is reached, the attack of this round ends. Otherwise, return to step 11.

[0025] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0026] (1) The present invention uses the trajectory approximation method to t ,by K The attack trajectory formed by all attacks in the first step is the best trajectory. K The attack trajectory formed by no attack is the worst trajectory. By approaching the best trajectory while avoiding the worst trajectory, N The attack strategy with the best attack perturbation distribution is selected from the random attack strategies. K The attack trajectory planning of the step-by-step approach can approximate the attack trajectory of the entire round in the reinforcement learning environment, maximize the probability of the attack trajectory with the minimum cumulative reward, and is suitable for environments with discrete action spaces and deterministic continuous action spaces.

[0027] (2) While maintaining the attack effect at a higher attack frequency, the attack effect at a lower attack frequency is effectively improved. It is superior to the existing selective attack method at a lower attack frequency and achieves a relatively good attack effect;

[0028] (3) The effect of selective attack can increase steadily with the increase of attack frequency, with little fluctuation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the structure of the deep reinforcement learning agent selection attack method based on trajectory approximation;

[0030] Figure 2 The attack effect of the present invention and the Strategically-Timed selection attack method on the DQN agent and the PPO agent in the Atari environment;

[0031] Figure 3 This is the attack effect of the present invention and the Tentative Frame selection attack method on the PPO agent in the MuJoCo environment. DETAILED DESCRIPTION

[0032] The present invention is further illustrated below in conjunction with specific examples. The examples are implemented based on the technical solutions of the present invention. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0033] like Figure 1 As shown, the deep reinforcement learning agent selection attack method based on trajectory approximation in this application is t by KThe attack trajectory formed by all attacks in the first step is the best trajectory. K The attack trajectory formed by no attack is the worst trajectory. By approaching the best trajectory while avoiding the worst trajectory, N The attack strategy with the best attack perturbation distribution is selected from the random attack strategies for subsequent K The attack is a step-by-step process. It includes four steps: Step 1 to Step 4:

[0034] Step 1: According to the time step t hour, t%K If it is 0, it determines whether attack trajectory planning is required at the current time step. If attack trajectory planning is required, predict the best attack trajectory and the worst attack trajectory respectively. t+K The state value at , and then proceed to the subsequent steps 2, 3, 4, otherwise, jump to step 43, t%K yes t Divide by K The remainder of

[0035] Step 11: First determine the time step t hour, t%K Is it 0? t%K A value of 0 means that at the time step t Need to follow up K Step 12 and 13 are performed to plan the attack trajectory of the next step, otherwise jump to step 43;

[0036] Step 12: If planning is required, predict the next K When the target agent is attacked at each step of the test environment, t+K Status value State_all ;

[0037] Step 13: Predictions in the future K When the target agent is not attacked at each step of the step, the environment is tested at time step t+K Status value State_no ;

[0038] In the Atari environment, t+K The state value of State_all and State_no The current actual environment state is taken from State_all and State_no The feature vector output by the previous layer of the logits layer after being input into the target agent policy network.

[0039] In the MuJoCo environment, the actual environment state vector is directly taken from the environment. The attack method for the Atari environment uses the FGSM method, and the attack method for the MuJoCo environment uses the optimal attack method.

[0040] Step 2: Generate N random attack strategies, and predict for each attack strategy that under the attack of this attack strategy, the environment at the time step t+K Status value:

[0041] Step 21: Generate using random method N*20 A random list is randomly sampled from it N A list containing only 0 or 1. N A random list as N Attack strategies;

[0042] Step 22: Predict the attack strategy for each attack strategy by accessing the test environment and the target agent’s policy network. Strategy_i Under the attack of t+K Status value State_i, 0≤i ≤N-1；

[0043] In the Atari environment, t+K The state value of State_i (0≤i≤N - 1) All are taken from the current actual environment state State_i The feature vector of the previous layer output of the logits layer after being input into the target agent's policy network. In the MuJoCo environment, it is directly taken from the actual environment state vector of the environment.

[0044] Step 3: Calculate each attack strategy at time step t+K Status value State_i and State_all Distance dis_ all_i ,as well as State_i and State_no Distance dis_no_i , and calculate the trajectory approximation value dis_relative_i :The calculation process is as follows:

[0045] right State_i - State_all Calculate the vector two norm and get dis_all_i ,right State_i - State_no Calculate the vector two norm and get dis_no_i , in the calculation dis_relative_i Before, for all dis_no_i Add 0.00000001, and then calculate dis_relative_i = dis_all_i / dis_no_i .

[0046] Step 4: dis_relative_i To filter the indicators, N Select the trajectory approximation value among the attack strategies dis_ relative_i Minimal attack strategy as optimal attack strategy:

[0047] Step 41: dis_relative_i To filter the indicators, N Select the trajectory approximation value among the attack strategies dis_ relative_i The minimum attack strategy is used as the optimal attack strategy;

[0048] Step 42: Overwrite the old best attack strategy with the best attack strategy obtained by planning;

[0049] Step 43: Take the current best attack strategy t%K The decision value at attack_or_not If it is 0, do not attack, if it is 1, attack. Time step t Add 1. If you reach the end of this round, the attack ends. Otherwise, return to step 11.

[0050] The effectiveness and efficiency of the method of the present invention are verified by the following experiments:

[0051] • Select attack effect - For the attack effect of the agent in deep reinforcement learning, the average reward of the agent over multiple rounds in the standard reinforcement learning environment is used as the indicator.

[0052] The proposed method was verified in the Atari environment and MuJoCo environment in the Gym series of reinforcement learning environments from OpenAI. The Atari environment contains a series of environments with discrete action spaces commonly used in the field of deep reinforcement learning attacks, and the MuJoCo environment contains a series of environments with continuous action spaces commonly used in the field of deep reinforcement learning attacks. Three representative environments were selected from the Atari environment: MsPacman-v4, Qbert-v4, and Seaquest-v4, and four representative environments were selected from the MuJoCo environment: Ant-v2, HalfCheetah-v2, Hopper-v2, and Walker-v2. It is worth noting that in the Atari environment, two agents with different performances are trained using the DQN algorithm and the PPO algorithm for each environment, and the agent trained by the PPO algorithm is usually stronger. In the MuJoCo environment, only one agent is trained using the PPO algorithm for each environment. In the Atari environment, the average reward of 20 rounds is used as the final reward. In the MuJoCo environment, the average reward of 50 rounds is used as the final reward.

[0053] like Figure 2 As shown, in Figure 2 The Strategically-Timed selection attack method is referred to as STA, and the method of the present invention is referred to as TA-relative.

[0054] Figure 2 Figures (a), (b), and (c) show the attack effects of the STA method and the TA-relative attack method on the DQN agents in the three Atari environments. It can be seen that the TA-relative attack method is significantly better than the STA method at lower attack frequencies in the DQN-Qbert and DQN-Seaquest environments, and they are basically the same at higher attack frequencies. In the DQN-MsPacman environment, the TA-relative attack method is slightly worse than the STA method.

[0055] Figure 2 (d), (e) and (f) in the figure show the attack effects of the STA method and the TA-relative attack method on the PPO agents in the three Atari environments. It should be noted that the attack effect of the STA method is not given in the original text. The present invention tests its attack effect using the attack method given by the STA method under the same experimental conditions. It can be seen that in the PPO-Qbert environment, the TA-relative attack method achieves significantly better results than the STA method at a lower attack frequency, while they are basically consistent at a higher attack frequency. In the PPO-MsPacman environment, the attack effect of the TA-relative attack method is slightly lower than that of STA at a low attack frequency, and its attack effect is better than that of STA at a higher attack frequency. In the PPO-Seaquest environment, the attack effect is slightly worse than that of STA.

[0056] like Figure 3 As shown, in Figure 2 The Tentative Frame selection attack method is referred to as Tentative, and the method of the present invention is referred to as TA-relative.

[0057] Figure 3(a), (b), (c), and (d) in the figure show the attack effects of the Tentative method and the TA-relative attack method on the PPO agent in four MuJoCo environments. It can be seen that in the four environments with continuous action spaces, the TA-relative attack method can achieve better attack effects at lower attack frequencies, and the trend is stable, which is significantly better than the Tentative attack method. At higher attack frequencies, the two are basically the same.

[0058] In general, the present invention uses a trajectory approximation method to t ,by K The attack trajectory formed by all attacks in the first step is the best trajectory. K The attack trajectory formed by no attack is the worst trajectory. By approaching the best trajectory while avoiding the worst trajectory, N The attack strategy with the best attack perturbation distribution is selected from the random attack strategies. K The attack trajectory planning of the step achieves the approximation of the attack trajectory of the entire round in the reinforcement learning environment, maximizes the probability of the attack trajectory with the minimum cumulative reward, and effectively improves the attack effect at a lower attack frequency while maintaining the attack effect at a higher attack frequency. The experimental results show that the selective attack method proposed in the present invention performs better than other selective attack methods and has universality.

[0059] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A deep reinforcement learning agent selection attack method based on trajectory approximation, applied in the field of artificial intelligence security, characterized by: We use adversarial attack techniques to study the weaknesses of deep reinforcement learning systems. We use adversarial attack techniques to study the weaknesses of deep reinforcement learning systems in terms of long sequence decision-making. t by K The attack trajectory formed by all attacks in the first step is the best trajectory. K The attack trajectory formed by no attack is the worst trajectory. By approaching the best trajectory while avoiding the worst trajectory, N The attack strategy with the best attack perturbation distribution is selected from the random attack strategies for subsequent K Step attack; The specific steps are as follows: Step 1: According to the time step t hour, t%K If it is 0, it determines whether the current time step needs to perform attack trajectory planning. If attack trajectory planning is required, it predicts the best attack trajectory and the worst attack trajectory respectively. t+K The state value at , and then proceed to the subsequent steps 2, 3, 4, otherwise, jump to step 43, t%K yes t Divide by K The specific implementation steps are as follows: Step 11: First determine the time step t hour, t%K Is it 0? t%K A value of 0 means that at the time step t Need to follow up K Step 12 and 13 are performed to plan the attack trajectory of the next step, otherwise jump to step 43. t%K yes t Divide by K The remainder of Step 12: If planning is required, predict the next K When the target agent is attacked at each step of the test environment, t+K Status value State_all ; Step 13: Predictions in the future K When the target agent is not attacked at each step of the step, the environment is tested at time step t+K Status value State_no ; Step 2: Generate N random attack strategies, and predict for each attack strategy that under the attack of this attack strategy, the environment at the time step t+K The specific implementation steps are as follows: Step 21: Generate using random method N*20 A random list is randomly sampled from it N A list containing only 0 or 1. N A random list as N Attack strategies; Step 22: Predict the attack strategy for each attack strategy by accessing the test environment and the target agent’s policy network. Strategy_i Under the attack of t+K Status value State_i, 0≤i≤N- 1； Step 3: Calculate each attack strategy at time step t+K The trajectory approximation value at ; Calculate each attack strategy at time step t+K Status value State_i and State_all Distance dis_all_i ,as well as State_i and State_no Distance dis_no_i , and calculate the trajectory approximation value dis_relative_i ; The calculation process is as follows: State_i - State_all Calculate the vector two norm and get dis_all_i ,right State_i - State_no Calculate the vector two norm and get dis_no_i , in the calculation dis_relative_i Before, for all dis_no_i Add 0.00000001, and then calculate dis_relative_i=dis_all_i / dis_no_i; Step 4: Use the calculation result of step 3 as the screening index, and select N Select the best attack strategy from the attack strategies and use this best attack strategy to perform time step t Follow-up K The specific implementation steps are as follows: Step 41: dis_relative_i To filter the indicators, N Select the trajectory approximation value among the attack strategies dis_ relative_i The minimum attack strategy is used as the optimal attack strategy; Step 42: Overwrite the old best attack strategy with the best attack strategy obtained by planning; Step 43: Take the current best attack strategy t%K The decision value at attack_or_not If it is 0, do not attack; if it is 1, attack; time step t Add 1. If the end of this round is reached, the attack of this round ends. Otherwise, return to step 11.

Images