A method and device for generating attack strategies for intelligent agents

Abstract

The present invention discloses a method and device for generating an attack strategy for an intelligent agent, which obtains the state of the intelligent agent at the current moment; calculates the attack time selection value at the current moment based on the attack time selection function and the state; when the attack time selection value is greater than a first threshold, selects an attack action from an attack action library to attack the intelligent agent; the present invention utilizes the attack time selection function to significantly reduce the number of attacks, thereby ensuring the concealment of the attack; utilizes a trained attack action selection strategy to select a target action that reduces the cumulative reward the most to perform a target attack, thereby ensuring the effectiveness of the attack, effectively reducing the time complexity of the method, and thus improving the attack efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information technology, and in particular relates to a method and device for generating an attack strategy for an intelligent body. Background Art

[0002] In recent years, deep reinforcement learning (DRL) has been widely applied in fields such as digital games and robotic control, achieving promising results. As a research method in artificial intelligence, DRL combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning, creating an end-to-end perception and control system. However, its safety has become a major concern.

[0003] Currently, deep reinforcement learning is applied to many decision-making tasks that require high security and safety. It usually uses deep neural networks (DNNs) to optimize the action-value function. However, current research has shown that DNNs are vulnerable to attacks. Therefore, although deep reinforcement learning has shown good performance, it also has to consider the attacks it may face.

[0004] In deep reinforcement learning, the intelligent agent interacts with the environment through a series of operations, each of which changes the corresponding state. Compared with other forms of adversarial attacks, adversarial attacks for deep reinforcement learning need to consider the impact of actions on subsequent states and determine whether to attack in a certain state, thereby achieving the goal of reducing the total benefit or even putting the intelligent agent into a dangerous state. This is different from ordinary attack systems that aim to reduce classification accuracy.

[0005] Depending on the attack target, adversarial attacks can be mainly divided into the following five categories: attacks on observations, attacks on the environment, attacks on rewards, attacks on strategies, and attacks on actions.

[0006] An observation attack involves an attacker perturbing the observed image received by an agent, causing it to perform the desired action. This is typically achieved by adding noise to the agent's image sensor. However, existing observation attacks often struggle to choose the right attack time and action, resulting in low attack efficiency, unsatisfactory results, and a lack of guaranteed stealth. Summary of the Invention

[0007] The purpose of the present invention is to provide a method and device for generating an attack strategy for an intelligent agent. By selecting appropriate attack time and attack actions, the effectiveness and concealment of the attack are improved. It can be used to detect the robustness of the intelligent agent under deep reinforcement learning and ensure its security in application.

[0008] The present invention adopts the following technical solution: a method for generating an attack strategy for an intelligent agent, comprising the following steps:

[0009] Get the state of the agent at the current moment;

[0010] Calculate the attack time selection value at the current moment based on the attack time selection function and the state;

[0011] When the attack time selection value is greater than the first threshold, an attack action is selected from the attack action library to attack the agent;

[0012] Among them, the attack time selection function is C(t)=α(π max -π min )+β(π max -π avg ), C(t) is the attack time selection value, t represents the current time sequence number, α and β are constants, π max is the maximum value of the output action probability in a given state, π min is the minimum value of the output action probability in a given state, π avg is the average probability of outputting an action in a given state.

[0013] Furthermore, π max pass Calculated, where s t is the state of the agent at time t, a t For s t The corresponding action of the intelligent agent, T is a hyperparameter, a tk is the probability that the agent outputs action k at time t;

[0014] π min pass Calculated;

[0015] π avg pass Calculated.

[0016] Furthermore, when the attack time selection value is greater than the first threshold, the method further includes:

[0017] Predict the next action a′ after attacking the agent t+1 ;

[0018] Calculate the next action a when the agent is not attacked t+1 ;

[0019] Compare a t+1 and a′ t+1 ;

[0020] when a t+1 and a′t+1 At different times, an attack action is selected from the attack action library to attack the agent.

[0021] Furthermore, when a t+1 and a′ t+1 If the two are the same, no attack will be launched against the agent.

[0022] Furthermore, selecting an attack action from the attack action library to attack the agent includes:

[0023] The attack action is selected from the attack action library with the goal of minimizing the agent's reward value during the attack cycle.

[0024] Furthermore, the reward value is calculated by the following formula:

[0025]

[0026] Among them, p t For state s t Action a′ t The probability of being selected, U adv (s t ) is the attack strategy library for the agent, U(s t ) is the agent’s strategy library, γ t is the discount factor at time t, r adv (s t ,a′ t ) is the agent in s t Select attack action a' in the state t The reward value obtained later.

[0027] Furthermore, after selecting an attack action from the attack action library to attack the agent, the following steps are further included:

[0028] Calculate the difference between the attack success rate and the attack frequency as the first indicator;

[0029] Calculate the change in the reward value after the agent is attacked as the second indicator;

[0030] An attack effect value of the attack strategy is determined based on the first indicator and the second indicator.

[0031] Furthermore, the attack effect value of the attack strategy is calculated by F = 0.5 × (ΔR + Δfrequency);

[0032] Among them, F is the attack effect value, ΔR is the second indicator, and Δfrequency is the first indicator.

[0033] Another technical solution of the present invention is a method for generating an attack strategy for an intelligent agent, comprising:

[0034] The acquisition module is used to obtain the state of the agent at the current moment;

[0035] A calculation module, configured to calculate an attack time selection value at a current moment based on an attack time selection function and a state;

[0036] An attack module, configured to select an attack action from an attack action library to attack the agent when the attack time selection value is greater than a first threshold;

[0037] Among them, the attack time selection function is C(t)=α(π max -π min )+β(π max -π avg ), C(t) is the attack time selection value, t represents the current time sequence number, α and β are constants, π max is the maximum value of the output action probability in a given state, π min is the minimum value of the output action probability in a given state, π avg is the average probability of outputting an action in a given state.

[0038] Another technical solution of the present invention: a device for generating an attack strategy for an intelligent agent, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned method for generating an attack strategy for an intelligent agent is implemented.

[0039] The beneficial effects of the present invention are as follows: the present invention can significantly reduce the number of attacks by utilizing the attack time selection function, thereby ensuring the concealment of the attack; and utilize the trained attack action selection strategy to select the target action that reduces the cumulative reward the most to perform the target attack, thereby ensuring the effectiveness of the attack, effectively reducing the time complexity of the method, and thus improving the attack efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flow chart of a method for generating an attack strategy for an intelligent agent according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of attack principle against observation in the prior art;

[0042] Figure 3 2 is a state comparison principle diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] This invention aims to test the robustness and security of DRL algorithms through adversarial attacks. It can also be used to perform adversarial training using adversarial examples generated by this method, thereby improving the security of DRL algorithms. Furthermore, this invention can be used to attack agents trained through deep reinforcement learning to engage in illegal activities.

[0045] Figure 2 The attack process against the observation is represented as follows: the agent outputs the probability π(x) of selecting each action in the state according to the observation value x. The attacker generates an adversarial sample x′ through the attack strategy Att(x,π(x)) and feeds it back to the agent. At this time, the observation value of the agent becomes x′, and an action a′ is selected. After execution, the environment feeds back a reward r to the agent, thus completing an attack.

[0046] In 2014, Goodfellow et al. proposed adversarial attacks by applying FGSM to different policy networks in black-box and white-box settings. However, this approach only considered prediction errors at each step, not the agent's ultimate goal. Furthermore, it launched attacks at each time step, ignoring the effectiveness and stealth of adversarial attacks. Lin et al. proposed two attack algorithms for deep reinforcement learning: temporal policy attack and enchantment attack.

[0047] Goodfellow et al. calculated the difference in action probabilities at each step and attacked when it exceeded a given threshold, reducing the number of attack steps to some extent. However, this approach only considered the attack effect at the current moment, ignoring the impact of the current attack on subsequent states and the ultimate attack effect. Furthermore, the adversary could not control the number of attack steps. Results showed that the number of attacks launched accounted for approximately 25% of the total number of steps, making it ineffective. Lin et al.'s attack aims to lure the agent into a specific state. They first sample possible action sequences and then use a state prediction model to select the action sequence closest to the target state. This sequence is used to create adversarial samples for the attack. While this approach considers the ultimate goal of the attack and global optimization, it is difficult to predict states and actions over long timescales, resulting in a low attack success rate.

[0048] Sun et al. proposed two attack algorithms to optimize effectiveness and concealment: the critical point attack and the antagonistic attack. The critical point attack establishes a local prediction model to predict the final state of all possible attack action sequences. It then sets a damage awareness parameter (DAM), representing the difference between the states before and after the attack, and selects the target action sequence by comparing the DAM with a given threshold. Experimental results show that compared to the time strategy attack, this method effectively reduces the number of attacks while achieving better attack effectiveness. However, it limits the attack time to between t and (t+N-1), ignoring subsequent moments. This means that the most critical moments between t and (t+M) cannot be selected. Furthermore, the selection of attack action sequences requires exhaustive selection, increasing the algorithm's time complexity. The antagonistic attack trains a domain-agnostic adversarial model to automatically identify the optimal attack strategy in a specific state. When the attack action probability under this model is greater than 0.5 and the number of attacks is within a given range, the action is selected as the target action for attack. While this method effectively reduces the final reward, it limits the number of attacks to N, failing to achieve optimal results.

[0049] Based on this background, the present invention utilizes an attack method that combines state comparison and the best attack strategy selection model. This method first records the probability of selecting the next action in each state, and selects the key attack moment through the attack moment selection function. Then, the best attack action is selected as the target action through the best attack strategy selection model in combination with the comparison state model, and FGSM is used to generate target adversarial samples to launch attacks.

[0050] Unlike existing techniques that only consider the reward reduction after an attack at the current moment, this invention introduces state comparison. By comparing the state and action of the agent in the next step after the attack, it determines whether the attack is effective and then decides whether to attack, thus ensuring the ultimate attack effect. Through this attack method, the attacker can achieve the desired attack effect with a minimum number of attacks, while ensuring the concealment and effectiveness of the attack.

[0051] Based on deep reinforcement learning, this paper proposes a solution based on attack time and attack action. Attack time is explored using an attack time selection function and a state comparison model. An optimal attack action selection strategy is proposed for attack actions. This strategy is implemented using existing adversarial sample generation algorithms, such as FGSM, CW, or PGD. In a specific experiment, an agent model was trained in Python using PyTorch and the Tianshou algorithm library. The attack process was implemented using the adversarial sample generation algorithm called by Advertorch.

[0052] The attack process of the present invention is as follows Figure 1As shown, the probability of selecting the next action in each state is first recorded. The attack moment selection function is used to select the critical attack moment. The optimal attack action is then selected as the target action using the optimal attack strategy selection model, combined with the state comparison model. Finally, a target adversarial sample is generated and the attack is launched. The state after the attack is recorded as s′, and the state before the attack is recorded as s. The next action choices in these two states are compared to see if they are the same. If they are different, the attack changes the agent's choice, making it a valid attack and proceeding to the next step. Otherwise, the attack is considered invalid, and state s remains the current state, proceeding to the next step. This attack method allows the attacker to achieve the desired attack effect with a minimal number of attacks while ensuring both stealth and effectiveness.

[0053] Specifically, the present invention discloses a method for generating an attack strategy for an intelligent agent, comprising the following steps: obtaining the state of the intelligent agent at the current moment; calculating the attack time selection value at the current moment based on the attack time selection function and the state; when the attack time selection value is greater than a first threshold, selecting an attack action from an attack action library to attack the intelligent agent.

[0054] Among them, the attack time selection function is C(t)=α(π max -π min )+β(π max -π avg ), C(t) is the attack time selection value, t represents the current time sequence number, α and β are constants, π max is the maximum value of the output action probability in a given state, π min is the minimum value of the output action probability in a given state, π avg is the average probability of outputting an action in a given state.

[0055] The main idea behind policy gradient-based algorithms is to parameterize the policy π and calculate the policy gradient with respect to the action. Then, along the gradient, the action is continuously adjusted to gradually obtain the optimal policy. Its output π(a|s) = p[a|s,θ] indicates that when the state is s, the action a satisfies a certain probability distribution with parameter θ. Therefore, in order to select the critical moment, the agent's action probability distribution is measured under a certain state. In this case, the C(t) function can be expressed as:

[0056] C(t)=α(π max -π min )+β(π max -π avg )>Δ (1)

[0057] Wherein, Δ is the first threshold.

[0058] For deep reinforcement learning algorithms based on value functions, such as DQN, the main idea is to use DNN to approximate the reward value function, and the output Q(s t ,a t ) represents the value of the action in this state. In the present invention, the output Q(s t ,a t ) value needs to be processed by the softmax function first (T = 1 in the experiment), and then the C(t) function can be expressed as:

[0059]

[0060] That is π max pass Calculated, where s t is the state of the agent at time t, a t For s t The corresponding action of the intelligent agent, T is a hyperparameter, a tk is the probability that the agent outputs action k at time t; π min pass Calculated; π avg pass Calculated.

[0061] This function consists of two parts: the first half represents the agent's bias toward a particular action in a given state, and the second half represents the difference in the probability distribution of actions in that state. A larger value for C(t) indicates a greater bias toward a particular action at that moment, meaning that the moment is more critical. Conversely, a smaller difference in the probability of action at that moment indicates that the moment is less likely to be the optimal attack moment. This function can be used to select the most critical moment for the next state comparison. Since α and β are unknown, preliminary experiments are required to determine their values.

[0062] On the basis of the attack time selection function, the present invention also introduces state comparison to further screen the attack time, thereby ensuring that the most critical attack moment is selected.

[0063] The specific process can be described as: predicting the next action a' after attacking the agent. t+1 ; Calculate the next action a when the agent is not attacking t+1 ; Compare a t+1 and a′ t+1 ; when a t+1 and a′ t+1 When different, select an attack action from the attack action library to attack the agent. t+1 and a′ t+1 If the two are the same, no attack will be launched against the agent.

[0064] That is to say, if Figure 3 As shown, after launching an attack, the agent state can be recorded as s′ t+1 , in this state the next action to be chosen by the agent is recorded as a′ t+1 , normal state when not attacking t+1 The action selection below is a t+1 , compare these two actions. If they are the same, it means that the impact of the attack on the agent can be ignored, so no attack is carried out; if the actions are different, it means that the state and behavior of the agent are affected. At this time, the attack is an effective attack and may have a significant impact on the entire process. Adversarial samples can be made for adversarial attacks.

[0065] After determining the attack moments within the attack cycle, it is necessary to select specific attack actions to execute. In this embodiment of the present invention, an optimal attack action selection strategy is designed. This strategy selects the action that minimizes the agent's cumulative reward under a given state. Specifically, it selects an attack action from a library of attack actions with the goal of minimizing the agent's reward within the attack cycle.

[0066] The high-level strategy is established based on a deep neural network, and the expected cumulative reward under this strategy at time t~(T-1) is expressed by the following formula 3.

[0067]

[0068] Among them, p t For state s t Action a t The probability of being selected, U adv (s t ) is the attack strategy library for the agent, U(s t ) is the agent’s strategy library, γ t is the discount factor at time t, r adv (s t ,a t ′) is the agent in s t Select attack action a in state t ′ after the reward value, r(s t ,a t ) That is, the normal agent is in state s at time t t Next select action a t The reward obtained later, , both satisfy r adv (s t ,a′ t )=-r(s t ,a t ).

[0069] The input of this strategy is state s t, the output is the best attack action a′ in this state t And the probability p of the action being selected t , p t The larger the value, the more important the action.

[0070] When training the model in this strategy, the original action and state space are used for training, and each training collects (s t ,a′ t ,p t ,r adv ) and update the parameters to obtain the attack action and state space.

[0071] During training and actual application, attack strategies need to be evaluated to determine whether they can continue to be used. The evaluation process is as follows: The difference between the attack success rate and the attack frequency is calculated as the first metric; the change in the agent's reward after being attacked is calculated as the second metric; and based on the first and second metrics, the attack effectiveness value of the attack strategy is determined. The attack effectiveness value of the attack strategy is calculated as F = 0.5 × (ΔR + Δfrequency); where F is the attack effectiveness value, ΔR is the second metric, and Δfrequency is the first metric.

[0072] For example, when calculating the attack result measurement index F (i.e., the attack effect value), the difference between the attack success rate and the attack frequency is used as the horizontal axis, that is:

[0073]

[0074] Δfrequency represents the difference between the attack success rate and the attack frequency after the attacker launches an attack in an attack cycle. It is one of the indicators (i.e., the first indicator) to measure whether the attack is effective. Ideally, the attack success rate should be much higher than the attack frequency.

[0075] The original cumulative reward R of the agent 正常max Subtract the lowest cumulative reward R in the attacked state 攻击min As the maximum change in reward, the ratio of the cumulative reward R after executing the attack algorithm to the maximum change is used as the vertical coordinate, that is:

[0076]

[0077] ΔR is a measure of the change in reward after an attack (i.e., the second metric). For an effective attack algorithm, the ideal attack effect is one where a higher attack success rate, a lower attack frequency, and a greater change in the final reward. Therefore, this paper proposes a measurement metric, F; a larger F value indicates a better attack effect.

[0078] The pseudo code of the program in the embodiment of the present invention is shown below:

[0079]

[0080] Finally, the attack strategy generation method of the embodiment of the present invention is compared with other methods. The intelligent agent obtained by the DQN algorithm and the A2C algorithm is experimentally compared with the attack results of the embodiment of the present invention and the general perturbation attack and the time strategy attack. Before the attack experiment, the model of the intelligent agent under different Atari games is trained. The experiment uses three convolutional layers and one fully connected layer, and the convolutional layer uses the ReLu function activation. During training, the DQNpolicy module and the A2CPolicy module in Tianshou are used, the discount factor is set to 0.99, the learning rate is 0.0001, and the policy network is updated 10,000 times per epoch. At the same time, α+β=1 is used as a constraint condition, and the C(t) function results under each value are recorded. The values ​​of α and β are determined to be 0.5 respectively through preliminary experiments. Under normal circumstances, the cumulative rewards of DQN and A2C agents in a game (i.e., an attack cycle) are as follows:

[0081]

[0082] Before the attack, the C(t) function values ​​of the top 20% of each game are determined as the attack time range, as shown in the following table:

[0083] Game Name Threshold Pong (0.016,0.247) Breakout (0.13,0.85) Qbert (0.107,0.899) MsPacman (0.10,0.22) SpaceInvaders (0.046,0.261)

[0084] The experimental results of the embodiment method (Ours) of the present invention, the universal perturbation attack (UN), the time strategy attack (ST) combined with the adversarial sample generation algorithms FGSM, PGD and CW on the Atari game dataset are as follows. The F value of the present invention is higher than that of the other two algorithms in most cases, which proves the effectiveness of the present invention.

[0085]

[0086]

[0087] In summary, the present invention has the following features: (1) Aiming at the problem of time selection for adversarial attacks in deep reinforcement learning, an attack moment selection function is proposed. By measuring the difference in the probability distribution of each action at a certain moment and the degree of bias towards a certain action, the attack moment is determined, thereby achieving a significant attack effect with a smaller number of attacks. (2) Unlike the previous method that only considers the reward reduction after the attack at the current moment, the present invention innovatively introduces state comparison. By comparing the state action of the next agent after the attack, it is determined whether the attack is effective, and then whether to attack is selected, thereby ensuring the final attack effect. (3) In terms of target action selection and adversarial sample production, the present invention trains a strategy that is opposite to the agent's strategy. After selecting the attack moment, the strategy is used to determine the action that reduces the reward the most. Then, the adversarial sample generation algorithm is used to generate adversarial samples of the target action, thereby achieving the attack purpose. (4) Unlike the previous method that only focuses on the reward change, the present invention introduces a measurement index F and compares the difference in attack effects through the F value.

[0088] In addition, through the above-mentioned means, the present invention also has the following advantages: (1) The present invention utilizes the attack time selection function and the state comparison strategy, which greatly reduces the number of attacks compared with other existing attack algorithms, thereby ensuring the concealment of the attack; (2) Unlike the traditional algorithm that randomly generates adversarial samples, the present invention utilizes the best attack action selection strategy to select the target action that reduces the cumulative reward the most to perform the target attack, thereby ensuring the effectiveness of the attack; (3) The present invention adopts the best attack action selection strategy that has been trained in advance, which effectively reduces the time complexity of the algorithm compared with the key point attack method, thereby improving the attack efficiency.

[0089] The present invention also discloses a method for generating an attack strategy for an intelligent agent, comprising: an acquisition module for acquiring the state of the intelligent agent at the current moment; a calculation module for calculating the attack time selection value at the current moment based on the attack time selection function and the state; an attack module for selecting an attack action from an attack action library to attack the intelligent agent when the attack time selection value is greater than a first threshold; wherein the attack time selection function is C(t)=α(π max -π min )+β(π max -π avg ), C(t) is the attack time selection value, t represents the current time sequence number, α and β are constants, π max is the maximum value of the output action probability in a given state, π min is the minimum value of the output action probability in a given state, π avg is the average probability of outputting an action in a given state.

[0090] It should be noted that the information interaction, execution process, etc. between the modules of the above-mentioned device are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0092] The present invention also discloses a device for generating an attack strategy for an intelligent agent, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for generating an attack strategy for an intelligent agent is implemented.

[0093] The device may be a computing device such as a desktop computer, laptop, PDA, or cloud server. The device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the device may include more or fewer components, or a combination of certain components, or different components, and may also include, for example, input / output devices, network access devices, and the like.

[0094] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0095] In some embodiments, the memory may be an internal storage unit of the device, such as a hard disk or memory of the device. In other embodiments, the memory may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Furthermore, the memory may also include both an internal storage unit of the device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory may also be used to temporarily store data that has been output or is to be output.

[0096] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

Claims

1. A method for generating an attack strategy for an intelligent agent, characterized in that: The following steps are involved: Before the attack experiment, the agent model was trained using three convolutional layers and one fully connected layer. Get the state of the agent at the current moment; Calculate the attack time selection value at the current moment based on the attack time selection function and the state; When the attack time selection value is greater than a first threshold, an attack action is selected from an attack action library to attack the agent; the attack is an observation attack, and the observation attack is to add a disturbance to the observation image received by the agent; Wherein, the attack time selection function is C(t)=α(π max -π min )+β(π max -π avg ), C(t) is the attack time selection value, t represents the current time sequence number, α and β are constants, π max is the maximum value of the output action probability in a given state, π min is the minimum value of the output action probability in a given state, π avg is the average probability of outputting an action in a given state; The difference between the attack success rate and the attack frequency is calculated as the first indicator; Calculating a change in the reward value of the agent after being attacked as a second indicator; Determining an attack effect value of the attack strategy based on the first indicator and the second indicator; The attack effect value of the attack strategy is calculated by F = 0.5 × (ΔR + Δfrequency); Among them, F is the attack effect value, ΔR is the second indicator, and Δfrequency is the first indicator; Selecting an attack action from the attack action library to attack the agent includes: An attack action is selected from an attack action library with the goal of minimizing the reward value of the agent within the attack cycle.

2. A method for generating an attack strategy for an intelligent agent according to claim 1, characterized in that: The π max pass Calculated, where s t is the state of the agent at time t, a t For s t The corresponding action of the intelligent agent, T is a hyperparameter, a tk is the probability that the agent outputs action k at time t; Q(s t ,a t ) represents state s t Next action a t The value of .

3. A method for generating an attack strategy for an intelligent agent according to claim 1 or 2, characterized in that: When the attack time selection value is greater than the first threshold, the following step further includes: Predict the next action a' after attacking the agent t+1 ; Calculate the next action a when the agent is not attacked t+1 ; Compare the a t+1 and a′ t+1 ; When the a t+1 and a′ t+1 At different times, an attack action is selected from an attack action library to attack the agent.

4. A method for generating an attack strategy for an intelligent agent as claimed in claim 3, characterized in that: When the a t+1 and a′ t+1 If the two are the same, no attack is launched against the agent.

5. A method for generating an attack strategy for an intelligent agent as claimed in claim 4, characterized in that: The reward value is calculated using the following formula: Among them, p t For state s t Action a t The probability of being selected, U adv (s t ) is the attack strategy library for the agent, U(s t ) is the agent’s strategy library, γ t is the discount factor at time t, r adv (s t ,a t ′) is the agent in s t Select attack action a in state t ’The reward value obtained after 6. A method for generating an attack strategy for an intelligent agent, characterized in that: include: Before the attack experiment, the agent model was trained using three convolutional layers and one fully connected layer. The acquisition module is used to obtain the state of the agent at the current moment; a calculation module, configured to calculate an attack time selection value at a current moment based on an attack time selection function and the state; an attack module, configured to, when the attack time selection value is greater than a first threshold, select an attack action from an attack action library to attack the agent; the attack is an observation attack, and the attack action is to add a disturbance to the observation image received by the agent; Wherein, the attack time selection function is C(t)=α(π max -π min )+β(π max -π avg ), C(t) is the attack time selection value, t represents the current time sequence number, α and β are constants, π max is the maximum value of the output action probability in a given state, π min is the minimum value of the output action probability in a given state, π avg is the average probability of outputting an action in a given state; The difference between the attack success rate and the attack frequency is calculated as the first indicator; Calculating a change in the reward value of the agent after being attacked as a second indicator; Determining an attack effect value of the attack strategy based on the first indicator and the second indicator; The attack effect value of the attack strategy is calculated by F = 0.5 × (ΔR + Δfrequency); Among them, F is the attack effect value, ΔR is the second indicator, and Δfrequency is the first indicator; Selecting an attack action from the attack action library to attack the agent includes: An attack action is selected from an attack action library with the goal of minimizing the reward value of the agent within the attack cycle.

7. A device for generating an attack strategy for an intelligent agent, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a method for generating an attack strategy for an intelligent agent as described in any one of claims 1-6.

Images