An attack testing method, device and equipment of an intelligent agent and a storage medium

Abstract

The application discloses an attack test method, device and equipment of an intelligent agent, and a storage medium, and is applied to the field of model test. The method comprises the following steps: determining a target intelligent agent from a plurality of intelligent agents of reinforcement learning, and constructing a data set by observing the observation value and the corresponding action output by the target intelligent agent under a black box condition; performing data enhancement on the data set, and training a substitute strategy model of the target intelligent agent based on the data set after the data enhancement; generating an adversarial disturbance for changing the decision action of the target intelligent agent based on the substitute strategy model; and adding the adversarial disturbance to the observation value to be disturbed of the target intelligent agent to perform an adversarial attack test. The application realizes a hidden and effective adversarial attack test under the condition that the internal information of the model is not accessed and frequent queries are not required by performing a multi-intelligent agent reinforcement learning black box adversarial attack based on strategy stealing.

Description

Technical Field

[0001] This invention relates to the field of model testing, and in particular to an attack testing method for intelligent agents, an attack testing device for intelligent agents, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Attack testing on agents helps discover security vulnerabilities, evaluate model performance, and guide subsequent model optimization. Teams of agents trained using multi-agent reinforcement learning algorithms are vulnerable to adversarial attacks. Attackers can cause agents to deviate from their optimal strategies by adding small perturbations to their observations, leading to team collaboration failure. Most existing adversarial attack testing methods are implemented in white-box environments, requiring access to the target agent model's internal parameters and gradient information, which is difficult to achieve in real-world deployments. Existing black-box attack methods (such as query-based methods) require frequent interaction with the target model, making them easily detectable by defense mechanisms. Summary of the Invention

[0003] The purpose of this invention is to provide an attack testing method, apparatus, device, and storage medium for intelligent agents, which are applied in the field of model testing. This method achieves covert and effective adversarial attack testing by using multi-agent reinforcement learning black-box adversarial attacks based on policy theft, without needing to access internal model information or frequently querying it.

[0004] To address the aforementioned technical problems, this invention provides a method for attack testing of intelligent agents, comprising: The target agent is identified from the multi-agent group in reinforcement learning, and a dataset is constructed by observing the output values ​​and corresponding actions of the target agent under black-box conditions. The dataset is augmented, and an alternative policy model for the target agent is trained based on the augmented dataset. Based on the alternative strategy model, generate adversarial perturbations that alter the decision-making actions of the target agent; The adversarial perturbation is added to the observations to be perturbed of the target agent to conduct adversarial attack testing.

[0005] Optionally, data augmentation may be performed on the dataset, including: Determine the target observation from the observed values, and determine the Jacobian matrix of the target observation; Determine the first-order partial derivatives of the target observations in the Jacobian matrix corresponding to the target actions; The updated observation value is obtained by updating the target observation value based on the first-order partial derivative of the target action; New data samples are constructed based on the updated observations and the target action and added to the dataset for data augmentation.

[0006] Optionally, data augmentation may be performed on the dataset, including: Determine the target observation from the observed values, and determine the Jacobian matrix of the target observation; Determine the first-order partial derivatives of the target observations in the Jacobian matrix corresponding to the target actions; Determine the maximum first-order partial derivative from the first-order partial derivatives of the corresponding target action, and determine the target observation index corresponding to the maximum first-order partial derivative; The updated observation value is obtained by updating the value of the target observation index in the target observation value based on the maximum first-order partial derivative. New data samples are constructed based on the updated observations and the target action and added to the dataset for data augmentation.

[0007] Optionally, data augmentation may be performed on the dataset, including: Identify target observations with the same actions from the dataset, and determine the cosine similarity between the target observations; The target observations with the highest cosine similarity are paired to obtain paired observations; The average observation between the paired observations is determined as the updated observation. New data samples are constructed based on the updated observation and the corresponding actions of the paired observations and added to the dataset for data augmentation.

[0008] Optionally, data augmentation may be performed on the dataset, including: Determine the target observation value from the observed values, and add a constant perturbation to the random observation index dimension of the target observation value to obtain the updated observation value; New data samples are constructed based on the corresponding actions of the updated observations and the target observations and added to the dataset for data augmentation.

[0009] Optionally, generating adversarial perturbations that alter the target agent's decision-making actions based on the alternative policy model includes: Initialize the counter-perturbation to be optimized; The observation to be disturbed is input into the alternative strategy model, and the action with the lowest confidence output by the alternative strategy model is determined as the first target action; The disturbance to be optimized is added to the observations to be disturbed to obtain the disturbance observations; The disturbance observations are input into the alternative strategy model, and the action with the highest confidence output by the alternative strategy model is determined as the second target action. If the first target action is the same as the second target action, then the countermeasure disturbance to be optimized is determined as the optimal countermeasure disturbance; If the first target action is different from the second target action, then a loss function is determined, the adversarial perturbation to be optimized is updated based on the loss function, and the process restarts from the step of adding the adversarial perturbation to be optimized to obtain the perturbation observation.

[0010] Optionally, determine the loss function, including: The constraint perturbation amplitude is determined based on the L2 norm of the perturbation to be optimized; The loss function is determined based on the magnitude of the constraint perturbation and the confidence level of the second target action.

[0011] To address the aforementioned technical problems, this invention provides an attack testing device for intelligent agents, comprising: The first module is used to identify the target agent from the multi-agent reinforcement learning, and to construct a dataset by observing the observed values ​​and corresponding actions output by the target agent under black-box conditions. The second module is used to perform data augmentation on the dataset and train an alternative policy model for the target agent based on the data-augmented dataset. The third module is used to generate adversarial perturbations that change the decision-making actions of the target agent based on the alternative strategy model. The fourth module is used to add the adversarial perturbation to the observation values ​​to be perturbed of the target agent for adversarial attack testing.

[0012] To solve the above-mentioned technical problems, the present invention provides an electronic device, comprising: Memory, used to store computer programs; A processor for implementing an attack testing method for the agent when executing the computer program.

[0013] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the attack testing method for the intelligent agent.

[0014] As can be seen, this invention identifies the target agent from a multi-agent reinforcement learning system, constructs a dataset by observing the target agent's output and corresponding actions under black-box conditions, performs data augmentation on the dataset, trains an alternative policy model for the target agent based on the augmented dataset, generates adversarial perturbations to change the target agent's decision-making actions based on the alternative policy model, and adds adversarial perturbations to the target agent's observations to be perturbed for adversarial attack testing. This invention achieves covert and effective adversarial attack testing through black-box adversarial attacks based on policy theft in multi-agent reinforcement learning, without requiring access to internal model information or frequent queries. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 A flowchart of an attack testing method for an intelligent agent provided in an embodiment of the present invention; Figure 2 This is a structural block diagram of an attack testing device for an intelligent agent provided in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Multi-agent reinforcement learning (MARL) has been widely applied in key areas such as drone control, autonomous driving, and traffic management. In cooperative MARL, multiple agents learn cooperative strategies through training to complete complex tasks in a team-based manner. The centralized training with decentralized execution (CTDE) framework is the mainstream paradigm in current MARL algorithm design, with the QMIX algorithm being a typical example. In the QMIX algorithm, each agent is controlled by a deep recurrent Q-network (DRQN). This network takes the agent's local observations as input and outputs the Q-value score for each available action. The agent selects the action with the highest Q-value to execute. During training, QMIX utilizes a centralized hybrid network to receive the global state to guide each agent in updating its parameters.

[0019] However, research shows that agents trained using multi-agent reinforcement learning algorithms are vulnerable to adversarial attacks. Attackers can cause agents to deviate from their optimal strategies by adding tiny perturbations to their observations, leading to teamwork failure and ultimately task failure. Adversarial attacks aim to add imperceptible perturbations to the model input, causing the model to output incorrect results at high confidence levels. In the security testing of multi-agent systems, adversarial attack methods are an important tool for evaluating system robustness.

[0020] Existing adversarial attack methods are mainly divided into two categories: white-box attacks and black-box attacks. White-box attack methods require obtaining the internal parameters and gradient information of the target model. Although the attack effect is significant, attackers usually cannot obtain complete information about the target model in real-world deployment scenarios, thus severely limiting the practical feasibility of white-box attacks. Black-box attack methods do not need to know the internal structure or gradient information of the model and are mainly achieved through two approaches: one is a query-based method, which estimates the gradient direction by repeatedly querying the target model, but this method requires frequent interaction with the target model and is easily detected by defense mechanisms; the other is a transferability-based method, which uses adversarial examples generated on alternative models to attack the target model.

[0021] In summary, existing technologies have the following shortcomings: First, they are highly dependent on white-box methods, as most effective attack methods require obtaining model parameters and gradient information, which is difficult to achieve in real-world deployment scenarios. Second, they have poor query concealment, as query-based black-box attack methods require frequent interaction with the target model, making them easy to detect and defend against. Third, model theft attacks in multi-agent scenarios have not been systematically studied, and there is a lack of complete technical solutions on how to utilize agent behavior observations to steal policy models and generate effective adversarial perturbations. Fourth, traditional data augmentation methods are not sufficiently applicable in multi-agent reinforcement learning scenarios, affecting the fidelity of alternative models.

[0022] This invention provides a black-box adversarial attack method for multi-agent reinforcement learning based on policy theft. This method constructs an alternative policy model by passively observing the behavior of the target agent, and then uses the alternative model to generate adversarial perturbations with cross-model transferability, thereby achieving covert and effective adversarial attacks without accessing internal model information or frequent queries.

[0023] The following combination Figure 1 , Figure 1 A flowchart of an attack testing method for an intelligent agent provided in an embodiment of the present invention is shown. The method includes: S101: Identify the target agent from the multi-agent group in reinforcement learning, and construct a dataset by observing the output values ​​and corresponding actions of the target agent under black-box conditions.

[0024] This embodiment can identify the target agent to be attacked from multiple agents in reinforcement learning. During the execution phase of the target agent, the dataset is constructed by observing the output values ​​and corresponding actions of the target agent under black-box conditions.

[0025] During the deployment and execution phases of the agent, since the agent tends to choose the optimal action at each time step, these actions are essentially the specific manifestation of the optimal path in the agent's experienced path. Therefore, by observing the execution process of the agent team, we can collect the observations of the target agent at time step t. and corresponding actions During this process, no additional information about the target agent, such as network structure or loss function, is required; the entire data collection process is completed in a black-box environment. After data collection is completed within a preset time window, a dataset for training the alternative policy model is constructed from multiple observation-action pairs.

[0026] S102: Perform data augmentation on the dataset, and train an alternative policy model for the target agent based on the augmented dataset.

[0027] For each observation-action pair, the observation is used as the model input, and the action selected by the corresponding target agent is used as the model output. The process of training the alternative policy model can be modeled as a multi-classification task training process on a labeled dataset. The training objective is to enable the alternative policy model to acquire the ability to select the correct action based on the observation.

[0028] To expand the dataset and improve the performance of the trained alternative policy model, thereby providing a foundation for improving the effectiveness of the next adversarial attack, this embodiment can perform data augmentation on the dataset.

[0029] This embodiment does not limit the specific method of data augmentation, and can be set according to the actual application.

[0030] This embodiment can determine the target observation from the observations, determine the Jacobian matrix of the target observation, determine the first-order partial derivative of the target action corresponding to the target observation in the Jacobian matrix, update the target observation based on the first-order partial derivative of the target action to obtain the updated observation, and construct new data samples based on the updated observation and the target action and add them to the dataset for data augmentation.

[0031] In this embodiment, the target observation is the observation used for data augmentation calculation. This embodiment does not limit the number of target observations or the selection method. Generally, all observations in the dataset can be determined as target observations.

[0032] Specifically, this embodiment proposes a data augmentation method based on the Jacobian matrix (JM). The Jacobian matrix is ​​the first-order partial derivative of each dimension of the model output with respect to the model input.

[0033] Assume the target observation is the observation of the target agent at time step t. The corresponding target action is This embodiment can determine the Jacobian matrix of the target observation. And determine the first-order partial derivatives of the target observations in the Jacobian matrix corresponding to the target actions. The updated observation is obtained by updating the target observation based on the first-order partial derivative of the target action, as shown in the following formula: ; In the formula, To update the observations, Let c be the target observation, sgn be a constant, and sgn be the sign function. Let Jacobian matrix be the target observation. It is the first-order partial derivative of the target action in the Jacobian matrix.

[0034] By updating the observations With target action Forming new samples Add to the dataset for data augmentation.

[0035] Furthermore, in this embodiment, the target observation value can be determined from the observed values, and the Jacobian matrix of the target observation value can be determined; the first-order partial derivative of the target action corresponding to the target observation value in the Jacobian matrix can be determined; the largest first-order partial derivative can be determined from the first-order partial derivative of the corresponding target action, and the target observation index corresponding to the largest first-order partial derivative can be determined; the value of the target observation index in the target observation value is updated based on the largest first-order partial derivative to obtain the updated observation value; and new data samples are constructed based on the updated observation value and the target action and added to the dataset for data augmentation.

[0036] Specifically, this embodiment proposes a data augmentation method based on the One Point of Jacobian Matrix (OPJM). Each feature dimension in the agent's observations is unique. Changing all feature dimensions simultaneously may lead to observations that deviate from the normal distribution, thus affecting the accuracy of the updated data. The OPJM method limits the range of data variation by selecting the feature dimension with the largest absolute value in the Jacobian matrix for data updates. Its mathematical expression is as follows: ; In the formula, max is the maximum value function. Let be the maximum first-order partial derivative of the target action in the Jacobian matrix of the target observations.

[0037] Generally, observations contain multiple observation metrics, i.e., feature dimensions. In this embodiment, the updated observations are obtained by selecting the observation metric corresponding to the largest first-order partial derivative. By updating the observations With target action Forming new samples Add to the dataset for data augmentation.

[0038] Furthermore, this embodiment can also identify target observations with the same actions from the dataset, determine the cosine similarity between the target observations, pair the target observations with the maximum cosine similarity to obtain paired observations, determine the average observation between the paired observations as the updated observations, and construct new data samples based on the corresponding actions of the updated observations and paired observations and add them to the dataset for data augmentation.

[0039] Specifically, this embodiment proposes a data augmentation method based on cosine similarity (CS), which identifies all observations that make the same decision as the target observations. This embodiment first calculates the cosine similarity between the target observations, and then selects the two different target observations with the highest similarity at each step. and Pairwise matching forms observation pairs, and their average observation value is used to form these pairs. As an update observation, with action Composition of augmented data Add it to the dataset.

[0040] This embodiment can also determine the target observation value from the observation values, add a constant perturbation to the random observation index dimension of the target observation value to obtain the updated observation value, and construct new data samples based on the corresponding actions of the updated observation value and the target observation value and add them to the dataset for data augmentation.

[0041] Specifically, this embodiment proposes a data augmentation method based on One Random Point (ORP). This embodiment constructs neighboring observations as updated observations by adding a constant perturbation to a random single dimension of the observations. Build new enhanced data Adding perturbations to the random dimension simulates the state transition process in reinforcement learning, thus achieving a data augmentation method that is more in line with the reinforcement learning environment.

[0042] During the training of the alternative policy model, this embodiment can expand the dataset by any combination of the four data augmentation methods described above, so as to construct different datasets and train different alternative policy models.

[0043] This embodiment does not limit the specific architecture of the alternative strategy model; it can generally be a fully connected neural network.

[0044] S103: Generate adversarial perturbations that alter the decision-making actions of the target agent based on an alternative policy model.

[0045] Suppose an attack is launched on the target agent at time t, and the observed value to be disturbed of the target agent at this time is... The action chosen by the target agent is At this point, because the accuracy of the target agent's decision-making fitted by the alternative strategy model cannot be guaranteed to be absolutely correct, the action selected by the alternative strategy model is... Not necessarily with Consistent.

[0046] In other words, adversarial perturbations that can change the decision-making actions of an alternative policy model may not necessarily change the decision-making actions of the target agent. Even if an adversarial perturbation that can change the action chosen by the alternative policy model is found... The minimum counter-perturbation, which may also fail to change .

[0047] Therefore, this embodiment aims to iteratively optimize the counter-perturbation under the condition of constraining the perturbation amplitude, so that it can change the decision-making actions of the target agent.

[0048] Specifically, this embodiment can first initialize the adversarial disturbance to be optimized. ; The observations to be disturbed are input into the alternative policy model, and the action with the lowest confidence output by the alternative policy model is determined as the first target action, as shown in the following formula: ; In the formula, min is the minimum value function. As an alternative strategy model, For the observations to be disturbed, The confidence level of each action output under the observed values ​​to be disturbed. The first target action is the one with the lowest confidence level.

[0049] Furthermore, in this embodiment, the adversarial disturbance to be optimized can be added to the observations to be disturbed to obtain the disturbed observations. .

[0050] The perturbation observations are input into the alternative policy model, and the action with the highest confidence output by the alternative policy model is determined as the second target action, as shown in the following formula: ; In the formula, For perturbation observations, This represents the confidence level of each action output under perturbed observations. The second target action is the one with the highest confidence level.

[0051] If the action of the first target is the same as the action of the second target, that is If the iteration terminates, the adversarial perturbation to be optimized is determined as the optimal adversarial perturbation; If the action of the first target is different from the action of the second target, that is If so, the loss function is determined, the adversarial perturbation to be optimized is updated based on the loss function, and the process restarts from the step of adding the adversarial perturbation to be optimized to the observations to be perturbed to obtain the perturbed observations.

[0052] This embodiment does not limit the specific method of determining the loss function. Generally, the constraint perturbation magnitude can be determined based on the L2 norm of the adversarial perturbation to be optimized; the loss function can be determined based on the constraint perturbation magnitude and the confidence of the second target action, as shown in the following formula: ; In the formula, loss is the loss function. To optimize the counter-disturbance, The L2 norm is used to optimize the disturbance response and constrain the disturbance amplitude. Let z be the confidence level of the second target action, and z be a constant.

[0053] Optimizing the adversarial perturbation on the aforementioned loss function allows us to constrain the magnitude of the adversarial perturbation while ensuring that the perturbation chooses actions that are not optimal towards the alternative strategy model. The optimization direction will be optimized, and the optimization method will continue to iterate until a pre-set maximum number of iterations is reached, or until the desired result is achieved during the iteration. Then exit and return to the anti-disturbance. .

[0054] S104: Add adversarial perturbations to the observations to be perturbed of the target agent to conduct adversarial attack tests.

[0055] This embodiment can add adversarial perturbations to the observations of the target agent to conduct adversarial attack tests. The adversarial attack test in this embodiment can mainly obtain the degree of action deviation of the model under adversarial attack, evaluate the performance loss based on the Q value of the action after deviation, and optimize the model to improve the model's security performance.

[0056] Based on the above embodiments, the present invention achieves covert and effective adversarial attack testing through a strategy-stealing multi-agent reinforcement learning black-box adversarial attack, without needing to access internal model information or frequently query it.

[0057] The following combination Figure 2 , Figure 2 This is a structural block diagram of an attack testing device for an intelligent agent provided in an embodiment of the present invention. The device may include: The first module 100 is used to identify the target agent from the multi-agent reinforcement learning, and to construct a dataset by observing the observed values ​​and corresponding actions of the target agent under black-box conditions. The second module 200 is used to augment the dataset and train an alternative policy model for the target agent based on the augmented dataset. The third module 300 is used to generate adversarial perturbations that change the decision-making actions of the target agent based on the alternative strategy model. The fourth module 400 is used to add adversarial perturbations to the observations to be perturbed of the target agent for adversarial attack testing.

[0058] Based on the above embodiments, the present invention achieves covert and effective adversarial attack testing through a strategy-stealing multi-agent reinforcement learning black-box adversarial attack, without needing to access internal model information or frequently query it.

[0059] Based on the above embodiments, the second module 200 may include: The first unit is used to determine the target observation from the observations and to determine the Jacobian matrix of the target observation; The second unit is used to determine the first-order partial derivatives of the target actions corresponding to the target observations in the Jacobian matrix; The third unit is used to update the target observations based on the first-order partial derivatives of the target action to obtain the updated observations; The fourth unit is used to construct new data samples based on updated observations and target actions and add them to the dataset for data augmentation.

[0060] Based on the above embodiments, the second module 200 may include: The fifth unit is used to determine the target observation from the observations and to determine the Jacobian matrix of the target observation; The sixth unit is used to determine the first-order partial derivatives of the target actions corresponding to the target observations in the Jacobian matrix; The seventh unit is used to determine the maximum first-order partial derivative from the first-order partial derivatives of the corresponding target action, and to determine the target observation index corresponding to the maximum first-order partial derivative. The eighth unit is used to update the target observation index in the target observation based on the maximum first-order partial derivative to obtain the updated observation; Unit 9 is used to construct new data samples based on updated observations and target actions and add them to the dataset for data augmentation.

[0061] Based on the above embodiments, the second module 200 may include: Unit 10 is used to identify target observations with the same actions from the dataset and to determine the cosine similarity between target observations. Unit 11 is used to pair target observations with the maximum cosine similarity to obtain paired observations. Unit 12 is used to determine the average observation between paired observations as the updated observation, and to construct new data samples based on the updated observation and the corresponding actions of the paired observations and add them to the dataset for data augmentation.

[0062] Based on the above embodiments, the second module 200 may include: The thirteenth unit is used to determine the target observation from the observations. A constant perturbation is added to the random observation index dimension of the target observation to obtain the updated observation. Unit 14 is used to construct new data samples based on the corresponding actions of updating observations and target observations and add them to the dataset for data augmentation.

[0063] Based on the above embodiments, the third module 300 may include: Unit 15 is used to initialize the adversarial perturbation to be optimized; The sixteenth unit is used to input the observations to be disturbed into the alternative strategy model, and to determine the action with the lowest confidence output by the alternative strategy model as the first target action; Unit 17 is used to add the adversarial disturbance to be optimized to obtain the disturbance observation; The eighteenth unit is used to input the perturbation observations into the alternative strategy model, and to determine the action with the highest confidence output by the alternative strategy model as the second target action; The nineteenth unit is used to determine the countermeasure perturbation to be optimized as the optimal countermeasure perturbation if the action of the first target is the same as the action of the second target. Unit 20 is used to determine the loss function if the first target action is different from the second target action, update the adversarial perturbation to be optimized based on the loss function, and restart execution from Unit 17.

[0064] Based on the above embodiments, the twentieth unit may include: The first sub-unit is used to determine the constraint perturbation amplitude based on the L2 norm of the perturbation to be optimized; The second sub-unit is used to determine the loss function based on the constraint perturbation magnitude and the confidence level of the second target action.

[0065] Based on the above embodiments, the present invention also provides an electronic device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the device may also include various necessary network interfaces, a power supply, and other components.

[0066] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an execution terminal or processor, can implement the method provided in the embodiments of the present invention; the storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0067] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for testing attacks on intelligent agents, characterized in that, include: The target agent is identified from the multi-agent reinforcement learning, and the dataset is constructed by observing the observed values ​​and corresponding actions of the target agent under black-box conditions. The dataset is augmented, and an alternative policy model for the target agent is trained based on the augmented dataset. Based on the alternative strategy model, generate adversarial perturbations that alter the decision-making actions of the target agent; The adversarial perturbation is added to the observations to be perturbed of the target agent to conduct adversarial attack testing.

2. The attack testing method for an intelligent agent according to claim 1, characterized in that, Data augmentation of the dataset includes: Determine the target observation from the observed values, and determine the Jacobian matrix of the target observation; Determine the first-order partial derivatives of the target observations in the Jacobian matrix corresponding to the target actions; The updated observation value is obtained by updating the target observation value based on the first-order partial derivative of the target action; New data samples are constructed based on the updated observations and the target action and added to the dataset for data augmentation.

3. The attack testing method for an intelligent agent according to claim 1, characterized in that, Data augmentation of the dataset includes: Determine the target observation from the observed values, and determine the Jacobian matrix of the target observation; Determine the first-order partial derivatives of the target observations in the Jacobian matrix corresponding to the target actions; Determine the maximum first-order partial derivative from the first-order partial derivatives of the corresponding target action, and determine the target observation index corresponding to the maximum first-order partial derivative; The updated observation value is obtained by updating the value of the target observation index in the target observation value based on the maximum first-order partial derivative. New data samples are constructed based on the updated observations and the target action and added to the dataset for data augmentation.

4. The attack testing method for an intelligent agent according to claim 1, characterized in that, Data augmentation of the dataset includes: Identify target observations with the same actions from the dataset, and determine the cosine similarity between the target observations; The target observations with the highest cosine similarity are paired to obtain paired observations; The average observation between the paired observations is determined as the updated observation. New data samples are constructed based on the updated observation and the corresponding actions of the paired observations and added to the dataset for data augmentation.

5. The attack testing method for an intelligent agent according to claim 1, characterized in that, Data augmentation of the dataset includes: Determine the target observation value from the observed values, and add a constant perturbation to the random observation index dimension of the target observation value to obtain the updated observation value; New data samples are constructed based on the corresponding actions of the updated observations and the target observations and added to the dataset for data augmentation.

6. The attack testing method for an intelligent agent according to claim 1, characterized in that, Based on the alternative policy model, adversarial perturbations that alter the target agent's decision-making actions are generated, including: Initialize the counter-perturbation to be optimized; The observation to be disturbed is input into the alternative strategy model, and the action with the lowest confidence output by the alternative strategy model is determined as the first target action; The disturbance to be optimized is added to the observations to be disturbed to obtain the disturbance observations; The disturbance observations are input into the alternative strategy model, and the action with the highest confidence output by the alternative strategy model is determined as the second target action. If the first target action is the same as the second target action, then the countermeasure disturbance to be optimized is determined as the optimal countermeasure disturbance; If the first target action is different from the second target action, then a loss function is determined, the adversarial perturbation to be optimized is updated based on the loss function, and the process restarts from the step of adding the adversarial perturbation to be optimized to obtain the perturbation observation.

7. The attack testing method for an intelligent agent according to claim 6, characterized in that, Determine the loss function, including: The constraint perturbation amplitude is determined based on the L2 norm of the perturbation to be optimized; The loss function is determined based on the magnitude of the constraint perturbation and the confidence level of the second target action.

8. An attack testing device for an intelligent agent, characterized in that, include: The first module is used to identify the target agent from the multi-agent reinforcement learning, and to construct a dataset by observing the observed values ​​and corresponding actions output by the target agent under black-box conditions. The second module is used to perform data augmentation on the dataset and train an alternative policy model for the target agent based on the data-augmented dataset. The third module is used to generate adversarial perturbations that change the decision-making actions of the target agent based on the alternative strategy model. The fourth module is used to add the adversarial perturbation to the observation values ​​to be perturbed of the target agent for adversarial attack testing.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the attack testing method for the agent as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the attack testing method for an intelligent agent as described in any one of claims 1 to 7.

Images