Multi-agent system robustness evaluation method and device based on black box attack
By optimizing the adversarial example method, the problem of inaccurate robustness evaluation in black-box attacks is solved, the accuracy of robustness evaluation of multi-agent systems is improved, and it is applicable to black-box scenarios.
Patent Information
- Application Number
- CN202511381877.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-25
- Publication Date
- 2026-01-06
AI Technical Summary
Existing black-box attack methods rely too heavily on the quality of adversarial examples, leading to inaccurate robustness assessments of multi-agent systems.
The quality of adversarial samples is improved by optimizing them based on robustness evaluation results after each attack. New adversarial samples are generated using crossover and mutation operations and optimized using multiple robustness evaluation functions until the preset number of attacks is reached.
It improves the attack strength of adversarial examples, significantly enhances the robustness evaluation accuracy of multi-agent systems, and is applicable to black-box scenarios where no structural and parameter information of the target model is required.
Smart Images
Figure CN121279484A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent security technology, specifically relating to a method and apparatus for robustness evaluation of multi-agent systems based on black-box attacks. Background Technology
[0002] Multi-agent deep reinforcement learning, as a cutting-edge research direction in artificial intelligence, has demonstrated significant advantages and immense application value in addressing decision-making problems in complex systems. This technology, by constructing a distributed agent collaborative learning framework, has achieved breakthroughs in areas such as UAV swarm control, intelligent traffic scheduling, and edge computing resource optimization. Taking UAV swarm collaborative reconnaissance as an example, multi-agent deep reinforcement learning can achieve autonomous path planning and task allocation in dynamic environments; in vehicle-to-everything (V2X) scenarios, this technology can support multi-vehicle collaborative navigation and conflict resolution; and in cloud computing environments, it can optimize the dynamic scheduling efficiency of distributed computing resources. This swarm intelligence paradigm overcomes the limitations of traditional single-agent systems in environmental perception and collaborative decision-making, significantly improving the overall performance of complex systems.
[0003] However, as the application scenarios of multi-agent deep reinforcement learning expand, the security challenges it faces are becoming increasingly prominent. Malicious attackers may induce agents to exhibit abnormal behavior by tampering with environmental parameters or contaminating observational data. More seriously, attackers may exploit information leakage vulnerabilities during model interaction to reverse engineer sensitive training data or environmental state information. Research has also found that multi-agent deep reinforcement learning systems exhibit particular vulnerability to adversarial attacks—carefully designed perturbation inputs can cause the group decision-making system to collapse. Furthermore, by implanting malicious agents, attackers can manipulate the convergence direction of the entire learning system. These security vulnerabilities severely restrict the practical application of multi-agent deep reinforcement learning technology in security-sensitive fields such as autonomous driving and industrial control.
[0004] Adversarial attacks, as an important technique for evaluating the robustness of reinforcement learning systems, have received widespread attention in recent years. Existing adversarial attack methods are divided into white-box attacks and black-box attacks. White-box attacks require multiple iterations of computation, especially against large deep learning models, resulting in significant computational overhead. Furthermore, adversarial examples generated against a particular white-box model may not be transferable to other models because the attack is highly dependent on specific parameters. Finally, attackers typically find it difficult to obtain all the details of the model (such as architecture, parameters, and training data).
[0005] In most real-world scenarios, models are partially or completely black boxes. In contrast, black-box attacks are more practical because they only require input-output interaction, do not require knowledge of the model's internal details, are more computationally efficient, and the generated adversarial examples often exhibit better cross-model transferability, making them more suitable for evaluating the robustness of deployed reinforcement learning systems. Generally, existing black-box attack processes involve adding random perturbations to samples to generate adversarial examples, then inputting these examples into the model to obtain the adversarial example's output. The adversarial example's output is then compared with the expected output of the original sample to assess system robustness. However, this attack process relies heavily on the quality of the adversarial examples. When the adversarial example is of low quality, its output is difficult to distinguish from the expected output, leading to inaccurate robustness assessments. Summary of the Invention
[0006] The purpose of this invention is to provide a method and apparatus for robustness evaluation of multi-agent systems based on black-box attacks, so as to solve the technical problem that black-box attacks rely too much on the quality of adversarial samples, resulting in inaccurate robustness evaluation.
[0007] To address the aforementioned technical problems, the present invention provides a technical solution for a robustness evaluation method for multi-agent systems based on black-box attacks: a robustness evaluation method for multi-agent systems based on black-box attacks, comprising:
[0008] 1) Perturbations are superimposed on the original training sample set of the multi-agent system to be evaluated to obtain the adversarial sample set;
[0009] 2) Input the adversarial samples in the adversarial sample set into the multi-agent system to be evaluated to complete an attack, determine the evaluation index corresponding to the adversarial sample for evaluating robustness based on the running results, and obtain the robustness evaluation result of the multi-agent system corresponding to the adversarial sample based on the evaluation index.
[0010] 3) Based on the robustness evaluation results of the multi-agent system, optimize the adversarial sample set with the minimum deviation between the evaluation index and the expected evaluation index as the optimization objective;
[0011] Re-execute step 2 based on the optimized adversarial sample set until the preset number of attacks is reached, and use the robustness evaluation result obtained from the last attack as the final robustness evaluation result.
[0012] The beneficial effects of the above technical solution are as follows: The technical solution of the robustness evaluation method for multi-agent systems based on black-box attacks of the present invention belongs to an improved invention. When performing a black-box attack, after each attack, the adversarial sample is optimized using an optimization algorithm based on the robustness evaluation results. This significantly improves the quality of the adversarial samples, increases the attack strength on the agents, and greatly improves the accuracy of the robustness evaluation results. The present invention solves the technical problem in the prior art where black-box attacks rely too heavily on the quality of adversarial samples, leading to inaccurate robustness evaluations.
[0013] Furthermore, the process of optimizing the adversarial example set includes:
[0014] 31) Randomly select two adversarial samples, AS1 and AS2, from the set of adversarial samples;
[0015] The probability of each adversarial sample being selected is determined based on the robustness evaluation result corresponding to that adversarial sample;
[0016] 32) Perform crossover and mutation operations on AS1 and AS2 in sequence to generate two new adversarial examples. and ;
[0017] 33) will Input the multi-agent system to be evaluated, and obtain Corresponding reward value q = 1 or 2;
[0018] like Then it will be from the adversarial sample set Sample replacement Sample; otherwise, no replacement. Indicates based on sample The reward received by the agent;
[0019] 34) Re-execute 31) until the preset number of optimized loops is reached.
[0020] Furthermore, in 31), each adversarial example Probability of being selected Determined according to the following formula:
[0021]
[0022] in, For the sample The corresponding robustness assessment results; This represents the number of adversarial samples in the adversarial sample set.
[0023] Furthermore, perform crossover operations on AS1 and AS2 as follows:
[0024]
[0025] in, and All samples are generated after crossover operations. B is a crossover template matrix of the same size as the adversarial sample, and each element in B is a randomly generated 0 or 1. 1 represents the Hadamard product; 1 represents a matrix of the same size as B with all elements being 1.
[0026] Furthermore, the mutation operation includes: the sample generated after the crossover operation. and A sample is randomly selected from the dataset, and a mutation operation is performed on the selected sample to create a new sample. The unselected sample itself is treated as a new sample. .
[0027] Furthermore, the selected samples are subjected to random mutation operations in the following manner: in the selected samples Randomly select some elements from the elements corresponding to the attributes of agents that have a high degree of influence on the output results of the multi-agent system, and superimpose a random number belonging to a preset mutation range onto the value of the selected elements.
[0028] Furthermore, the selected samples are subjected to random mutation operations in the following manner:
[0029]
[0030] in, The sample before the mutation; for The mutated sample; matrices C and E are both matrices of the same size as the adversarial sample. Elements in matrix C are random numbers within a preset mutation interval. Matrix E is a binary matrix composed of 0s and 1s. The element in the i-th row and j-th column of matrix E is E_i. i, j Determined according to the following formula:
[0031]
[0032] in, Let represent the degree of influence of the i-th attribute of agent j on the output of the multi-agent system. The weight threshold is used; rand(0,1) represents a random number between 0 and 1; P c This represents the probability of element mutation.
[0033] Furthermore, the degree of influence of the j-th attribute of agent i on the output of the multi-agent system is obtained as follows: using the training sample set of the multi-agent system to be evaluated and its corresponding result data as training data, optimizing the Gaussian process regression model with anisotropic RBF as the kernel function to obtain hyperparameters with the goal of maximizing marginal likelihood, and using the length scale l in the hyperparameters... d The degree of influence of the reciprocal of the value of agent i as the j-th attribute on the output of the multi-agent system .
[0034] Furthermore, in 2), based on the number of attacks and evaluation metrics, all attacks are divided into exploration, development, and stabilization phases, and the robustness evaluation result is obtained for each phase according to the robustness evaluation function corresponding to that phase.
[0035] Furthermore, the evaluation metrics include the agent's reward, the agent's survival time, and the number of wins; the robustness evaluation function includes... , and :
[0036]
[0037]
[0038]
[0039]
[0040] in, The preset number of attacks; This represents the current number of attacks. , Represents the reward for the intelligent agent. Indicates the lifespan of an intelligent agent. This indicates the number of wins by the agent. Indicates the original sample; Indicates adversarial examples; and These represent the evaluation index x obtained based on the original sample OS and the adversarial sample AS, respectively. It is a non-negative minimum value that is not zero; Percentage of attack progress; Percentage of attack progress;
[0041] During the exploration phase, random selection , and Any one of them yields the corresponding robustness evaluation result; during the development phase, randomly select and Any one of them yields the corresponding robustness evaluation result; in the stable phase, according to The corresponding robustness evaluation results were obtained.
[0042] Further, in 1), the perturbation is a perturbation obtained according to the following method and following a Gaussian distribution: setting at least two different sets of variance parameters within a preset variance threshold range, and adjusting the mean within a preset mean range, introducing L... 0-norm Norm controls noise point density.
[0043] The present invention also provides a technical solution for a robustness evaluation device for multi-agent systems based on black-box attacks: a robustness evaluation device for multi-agent systems based on black-box attacks, comprising a processor, the processor being used to execute a computer program to implement the steps of the robustness evaluation method for multi-agent systems based on black-box attacks as described above. Attached Figure Description
[0044] Figure 1 This is a schematic diagram of the evaluation framework for an implementation of the robustness evaluation method for multi-agent systems based on black-box attacks of the present invention.
[0045] Figure 2 This is a flowchart illustrating an implementation method for the robustness evaluation method of a multi-agent system based on black-box attacks according to the present invention.
[0046] Figure 3 This is a schematic diagram illustrating the attack phase division of the implementation method of the multi-agent system robustness evaluation method based on black-box attack of the present invention.
[0047] Figure 4 This diagram illustrates the selection of adversarial examples in an implementation of the robustness evaluation method for multi-agent systems based on black-box attacks according to the present invention. Detailed Implementation
[0048] This invention, during black-box attacks, optimizes adversarial samples using an optimization algorithm after each attack based on robustness evaluation results. This significantly improves the quality of the adversarial samples, increases the attack strength against the agent, and greatly enhances the accuracy of the robustness evaluation results. This invention solves the technical problem in existing technologies where black-box attacks rely too heavily on the quality of adversarial samples, leading to inaccurate robustness evaluations.
[0049] Implementation methods for robustness evaluation of multi-agent systems based on black-box attacks:
[0050] This implementation method was tested on the highway environment, a simulation platform specifically designed for autonomous driving decision-making. Our environment simulates multi-vehicle interaction, dynamic road conditions, and complex driving tasks. In our experimental scenario, 5 controllable vehicles (agents) and 50 behavior-driven vehicles (agents) are evenly distributed on an 8-lane highway (i.e., a total of 55 agents). The behavior-driven vehicles are divided into three categories: ordinary, aggressive, and conservative.
[0051] In this context, behavior-driven vehicles refer to vehicles not controlled by reinforcement learning algorithms. They move within the environment based on pre-defined agent attributes and are categorized into three types: ordinary, aggressive, and conservative. Controllable vehicles are those capable of executing agent actions under the control of reinforcement learning algorithms; these are the primary evaluation objects in this implementation. The evaluation method of this implementation allows for more accurate robustness assessment of each controllable agent vehicle, facilitating subsequent upgrades, iterations, and maintenance of the multi-agent vehicle system.
[0052] Each experiment has 90 time steps, meaning the maximum length is 90. If all vehicles collide within 90 time steps, the next round of the experiment begins (in this round, num is 0). Agent attributes include starting coordinates (x, y axes), initial velocity, and acceleration. Agent actions include acceleration, constant speed, deceleration, left turn, and right turn. The evaluation metric is the total reward received by the controllable vehicles, num is the number of controllable vehicles that do not collide at the end of each round, and length is the average time it takes for controllable vehicles to avoid collisions.
[0053] A robustness evaluation method for multi-agent systems based on black-box attacks, such as Figure 1 and Figure 2 As shown, the method includes:
[0054] S1. Obtain a batch of training samples X and their corresponding result evaluation indicators Y for training the multi-agent system. Put the training samples X and their corresponding result evaluation indicators Y into the Gaussian process regression model. Obtain the weight matrix of different data dimensions by optimizing the hyperparameters of the kernel function of the Gaussian process regression model.
[0055] In Gaussian process regression, this implementation uses the anisotropic RBF (Radial Basis Function) kernel function, which allows different input dimensions to have independent length scaling parameters. Its mathematical expression is as follows:
[0056]
[0057] in, Both are D-dimensional input vectors; The signal variance is used to control the overall amplitude of the kernel function; d is the length scale of the d-th dimension of the data, used to control the smoothness of that dimension.
[0058] The hyperparameters of the kernel function are optimized by maximizing the marginal likelihood. and l d As shown in the following formula:
[0059]
[0060] Where X represents the training samples. Y is the result evaluation index corresponding to X. n is the number of training samples; K is the kernel matrix, and the elements in K are Ki, Kj ... ij =k(x i , x j ); Let be the noise variance; θ be the set of hyperparameters, including... and l d .
[0061] Length scale l in anisotropic RBF kernel function d This can be interpreted as the weights of dimension d. The length scale l d The smaller the value, the greater the impact of that dimension on the output; conversely, the larger the value, the greater the impact of the length scale l. d The larger the value, the smaller the impact of that dimension on the output.
[0062] Before calculating the weights, the input sample dimension is reduced to 1 dimension, resulting in a total of N×M dimensional data (N being the number of controllable vehicles + behavior-driven vehicles, and M being the attributes of each vehicle). The calculated length scale is still one-dimensional, not two-dimensional. The weight matrix, however, transforms the length scale into two dimensions, with l1 corresponding to w. 1,1 This represents the weight of the first attribute of the first agent. Assuming each agent has two attributes, then w corresponds to l3. 2,1 This represents the weight of the first attribute of the second agent. That is:
[0063]
[0064] Dimension weight is defined as w d The dimension weights are then normalized, as shown in the following mathematical expression:
[0065]
[0066] By using dimension weights, the importance of each dimension is quantified, resulting in the weight matrix W shown below:
[0067]
[0068] in, This represents the degree of influence of the j-th attribute of agent i on the final output result Y.
[0069] S2. Add perturbations to the original sample S to obtain a batch of adversarial samples.
[0070] During the random perturbation initialization process, we adopt a random noise generation strategy based on Gaussian distribution, which generates initial adversarial samples by superimposing perturbation η on the original sample S.
[0071] Specifically, the original sample S is first normalized. In this implementation, the normalization is performed according to the following formula:
[0072]
[0073] Among them, S max and S min These are the maximum and minimum values in the original sample S, respectively; S norm This is the sample after normalizing the original sample S.
[0074] Next, we perturb and superimpose the normalized samples to obtain the adversarial sample S. perturb Its mathematical formula is as follows:
[0075]
[0076] in, This indicates that the mean is μ and the variance is σ. 2 The Gaussian distribution.
[0077] Then, the out-of-bounds data in the adversarial examples is truncated:
[0078]
[0079] The above formula means that S perturb All samples with values less than 0 are forcibly assigned the value 0, and all samples with values greater than 1 are forcibly assigned the value 1. In the final set of adversarial samples, the values of all adversarial samples AS are in the interval [0,1].
[0080] To enhance the diversity of initial adversarial examples, we designed a multimodal perturbation generation strategy, which includes the following steps:
[0081] 1. Set the variance parameter ,in This generates multi-scale random noise.
[0082] 2. Dynamically adjust the mean μ. With diverse perturbation directions, the mean parameter μ follows a uniform distribution on the interval [-0.2, 0.2].
[0083] 3. Introducing L 0-norm Norm control noise point density η density This ensures the sparsity of the perturbation.
[0084] S3. Input the adversarial samples AS from the adversarial sample set into the multi-agent deep reinforcement learning model to obtain the result index corresponding to the adversarial samples; and conduct robustness evaluation based on the obtained result index.
[0085] Specifically, in this embodiment, the multi-agent deep reinforcement learning model has three evaluation metrics after it runs: agent reward, agent survival time, and number of agent wins. The reward is the total reward received by the controllable vehicles, num is the number of controllable vehicles that do not collide after each round of experiments, and length is the average time that the controllable vehicles do not collide.
[0086] Specifically, in this embodiment, different robustness evaluation functions are designed based on the above three evaluation indicators, as shown below:
[0087] 1) Evaluate the robustness of the model using num and length. :
[0088]
[0089] in, This represents the total number of attacks required for a black-box attack. This represents the current number of attacks. and These represent the evaluation metrics of the number of agent wins and the agent survival time obtained based on adversarial examples (AS), respectively. It is 0.15, which means that the attack progress is 15% of T.
[0090] It should be noted that, in this implementation, an attack is considered complete when one execution of "putting all adversarial samples into the multi-agent deep reinforcement learning model from the adversarial sample set" is performed. An agent wins if, under the attack of adversarial samples, the agent makes a reasonable decision (e.g., in a multi-vehicle cooperative navigation scenario, there is no collision with other agents).
[0091] 2) Evaluate the robustness of the model using length and reward. :
[0092]
[0093] in, and These represent the evaluation metrics for agent reward and agent survival time obtained based on adversarial examples (AS), respectively. It is 0.45, which means that the attack progress is 45% of T.
[0094] 3) Evaluate model robustness using rewards. :
[0095]
[0096] in, This represents the agent reward obtained based on the original sample OS and the adversarial sample AS.
[0097] for:
[0098]
[0099] in, ; Indicates the original sample; Indicates adversarial examples; It is a non-negative minimum value that is not zero, in order to prevent the preceding term from being zero.
[0100] In this implementation, based on the increase in the number of attacks and the convergence process of the evaluation metrics, such as Figure 3 As shown, the attack phases are divided based on the degree of decrease in the three metrics: reward, num, and length. The attack process is divided into three phases: exploration, development, and stabilization. In the exploration phase, the attack is... , and Three robustness evaluation functions are used, and one of the three metrics is randomly selected as the robustness metric for this attack; during the development phase, the following is used: and Two robustness evaluation functions are used, and either of the two metrics is randomly selected as the robustness metric for this attack; during the stable phase, only one is used. This serves as an indicator of the robustness of this attack.
[0101] S4. Optimize the adversarial samples to obtain near-globally optimal adversarial samples, preparing for the next attack.
[0102] This implementation method will search for an approximate global optimal solution AS of the adversarial example AS. opt The problem is modeled as an optimization problem, namely, finding the adversarial example that best satisfies the following formula:
[0103]
[0104] in, This refers to the evaluation indicators mentioned above.
[0105] Specifically, this implementation uses a genetic algorithm to optimize adversarial examples, including:
[0106] S41. Select two adversarial examples AS from the adversarial examples using the roulette wheel selection method. i and AS j .
[0107] Specifically, for a given adversarial sample AS n In this regard, the probability of being selected can be determined by the robustness index finally obtained from S3 above, as shown in the following formula:
[0108] in, For the sample The corresponding robustness evaluation results (i.e., the robustness index obtained from S3); This represents the number of adversarial samples in the adversarial sample set.
[0109] Furthermore, the cumulative probability of the i-th individual can be calculated using the following mathematical formula:
[0110] Furthermore, two values are randomly generated in the interval (0,1) to select two adversarial samples.
[0111] Specifically, each sample is mapped to an interval between 0 and 1 based on its probability of being selected. The length of the interval is the probability of that sample being selected, and the right endpoint of the interval is the cumulative probability of that sample. The sample corresponding to the interval where the generated random number falls is selected.
[0112] like Figure 4 As shown, 0.088 and 0.88 are randomly generated. For 0.088, if it falls within the interval (0.08, 0.32], then the AS2 sample corresponding to that interval is selected. Similarly, for 0.88, if it falls within the interval (0.8, 1], then the AS5 sample corresponding to that interval is selected.
[0113] S42. For the two selected adversarial samples AS i and AS j Perform a crossover operation to generate two new adversarial examples. and .
[0114] Two new adversarial examples are generated using the following formula. and :
[0115]
[0116] Among them, ASi and AS j S41 represents the adversarial sample selected from S41; B is a cross-template matrix of the same size as the adversarial sample, and each element in B is a randomly generated 0 or 1. The product of the two matrices is called the Hadamard product, which is calculated by multiplying the corresponding elements of the two matrices. The 1 in the above formula represents a matrix of the same size as B with all elements being 1.
[0117] For example: two adversarial examples AS i and AS j They are respectively:
[0118] AS i = AS i = .
[0119] Template matrices B and 1-B are respectively:
[0120] B= ;1-B= .
[0121] Two new adversarial samples were obtained based on the above calculation. and :
[0122] = ; = .
[0123] S43. Determine the samples generated by the crossover operation. or Has a mutation occurred?
[0124] Specifically, the judgment method is to set a threshold. Then, a random number is generated from (0,1). If this number is less than a threshold, a mutation occurs; otherwise, no mutation occurs. The mathematical expression for this judgment is as follows:
[0125]
[0126] In other words, mutation operations are performed randomly on the samples generated by the crossover operation.
[0127] In this embodiment, the sample is mutated using the dimension matrix W obtained from S1 to obtain a new sample. or Specifically, the mutation is performed according to the following formula:
[0128]
[0129] Where q = 1 or 2; C and E are matrices of the same size as the adversarial sample, the elements in matrix C are random numbers between [0, 0.5], matrix E is a binary matrix composed of 0 and 1, and the element in the i-th row and j-th column of matrix E is E i, j Determined according to the following formula:
[0130]
[0131] in, For elements in the weight matrix W ,Right now ; In this embodiment, the weighted threshold is used. P is the critical value of the top 20% quantile among the elements in the weight matrix W. c This represents the probability of element mutation, and is only true if the dimension weight ranking is within the top 20% and rand(0,1) < P. c Only when the dimension changes will a mutation occur. rand(0,1) represents a random number between 0 and 1.
[0132] That is, only the data corresponding to the top 20% of the weights that have the greatest impact on the output will undergo random mutations.
[0133] Without mutating the sample, the new sample or That is or ;Right now .
[0134] S44. The new sample obtained in S43 The input is fed into a multi-agent system to obtain the corresponding new sample. The agent is rewarded, and a new sample is replaced based on the reward.
[0135] Specifically, if Then the new sample adversarial examples Replacement, similarly, if Then the new sample adversarial examples Replace; otherwise, do not replace.
[0136] Then, determine whether the number of iterations has reached the preset optimization count H / 2; H is the number of samples. If the number of iterations has not reached H / 2, restart execution from S41; otherwise, determine whether the number of attacks has reached the preset attack count T. If the preset attack count T has not been reached, restart execution from S3.
[0137] If the preset number of attacks T is reached, the robustness of the multi-agent deep reinforcement learning model is evaluated based on the robustness evaluation metric.
[0138] Specifically, the smaller the reward value obtained from the attack, the worse the model's robustness. When the reward value is not significantly different from the value before the attack, the model's adversarial robustness is stronger. This results in a more robust multi-agent deep reinforcement learning model. The evaluation process of this invention is applicable to black-box scenarios, requiring no structural or parameter information of the target model, greatly improving the security of the model robustness evaluation process.
[0139] Implementation of a robustness evaluation device for multi-agent systems based on black-box attacks:
[0140] A robustness evaluation apparatus for multi-agent systems based on black-box attacks includes a processor for executing a computer program to implement the steps of the robustness evaluation method for multi-agent systems based on black-box attacks as described above. The specific robustness evaluation method for multi-agent systems based on black-box attacks has been described in sufficient detail above and will not be repeated here.
[0141] Specifically, the processor can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor can also be a processor supporting the Advanced Reduced Instruction Set Machine (ARM) architecture.
[0142] This invention has the following characteristics:
[0143] This invention utilizes an evolutionary algorithm to optimize adversarial examples and employs a multiple robustness evaluation function, which helps avoid getting trapped in local optima when searching for the optimal adversarial example and reduces the perturbation in the search space. Furthermore, we calculate weights for each dimension of the initial data, making the evolutionary module more efficient. In addition, this invention is applicable to black-box scenarios, requiring no structural or parameter information of the target model, significantly improving the security of the robustness evaluation process for multi-agent deep reinforcement learning models.
[0144] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still make modifications to the technical solutions described in the foregoing embodiments without creative effort, or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for evaluating robustness of a multi-agent system based on black-box attack, characterized in that, The method comprises: 1) superimposing perturbations on an original training sample set of a multi-agent system to be evaluated to obtain an adversarial sample set; 2) inputting adversarial samples in the adversarial sample set into the multi-agent system to be evaluated to complete an attack, determining an evaluation index corresponding to the adversarial samples for evaluating robustness according to a running result, and obtaining a robustness evaluation result of the multi-agent system corresponding to the adversarial samples according to the evaluation index; 3) optimizing the adversarial sample set with a deviation between the evaluation index and an expected evaluation index being minimized as an optimization target according to the robustness evaluation result of the multi-agent system; re-executing 2) according to the optimized adversarial sample set until a preset attack number is reached, and taking a robustness evaluation result obtained by the last attack as a final robustness evaluation result.
2. The method of claim 1, wherein, The process of optimizing the adversarial sample set comprises: 31) randomly selecting two adversarial samples AS1 and AS2 from the adversarial sample set; wherein a probability of each adversarial sample being selected is determined according to a robustness evaluation result corresponding to the adversarial sample; 32) performing cross and mutation operations on AS1 and AS2 in turn to generate two new adversarial samples and ; 33) to input into the multi-agent system to be evaluated corresponding reward value q = 1 or 2; If , then replace the sample from the adversarial sample set with the sample ; otherwise, do not replace; represents the agent reward obtained based on the sample 34) re-executing 31) until a preset optimization cycle number is reached.
3. The method of claim 2, wherein, In 31), each adversarial sample the probability of being selected is determined according to the following formula: wherein, is a sample corresponding robustness evaluation results; is the number of adversarial samples in the adversarial sample set.
4. The method of claim 2, wherein, The cross operation is performed on AS1 and AS2 in the following manner: wherein, and are samples generated after cross operation, B is a cross template matrix with the same size as the adversarial sample, each element in B is randomly generated 0 or 1; represents Hadamard product; 1 represents a full 1 matrix with the same size as B and all elements are 1.
5. The method of claim 2, wherein, The mutation operation includes generating a sample after the crossover operation and randomly selecting a sample from among the generated samples and performing a mutation operation on the selected sample as a new sample ; and the sample not selected itself as a new sample .
6. The method of claim 5, wherein, The random mutation operation is performed on the selected sample in the following manner: in the selected sample corresponding to the agent attribute with a high degree of influence on the output result of the multi-agent system, a part of elements are randomly selected, and a random number belonging to a preset mutation interval is superimposed on the numerical value of the selected element.
7. The method of claim 5 or 6, wherein, The random mutation operation is performed on the selected sample in the following manner: wherein, is a sample before mutation; is is a sample after mutation; matrix C and matrix E are both matrices with the same size as the adversarial sample, the elements in matrix C are all random numbers within a preset mutation interval, and matrix E is a binary matrix composed of 0 and 1, the element E i, j is determined according to the following formula: wherein, is the degree of influence of the i-th attribute of agent j on the output result of the multi-agent system; is the weight threshold value; rand(0,1) represents a random number between 0 and 1; P c represents the element mutation probability.
8. The method of claim 7, wherein, The influence degree of the jth attribute of the intelligent agent i on the output result of the multi-agent system is obtained according to the following manner: taking the training sample set of the multi-agent system to be evaluated and the corresponding result data thereof as training data, optimizing a Gaussian process regression model with anisotropic RBF as a kernel function to obtain hyperparameters, taking the reciprocal of the length scale l d in the hyperparameters as the influence degree of the jth attribute of the intelligent agent i on the output result of the multi-agent system .
9. The method of claim 1, wherein, In 2), all attacks are divided into an exploration stage, a development stage and a stable stage according to an attack number and an evaluation index, and a robustness evaluation result is obtained according to a robustness evaluation function corresponding to each stage.
10. The method of claim 9, wherein, The evaluation indexes include a reward of the agent, a survival time length of the agent, and a number of victories of the agent; and the robustness evaluation function includes , and : wherein, is a preset number of attacks; is a current number of attacks; , represents a reward of an agent, represents a survival time of an agent, represents a number of wins of an agent, represents an original sample; represents an adversarial sample; and respectively represent x evaluation indexes based on the original sample OS and the adversarial sample AS; is a non-negative minimum value other than 0; is a percentage of attack progress; is a percentage of attack progress; In the exploration phase, randomly selecting any one of , and gets the corresponding robustness evaluation result; in the development phase, randomly selecting any one of and gets the corresponding robustness evaluation result; in the stable phase, according to gets the corresponding robustness evaluation result.
11. The method of claim 1, wherein, In 1), the perturbation is a perturbation obtained according to the following manner and subject to a Gaussian distribution: at least two groups of different variance parameters are set in a preset variance threshold range, and the mean value is adjusted in a preset mean value range, L 0-norm The norm controls the density of noise points.
12. A black-box attack based multi-agent system robustness evaluation apparatus, comprising a processor, characterized in that, The processor is configured to execute a computer program to implement the steps of the multi-agent system robustness evaluation method based on black-box attack according to any one of claims 1-11.