Multi-agent robust control method and device driven by population iterative optimization

Abstract

The application relates to a multi-agent robust control method and device driven by population iteration optimization, and belongs to the technical field of multi-agent control. The method comprises the following steps: initializing existing disturbance strategies in a disturbance strategy population pool and a target strategy to be trained; sampling a parent disturbance strategy set from the disturbance strategy population pool; performing sequential state disturbance to obtain a current round of disturbance state and collect strategy optimization sample data; updating the target strategy based on the current round of disturbance state; generating a new state disturbance strategy based on the strategy optimization sample data, taking the minimization of the cumulative income of the target multi-agent as an optimization target, and combining a population diversity regular constraint optimization; calculating a strategy distance and updating the disturbance strategy population pool; and repeating iteration until a training termination condition is met, so that a trained robust multi-agent strategy is obtained. The application effectively improves the robustness of the target strategy and ensures stable operation of the target strategy in a disturbance environment.

Description

Technical Field

[0001] This invention relates to the field of multi-agent control technology, and in particular to a population-based iterative optimization-driven multi-agent robust control method and apparatus. Background Technology

[0002] In recent years, multi-agent reinforcement learning has shown broad application prospects in tasks such as cooperative control of unmanned systems and intelligent warehouse scheduling. However, most existing multi-agent reinforcement learning methods are usually based on an ideal assumption: that the agents can accurately acquire the state or observation information of the environment. Under this premise, the cooperative policy trained will often contain noise or be tampered with once deployed in a real open environment due to sensor measurement errors or malicious interference from the outside. In this case, the policy that originally performed well in the ideal environment will significantly degrade in performance or even fail completely. Therefore, how to improve the policy robustness of multi-agent systems under conditions of uncertain or disturbed state information is an important guarantee for this technology to move from simulation to practical application.

[0003] To address the aforementioned issues, some existing technologies attempt to improve policy adaptability through iterative optimization that generates challenging samples and enhances the policy. The core of these methods is to provide the agent with diverse and challenging sample data during the training phase, enabling the agent to optimize under difficult sample conditions, thereby improving its generalization ability and robustness. However, existing methods have two main drawbacks. First, most methods directly learn the optimal state adjustment mode in the state space. Since the state space of multi-agent systems is typically high-dimensional and large-scale, direct optimization in a high-dimensional state space faces challenges such as high optimization difficulty and slow convergence speed, making it difficult to generate high-quality training samples. Second, existing methods often employ fixed adjustment patterns during training, leading to the agent policy easily overfitting to a specific type of input bias. When faced with novel input biases not encountered during training, the policy's robustness is poor, failing to meet the needs of dealing with diverse and unknown input biases in practical applications. Therefore, how to provide high-quality and diverse challenging samples for multi-agent policy optimization and reduce the learning difficulty of sample generation strategies is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] Therefore, it is necessary to provide a robust multi-agent control method and device driven by population iterative optimization that can reduce the difficulty of sample generation and provide diverse and challenging samples to address the above-mentioned technical problems.

[0005] A population-based iterative optimization-driven multi-agent robust control method, the method comprising: Step 1: Initialize multiple existing perturbation policies in the perturbation policy population pool, and initialize the target policy to be trained at the same time; Step 2: Sample the parent perturbation strategy set for the current round from the perturbation strategy population pool; Step 3: Perform sequential state perturbation based on the parent perturbation strategy set to obtain the perturbation state of the current round, and at the same time collect the strategy optimization sample data of the current round of iterative interaction; Step 4: Robustly train and update the target policy based on the current perturbation state; Step 5: Optimize the sample data based on the strategy, with the goal of minimizing the cumulative benefit of the target multi-agent, and generate a new state perturbation strategy by combining population diversity regularization constraints. Step 6: Calculate the policy distance between the new state perturbation policy and each parent perturbation policy in the parent perturbation policy set. When the policy distance is greater than a preset distance threshold, add the new state perturbation policy to the perturbation policy population pool; otherwise, eliminate the new state perturbation policy to obtain an updated perturbation policy population pool. Step 7: Based on the updated perturbation policy population pool, repeat steps 2 to 6 to complete multiple rounds of population iteration until the preset training termination condition is met, and obtain the trained robust multi-agent policy.

[0006] On the other hand, a population-iterative optimization-driven multi-agent robust control device is also provided, comprising: The policy initialization module is used to initialize multiple existing perturbation policies in the perturbation policy population pool, and at the same time initialize the target policy to be trained. The parent strategy sampling module is used to sample the set of parent perturbation strategies for the current round from the perturbation strategy population pool; The sequential perturbation generation and data acquisition module is used to execute sequential state perturbation based on the parent perturbation strategy set to obtain the perturbation state of the current round, and at the same time collect the strategy optimization sample data of the current round of iterative interaction. The target policy robust training and update module is used to perform robust training and update of the target policy based on the current perturbation state. The new perturbation strategy generation module is used to optimize sample data based on the strategy, with the goal of minimizing the cumulative benefit of the target multi-agent, and to generate a new state perturbation strategy by combining population diversity regularization constraints. The strategy distance calculation and population update module is used to calculate the strategy distance between the new state perturbation strategy and each parent perturbation strategy in the parent perturbation strategy set. When the strategy distance is greater than a preset distance threshold, the new state perturbation strategy is added to the perturbation strategy population pool; otherwise, the new state perturbation strategy is eliminated to obtain the updated perturbation strategy population pool. The population iteration loop control module is used to repeatedly execute the process from the parent policy sampling module to the policy distance calculation and population update module based on the updated perturbation policy population pool, to complete multiple rounds of population iteration until the preset training termination condition is met, and obtain the trained robust multi-agent policy.

[0007] Compared with existing technologies, the population-iterative optimization-driven multi-agent robust control method and apparatus provided by this invention have the following advantages: 1. By initializing the perturbation policy population pool and performing multiple rounds of population iteration, combined with sequential state perturbation to generate diverse perturbation states, the problem of single and incomplete coverage of perturbation scenarios in traditional multi-agent robust control is effectively solved. It can more comprehensively simulate the perturbation environment that the target multi-agent may face, significantly improve the robustness of the target policy, and ensure that it can still operate stably under complex perturbation scenarios.

[0008] 2. With minimizing the cumulative benefit of the target multi-agent as the optimization objective, new state perturbation strategies are generated by combining population diversity regularization constraints. The parent perturbation strategy set is updated by filtering through policy distance, which avoids the homogenization of perturbation strategies and ensures the diversity and effectiveness of the perturbation strategy population. This allows the perturbations generated in each iteration to accurately target the weak links of the target strategy, thereby improving the targeting and efficiency of robust training.

[0009] 3. A closed-loop iterative process of "perturbation generation, robust training, policy optimization, and population update" was constructed. From the initialization of the perturbation policy population and perturbation generation to the training of the target policy and population iteration, the entire process can be completed automatically without manual intervention, simplifying the implementation process of multi-agent robust control, improving the practicality and operability of the method, and reducing the implementation cost of multi-agent robust control.

[0010] 4. By continuously optimizing the perturbation strategy population pool through multiple rounds of population iteration, and continuously updating the target strategy, the target strategy can adapt to diverse perturbation scenarios during the iteration process, gradually improve its anti-perturbation ability, and finally obtain a robust multi-agent strategy after training. Compared with the traditional non-iterative perturbation training method, the training effect is more stable and the robustness is significantly improved. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention, and those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating the population-iterative optimization-driven multi-agent robust control method provided in Example 1; Figure 2 This is a structural block diagram of the multi-agent robust control device driven by population iterative optimization in Example 2.

[0013] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0015] It should be noted that in this invention, the use of terms such as "first," "second," etc., is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0016] It is understood that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0017] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0018] Example 1 like Figure 1 As shown, a population-based iterative optimization-driven multi-agent robust control method is provided, comprising the following steps: Step 1: Initialize multiple existing perturbation policies in the perturbation policy population pool, and initialize the target policy to be trained.

[0019] Step 2: Sample the parent perturbation policy set for the current round from the perturbation policy population pool.

[0020] Step 3: Perform sequential state perturbation based on the parent perturbation policy set to obtain the perturbation state of this round, and at the same time collect policy optimization sample data of the current round of iteration interaction.

[0021] Step 4: Robustly train and update the target policy based on the current perturbation state.

[0022] Step 5: Based on the policy optimization sample data, with the goal of minimizing the cumulative benefit of the target multi-agent, and combined with the population diversity regularization constraint, generate a new state perturbation policy.

[0023] Step 6: Calculate the policy distance between the new state perturbation policy and each parent perturbation policy in the parent perturbation policy set. When the policy distance is greater than the preset distance threshold, add the new state perturbation policy to the perturbation policy population pool; otherwise, eliminate the new state perturbation policy and obtain the updated perturbation policy population pool.

[0024] Step 7: Based on the updated perturbation policy population pool, repeat steps 2 to 6 to complete multiple rounds of population iteration until the preset training termination condition is met, and obtain the trained robust multi-agent policy.

[0025] In the aforementioned population-based iterative optimization-driven multi-agent robust control method, a perturbation policy population pool is constructed and multiple rounds of population iterative optimization are carried out. Perturbation states are generated by combining sequential state perturbation. The perturbation policy is optimized with the goal of minimizing the cumulative benefit of the target multi-agent. At the same time, the parent perturbation policy set is updated based on policy distance. This can effectively reduce the optimization difficulty of perturbation generation in high-dimensional space, quickly generate high-quality perturbation training samples, and avoid the overfitting problem of the target policy caused by a single perturbation mode. It comprehensively improves the robustness and generalization ability of the target policy in the perturbation environment, and solves the technical problems of high optimization difficulty, low sample quality and insufficient policy robustness of traditional methods.

[0026] Before implementing step 1, the multi-agent system is modeled as a distributed partially observed Markov decision process under state perturbation, using tuples. To represent, where, To disturb the intelligent agent system, For the target intelligent agent system, For a set of perturbation agents, For the target set of intelligent agents, This represents the current real-world state set of the target agent's environment. For the set of perturbation states, To disturb the motion space, where, To disturb the intelligent agent The One action, To control the action space for the target, Let be the state transition probability. Disturb the observation space of the intelligent agent. For the target intelligent agent's observation space, The reward function for the perturbation agent. Let the reward function be the target agent. As a return discount factor, This is the perturbation threshold. The perturbation agent is based on the perturbation direction vector. And through the state perturbation function The state after the disturbance is generated The target agent is based on the perturbation state. Output control action and gain immediate benefits by interacting with the environment. The benefit of perturbing the intelligent agent is In other words, the perturbation agents aim to minimize the cumulative gain of the target agent. The joint perturbation policy of all perturbation agents is denoted as... The joint policy of all target agents is denoted as The optimization objective of the perturbed agent is to minimize the state-value function of the target agent. or The optimization objective of the target agent is to maximize or Based on this, diverse perturbation strategies are generated through population iteration. To enable the target agent's strategy It can maintain high returns under different perturbations, thereby improving its robustness.

[0027] In step 1, the perturbation policy population pool stores multiple policies that can perturb the state of the environment in which the target multi-agent exists. Each perturbation policy corresponds to a state perturbation method and is represented by a set of learnable neural network parameters. The target policy to be trained refers to the initial policy used to control the agents to perform cooperative tasks, and its network parameters are randomly initialized.

[0028] In the specific implementation of step 1, the maximum capacity of the perturbation strategy population pool is first set. Initialize the perturbation strategy population pool. , which includes A perturbation agent, initially a perturbation agent Can be less than A positive integer is represented as: In the perturbation strategy population pool In, each perturbation strategy It is a multi-agent joint strategy used to generate perturbation direction quantities for each perturbation agent.

[0029] At the same time, initialize the target policy to be trained. Its network parameters are randomly initialized. The target policy adopts the policy network structure in multi-agent reinforcement learning, which is used to output the control actions of each agent based on the current state or the state after being disturbed.

[0030] Both the perturbation policy and the target policy adopt the Bayesian product policy network. During initialization, the network parameters of the perturbation policy and the target policy also need to be initialized.

[0031] The perturbation strategies include existing perturbation strategies, parent perturbation strategies, and new state perturbation strategies, and their Bayesian product strategy network expression is as follows: ; The Bayesian product policy network expression for the objective policy is: ; In the formula, This represents the joint policy distribution of the perturbation policy; Represents the joint policy distribution of the target policy; Represents the set of perturbed agents; Represents the set of target intelligent agents; This represents the perturbation direction quantity output by the perturbation agent; This represents the control action output by the target intelligent agent; Indicates the current actual state; Indicates the first The current true state of each perturbed agent; Indicates a disturbance state; In the perturbation strategy, the first The set of parent actions of a perturbed agent; Indicating the first in the target strategy The parent action set of each target intelligent agent; Indicates the first The perturbation direction quantity output by a perturbation agent; Indicates the first The control actions output by the target intelligent agent; In the perturbation strategy, the first Local perturbation strategies of individual perturbation agents; Indicating the first in the target strategy Local control strategies for individual target intelligent agents; The product symbol is used to represent the product.

[0032] Furthermore, the state value function and action value function of the perturbed agent are defined, with the following expressions: ; ; In the formula, Indicates the first The state value function of a perturbed intelligent agent; This represents the expectation operation; Indicates the first A perturbation agent in The benefits at any given moment; express The true state at any given moment; Indicates the first Action value function of a perturbation agent.

[0033] Define the state-value function and action-value function of the target agent, with the following expressions: ; ; In the formula, Indicates the first The state-value function of a target intelligent agent; Indicates the first The perturbation state of the target intelligent agent; Indicates the first A target intelligent agent in The benefits at any given moment; Indicates the first Action value function for a target agent.

[0034] This step establishes a complete infrastructure for subsequent population iteration and robust training by completing the initialization of the dual-policy network and value function, ensuring the standardized execution of perturbation generation and policy optimization.

[0035] In step 2, the parent perturbation policy set refers to a group of high-quality perturbation policies selected from the perturbation policy population pool for training the target policy in the current round. The number of parent perturbation policies is denoted as... Its value is less than the number of existing policies in the population pool. The sampling is based on the fitness index of each perturbation policy. The higher the fitness, the stronger the ability of the perturbation policy to disrupt the target policy, and the greater the probability of it being selected.

[0036] In the specific implementation of step 2, the perturbation strategy population pool is first calculated. Each existing perturbation strategy The fitness index is expressed as: ; In the formula, Indicates the first Fitness metrics for existing perturbation strategies; Indicates the first One existing perturbation strategy; This represents the expectation operation; This represents the return discount factor; This represents the immediate benefit to the target intelligent agent; Indicate the target strategy; The time value indicates the moment. A higher fitness index indicates that the perturbation strategy can reduce the cumulative return of the target strategy.

[0037] Calculate the sampling probability distribution based on the fitness index. The expression for the sampling probability distribution is: ; In the formula, Indicates the first The sampling probability of an existing perturbation strategy; This represents the number of existing perturbation strategies in the perturbation strategy population pool. Based on this probability distribution, ... The probability distribution-based sampling strategy for selecting the parent generation is used to... The probability is randomly initialized with a new perturbation policy and added to the parent set. Sampling is repeated until a result is obtained. Each parent perturbation strategy constitutes the parent perturbation strategy set for the current round. .

[0038] This step uses fitness-weighted sampling to screen for high-quality perturbation strategies, improves the perturbation quality of parent strategies, and provides a high-quality strategy foundation for subsequent sequential perturbation generation.

[0039] In step 3, sequential state perturbation refers to applying perturbation to the current environmental state sequentially based on each parent perturbation policy in the parent perturbation policy set, generating perturbed state samples. The perturbation process follows a preset norm constraint to ensure that the perturbed state remains within a reasonable range. Simultaneously, during the interaction with the environment, policy optimization sample data required for policy optimization is collected, including the state, the perturbed state, actions, and rewards.

[0040] In the specific implementation of step 3, the current real state of the environment in which the target agent is located is first obtained. Disturbance strategy set of the parents Each parent perturbation strategy in the algorithm is based on the current real state of the perturbation agent. After policy space optimization, the output perturbation direction is calculated. .

[0041] Then, define the set of policies that can be allowed to jointly perturb: ; In the formula, This represents the set of policies that can be jointly perturbed; Represents the state perturbation function; Represents the set of joint perturbations of permissible states; Indicates the first The perturbation strategy of a perturbation agent; Indicating the first in the target strategy Local control strategies for a perturbed intelligent agent.

[0042] This set defines the legal boundaries at the policy level for the entire perturbation generation process, and it is combined with the permissible state perturbation set at the state level. A one-to-one constraint relationship is formed, where The norm constraint range of the state perturbation is defined, and This constraint is then mapped to the policy level, ensuring that each subsequently generated perturbation policy corresponds to a state perturbation that satisfies the norm constraint. Furthermore, this state perturbation can be ensured through an equality relationship that the Bayesian product output of the perturbation policy is consistent with the Bayesian product output of the target policy under the perturbation state, thereby guaranteeing the legality and specificity of the perturbation policy. This set is a theoretical constraint definition and does not directly participate in subsequent calculations. The process of generating perturbation state samples by the subsequent state perturbation function is executed under the constraints of this set, and there is no need to repeatedly refer to this symbol.

[0043] Based on the constraints of this set, perturbation state samples are calculated within the predefined norm constraint space using a state perturbation function. The expression for the state perturbation function is as follows: ; The constraints are: st ; In the formula, Represents the state perturbation function; This represents the perturbation direction quantity output by the perturbation agent; It represents the current true state of the environment in which the target intelligent agent is located; Indicates a disturbance state; Indicates the current true state Centered on a preset disturbance threshold The space is constrained by a predefined norm for the radius; Indicate the target strategy; Indicates the transpose operation; The norm operation is represented. The state perturbation function guarantees that the perturbed state lies within the allowed norm sphere and maximizes the output change of the target policy.

[0044] The predefined norm constraint space is Norm-constrained space The set of joint perturbations of admissible states corresponding to the norm constraint space is: ; And the following constraints are met: ; In the formula, Represents the set of joint perturbations of permissible states; Indicates the number of perturbed agents; Indicates the first The perturbation function of a perturbed agent; Indicates the first Current true state of the target intelligent agent Centered on a preset disturbance threshold radius Norm-constrained space.

[0045] Simultaneously, while generating perturbation state samples, policy optimization sample data of the current round of iteration interaction are collected, including the original state, perturbation state, actions of each agent, environmental feedback benefits, and the state at the next moment, and stored in the population policy experience pool.

[0046] This step replaces high-dimensional state space optimization with policy space optimization, reducing the difficulty of perturbation generation. At the same time, it generates effective perturbation samples and collects training data, balancing perturbation quality and data acquisition efficiency.

[0047] In step 4, robust training update refers to using the perturbed state sample as input to the target policy and updating the network parameters of the target policy according to the multi-agent reinforcement learning algorithm, so that it can still obtain a high cumulative reward under the perturbed state, thereby improving robustness.

[0048] In the specific implementation of step 4, a small batch of perturbation states are randomly sampled from the population strategy experience pool. Target Strategy The joint policy is optimized based on the aforementioned Bayesian product policy network, and the joint control action is output. .

[0049] The optimization of the objective policy employs the proximal policy optimization method, with the policy gradient loss function being: ; In the formula, Represents the policy gradient loss of the target policy; Indicates the number of target intelligent agents; Indicates the network parameters of the target policy; Indicates the first The old parameters of the round-iteration target strategy; Indicates the first The distribution of old states in the round-iteration target strategy; This indicates the current target strategy that needs to be updated. Indicates the first The old perturbation strategy of the round-iteration target strategy; Indicates the first The perturbation direction quantity output by a target intelligent agent; Indicating the first in the target strategy The parent action set of each target intelligent agent; Indicates the first Round of iteration The policy advantage function corresponding to each target agent; This represents the truncation function; This indicates that the strategy updates the truncation coefficient.

[0050] Among them, the Round of iteration The policy advantage function expression for each target agent is: ; In the formula, Indicates the first During the first iteration A target intelligent agent in The advantage function at time; This represents the return discount factor; Represents the network parameters of the objective value function; This represents the generalized advantage estimation coefficient; Indicates the first A target intelligent agent in The benefits at any given moment; This represents the state-value function corresponding to the target policy; Indicates the first A target intelligent agent in The state of disturbance at any given moment; Indicates the first A target intelligent agent in The state of disturbance at any given moment.

[0051] The network update expression for the state-value function of the target policy is: ; In the formula, Represents the loss of the target policy value function; Indicates the first A target intelligent agent from Accumulated discounts starting from the moment you enter the game.

[0052] By minimizing the aforementioned loss function, the target policy network parameters are updated via backpropagation, enabling it to still output high-yield actions under diverse perturbation states.

[0053] This step drives iterative optimization of the target policy through perturbation samples, enabling the target policy to gradually adapt to the perturbation environment and continuously improve its anti-disturbance and stable operation capabilities.

[0054] In step 5, the new state perturbation strategy refers to the new strategy obtained by updating the current perturbation strategy through policy gradient. Its optimization objectives include two aspects: first, minimizing the cumulative benefit of the target strategy, i.e., increasing the destructiveness of the perturbation; second, maintaining the difference between the new strategy and the existing strategies in the parent strategy set, so as to enrich the diversity of the population pool.

[0055] In the specific implementation of step 5, policy optimization sample data is first sampled from the population policy experience pool, including the original state, perturbation direction, and reward. The new state perturbation policy is then implemented. Similarly, a Bayesian product strategy network is used for modeling, and the formula is detailed in step 1.

[0056] The optimization of the new perturbation policy involves two losses: policy gradient loss and population diversity regularization loss. The expression for calculating the policy gradient loss is as follows: ; In the formula, This represents the policy gradient loss of the perturbation policy; Indicates the number of perturbed agents; Indicates the first The distribution of old states in the round-iteration perturbation strategy; Network parameters representing the new state perturbation strategy; Network parameters representing the old perturbation strategy; This indicates the new state perturbation strategy to be updated; Indicates the first The old perturbation strategy corresponding to the round of iteration; Indicates the first The perturbation direction quantity output by a perturbation agent; In the perturbation strategy, the first The set of parent actions of a perturbed agent; Indicates the first Round of iteration The policy advantage function corresponding to each perturbation agent; This represents the truncation function; This indicates that the strategy updates the truncation coefficient.

[0057] Among them, the Round of iteration The policy advantage function expression for each perturbation agent is: ; In the formula, Indicates the first During the first iteration A perturbation agent in The advantage function at time; This represents the return discount factor; This represents the generalized advantage estimation coefficient; Indicates the first A perturbation agent in The payoff at any given moment is the negative of the immediate environmental payoff obtained by the target multi-agent; This represents the state-value function corresponding to the perturbation strategy; Indicates the first A perturbation agent in The current real state at any given moment; Indicates the first A perturbation agent in The current real state at any given moment. The state-value function network update expression for the perturbation policy is: ; In the formula, This represents the loss of the perturbation policy value function; Indicates the first A perturbation agent from Accumulated discounts starting from the moment you enter the game.

[0058] The population diversity regularization loss is used to ensure that newly generated perturbation policies remain different from those in the parent policy set, and its expression is: ; In the formula, Indicates the regularized loss of population diversity; Indicates the number of parental perturbation strategies; Indicates the first The first of the parental adjustment strategies The perturbation policy distribution of each perturbation agent; Indicates the first The average perturbation policy distribution of each perturbation agent; Represents the KL divergence operation; This represents the current true state of the environment in which the target multi-agent exists. Among them, .

[0059] The total optimization loss is a weighted sum of the policy gradient loss and the diversity regularization loss, expressed as: ; In the formula, This represents the total optimization loss; This represents the diversity loss weighting coefficient.

[0060] Update network parameters by minimizing the total optimization loss. The newly generated state perturbation strategy is obtained.

[0061] This step introduces a diversity regularization term into the optimization objective, ensuring that the new perturbation strategy is not only highly perturbative but also significantly different from existing strategies in the population. This provides more diverse challenge samples for the target strategy and avoids overfitting of the target strategy due to strategy homogenization.

[0062] In step 6, policy distance is used to measure the degree of difference between two perturbation policies, calculated using JS divergence. A preset distance threshold is used to determine whether the new policy has sufficient novelty. The population pool has a capacity limit; when the limit is reached, it is updated according to the replacement rules.

[0063] In the specific implementation of step 6, the preset capacity limit of the perturbation strategy population pool is first obtained. For the new state perturbation strategy and the parent perturbation strategy set, each parent perturbation strategy... Calculate the policy distance between the two. The expression for calculating the policy distance is: ; In the formula, Indicates the perturbation strategy of the offspring state Perturbation strategy of parent generation in the population pool The strategic distance between them; This indicates the number of disturbers in the current disturbed population pool; This indicates the number of disturbers in the parent disturber population pool; Indicates JavaScript divergence operation; In the policy of perturbing the child state, the first... The policy distribution of the perturbation agents; This indicates the first generation in the parent perturbation strategy. The policy distribution of the perturbation agents.

[0064] After calculating the policy distances between the new policy and all parent perturbation policies, the minimum value is selected and recorded as the minimum policy distance. .

[0065] The preset distance threshold is denoted as If the minimum strategy distance Greater than the preset distance threshold This indicates that the new strategy is sufficiently different from the parent perturbation strategy. Therefore, the number of perturbation agents in the current population pool is further determined. If the number of perturbation agents in the perturbation strategy population pool is... Maximum capacity not reached This indicates that the perturbation strategy population pool is not full, so the new state perturbation strategy is directly added to the population pool.

[0066] If the minimum strategy distance Greater than the preset distance threshold However, the number of perturbation agents in the perturbation strategy population pool Maximum capacity has been reached. This indicates that although the new strategy is sufficiently different from the parent perturbation strategy, the perturbation strategy population pool is already full. Therefore, the parent perturbation strategy with the smallest policy distance to the new state perturbation strategy is found, and the new state perturbation strategy replaces the one with the smallest policy distance. The corresponding parent perturbation strategy. After the population pool is updated, the new population pool will be used for the next iteration.

[0067] This step uses strategy distance filtering and dynamic replacement rules to achieve the survival of the fittest in the perturbation strategy population, continuously improving the overall perturbation quality and diversity of the population and preventing population degradation.

[0068] In step 7, multi-round population iteration refers to the process of repeatedly executing steps 2 to 6: parent sampling, perturbation generation, target policy training, new perturbation policy generation, and population update. The preset training termination condition can be reaching the maximum number of iteration rounds, or the target policy's robustness performance index under the verification perturbation reaching a preset threshold.

[0069] In the specific implementation of step 7, a maximum number of training rounds is set, and a population perturbation update interval is also set. That is, every time The population pool is updated only once per round, i.e., step 6; in other rounds, only the perturbation generation and target policy training from steps 2 to 5 are performed.

[0070] The specific execution flow for each round is as follows: Step 2: Sample the parent perturbation policy set from the current population pool; Step 3: Generate perturbation state samples and collect sample data; Step 4: Perform robust training and update of the target policy; If the current round number meets the update interval condition, Step 5: Generate a new perturbation policy and Step 6: Update the population pool.

[0071] Repeat the above process until the maximum number of training rounds is reached, or until the robustness performance metric of the target policy is detected to stably exceed the preset threshold. At this point, the target policy updated last time is output as the robust multi-agent policy after training is complete.

[0072] This step continuously optimizes the perturbation strategy and the target strategy through a closed-loop iterative mechanism, enabling the target strategy to adapt and improve under diverse and strong perturbations, gradually strengthening the robustness of the target strategy, and ultimately obtaining a robust multi-agent strategy that can operate stably in complex perturbation environments.

[0073] It should be understood that, although this embodiment Figure 1The steps are shown sequentially as indicated by the arrows, but they are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are performed; they can be executed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0074] Example 2 Based on the population-iterative optimization-driven multi-agent robust control method in Example 1, this example discloses a population-iterative optimization-driven multi-agent robust control device, such as... Figure 2 As shown, the multi-agent robust control device driven by population iterative optimization includes: a policy initialization module 801, a parent policy sampling module 802, a sequential perturbation generation and data acquisition module 803, a target policy robust training and update module 804, a new perturbation policy generation module 805, a policy distance calculation and population update module 806, and a population iterative loop control module 807, wherein: The policy initialization module 801 is used to initialize multiple existing perturbation policies in the perturbation policy population pool, and at the same time initialize the target policy to be trained.

[0075] The parent strategy sampling module 802 is used to sample the parent perturbation strategy set for the current round from the perturbation strategy population pool.

[0076] The sequential perturbation generation and data acquisition module 803 is used to execute sequential state perturbations based on the parent perturbation policy set to obtain the perturbation state of the current round, and at the same time collect the policy optimization sample data of the current round of iteration interaction.

[0077] The target policy robust training and update module 804 is used to perform robust training and update of the target policy based on the current perturbation state.

[0078] The new perturbation policy generation module 805 is used to generate new state perturbation policies based on policy optimization sample data, with the goal of minimizing the cumulative benefit of the target multi-agent, and combining population diversity regularization constraints.

[0079] The strategy distance calculation and population update module 806 is used to calculate the strategy distance between the new state perturbation strategy and each parent perturbation strategy in the parent perturbation strategy set. When the strategy distance is greater than the preset distance threshold, the new state perturbation strategy is added to the perturbation strategy population pool; otherwise, the new state perturbation strategy is eliminated, and the updated perturbation strategy population pool is obtained.

[0080] The population iteration loop control module 807 is used to repeatedly execute the process from the parent policy sampling module to the policy distance calculation and population update module based on the updated perturbation policy population pool, to complete multiple rounds of population iteration until the preset training termination condition is met, and to obtain the trained robust multi-agent policy.

[0081] In this embodiment, the specific working process and working principle of the policy initialization module 801, parent policy sampling module 802, sequential perturbation generation and data acquisition module 803, target policy robust training and update module 804, new perturbation policy generation module 805, policy distance calculation and population update module 806, and population iteration loop control module 807 are the same as those in Embodiment 1, and therefore will not be described again in this embodiment. Each unit module can be implemented entirely or partially through software, hardware, or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above unit modules.

[0082] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0083] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0084] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A population-based iterative optimization-driven multi-agent robust control method, characterized in that, The method includes: Step 1: Initialize multiple existing perturbation policies in the perturbation policy population pool, and initialize the target policy to be trained at the same time; Step 2: Sample the parent perturbation strategy set for the current round from the perturbation strategy population pool; Step 3: Perform sequential state perturbation based on the parent perturbation strategy set to obtain the perturbation state of the current round, and at the same time collect the strategy optimization sample data of the current round of iterative interaction; Step 4: Robustly train and update the target policy based on the current perturbation state; Step 5: Optimize the sample data based on the strategy, with the goal of minimizing the cumulative benefit of the target multi-agent, and generate a new state perturbation strategy by combining population diversity regularization constraints. Step 6: Calculate the policy distance between the new state perturbation policy and each parent perturbation policy in the parent perturbation policy set. When the policy distance is greater than a preset distance threshold, add the new state perturbation policy to the perturbation policy population pool; otherwise, eliminate the new state perturbation policy to obtain an updated perturbation policy population pool. Step 7: Based on the updated perturbation policy population pool, repeat steps 2 to 6 to complete multiple rounds of population iteration until the preset training termination condition is met, and obtain the trained robust multi-agent policy.

2. The population-based iterative optimization-driven multi-agent robust control method according to claim 1, characterized in that, In step 2, the parent perturbation policy set for the current round is sampled from the perturbation policy population pool, including: Calculate the fitness index for each existing perturbation strategy, and calculate the sampling probability distribution based on the fitness index; Sampling is performed according to the sampling probability distribution to obtain the parent perturbation strategy set for the current round; The fitness index is expressed as follows: ； The expression for the sampling probability distribution is: ； In the formula, Indicates the first The sampling probability of an existing perturbation strategy; Indicates the first Fitness metrics for existing perturbation strategies; Indicates the first One existing perturbation strategy; This represents the number of existing perturbation strategies in the perturbation strategy population pool; This represents the expectation operation; This represents the return discount factor; This represents the immediate benefit to the target intelligent agent; Indicate the target strategy; Indicates the time.

3. The population-based iterative optimization-driven multi-agent robust control method according to claim 1, characterized in that, In step 3, sequential state perturbation is performed based on the parent perturbation strategy set, including: The current real state of the environment in which the target multi-agent is located is obtained. The parent perturbation policy set is optimized in the policy space based on the current real state and then calculated in the state perturbation function within the preset norm constraint space to obtain the perturbation state of this round. The expression for the state perturbation function is: ； The constraints are: s.t ; In the formula, Represents the state perturbation function; This represents the direction of the perturbation output by the perturbation agent; It represents the current true state of the environment in which the target intelligent agent is located; Indicates a disturbance state; Indicates the current true state Centered on a preset disturbance threshold The space is constrained by a predefined norm for the radius; Indicate the target strategy; Indicates the transpose operation; Represents norm operations.

4. The population-based iterative optimization-driven multi-agent robust control method according to claim 3, characterized in that, The preset norm constraint space is Norm-constrained space, the The set of joint perturbations of admissible states corresponding to the norm constraint space is: ； And the constraints are satisfied: ； In the formula, Represents the set of joint perturbations of permissible states; Indicates the number of perturbed agents; Indicates the first The perturbation function of a perturbed agent; Indicates the first The current true state of each target intelligent agent; Indicates the first Current true state of the target intelligent agent Centered on a preset disturbance threshold radius Norm-constrained space; The product symbol is used to represent the product.

5. The population-based iterative optimization-driven multi-agent robust control method according to any one of claims 1 to 4, characterized in that, In step 5, a new state perturbation strategy is generated by combining population diversity regularization constraints, including: The policy gradient loss is calculated based on the policy optimization sample data. The policy gradient loss is then weighted and summed with the population diversity regularization loss to obtain the total optimization loss. The new state perturbation policy is then updated based on the total optimization loss. The expression for the policy gradient loss is: ； The expression for the population diversity regularization loss is: ； The expression for the total optimization loss is: ； In the formula, This represents the policy gradient loss of the perturbation policy; Indicates the regularized loss of population diversity; This represents the total optimization loss; This represents the diversity loss weighting coefficient; Indicates the number of perturbed agents; Indicates the first The distribution of old states in the round-iteration perturbation strategy; Network parameters representing the new state perturbation strategy; Network parameters representing the old perturbation strategy; This indicates the new state perturbation strategy to be updated; Indicates the first The old perturbation strategy corresponding to the round of iteration; Indicates the first The current true state of each target intelligent agent; This represents the expectation operation; Indicates the first The perturbation direction quantity output by each perturbation agent; In the perturbation strategy, the first The parent action set of a perturbed agent; Indicates the first Round of iteration The policy advantage function corresponding to each perturbation agent; This represents the truncation function; This indicates that the strategy updates the truncation coefficient; Indicates the number of parental perturbation strategies; Indicates the first The first of the parental adjustment strategies The perturbation policy distribution of each perturbation agent; Indicates the first The average perturbation policy distribution of each perturbation agent; Represents the KL divergence operation; It represents the current true state of the environment in which the target multi-agent exists.

6. The population-based iterative optimization-driven multi-agent robust control method according to claim 5, characterized in that, The first Round of iteration The policy advantage function expression for each perturbation agent is: ； In the formula, Indicates the first During the first iteration A perturbation agent in The advantage function at time; This represents the return discount factor; This represents the generalized advantage estimation coefficient; Indicates the first A perturbation agent in The payoff at any given moment is the negative of the immediate environmental payoff obtained by the target multi-agent; This represents the state-value function corresponding to the perturbation strategy; Indicates the first A perturbation agent in The current real state at any given moment; Indicates the first A perturbation agent in The current real state at any given moment.

7. The population-based iterative optimization-driven multi-agent robust control method according to any one of claims 1 to 4, characterized in that, In step 6, the expression for calculating the policy distance is: ； In the formula, Indicates the new state perturbation strategy Disturbance strategies of the parents The strategic distance between them; This indicates the parent's perturbation strategy; Indicates a perturbation agent; Indicates JavaScript divergence operation; In the new state perturbation strategy, the first... The policy distribution of the perturbation agents; This indicates the first generation in the parent perturbation strategy. The policy distribution of the perturbation agents; It represents the current true state of the environment in which the target multi-agent exists.

8. The population-based iterative optimization-driven multi-agent robust control method according to any one of claims 1 to 4, characterized in that, Both the perturbation strategy and the target strategy are jointly optimized using a Bayesian product strategy network. The Bayesian product policy network expression for the perturbation strategy is as follows: ； The Bayesian product policy network expression for the target policy is: ； In the formula, This represents the joint policy distribution of the perturbation policy; Represents the joint policy distribution of the target policy; Represents the set of perturbed intelligent agents; Represents the set of target intelligent agents; This represents the direction of the perturbation output by the perturbation agent; This represents the control action output by the target intelligent agent; It represents the current true state of the environment in which the target multi-agent exists; Indicates the first The current true state of each perturbed agent; Indicates a disturbance state; In the perturbation strategy, the first The parent action set of a perturbed agent; Indicating the first in the target strategy The parent action set of each target intelligent agent; Indicates the first The perturbation direction quantity output by each perturbation agent; Indicates the first The control actions output by the target intelligent agent; In the perturbation strategy, the first Local perturbation strategies of individual perturbation agents; Indicating the first in the target strategy Local control strategies for individual target intelligent agents; The product symbol is used to represent the product.

9. The population-based iterative optimization-driven multi-agent robust control method according to any one of claims 1 to 4, characterized in that, In step 6, when the policy distance is greater than a preset distance threshold, the new state perturbation policy is added to the perturbation policy population pool, including: Obtain the preset capacity limit of the perturbation strategy population pool. ; Calculate the policy distance between the new state perturbation policy and each parent perturbation policy in the parent perturbation policy set, and then select the policy distance with the minimum policy distance. ; If the minimum strategy distance Greater than the preset distance threshold And the number of perturbation agents in the perturbation strategy population pool Maximum capacity not reached If so, the new state perturbation strategy is directly added to the perturbation strategy population pool; If the minimum strategy distance Greater than the preset distance threshold And the number of perturbation agents in the perturbation strategy population pool Maximum capacity has been reached. Then, the new state perturbation strategy will replace the minimum strategy distance. The corresponding parent perturbation strategy is then added to the perturbation strategy population pool.

10. A population-based iterative optimization-driven multi-agent robust control device, characterized in that, The device includes: The policy initialization module is used to initialize multiple existing perturbation policies in the perturbation policy population pool, and at the same time initialize the target policy to be trained. The parent strategy sampling module is used to sample the set of parent perturbation strategies for the current round from the perturbation strategy population pool; The sequential perturbation generation and data acquisition module is used to execute sequential state perturbation based on the parent perturbation strategy set to obtain the perturbation state of the current round, and at the same time collect the strategy optimization sample data of the current round of iterative interaction. The target policy robust training and update module is used to perform robust training and update of the target policy based on the current perturbation state. The new perturbation strategy generation module is used to optimize sample data based on the strategy, with the goal of minimizing the cumulative benefit of the target multi-agent, and to generate a new state perturbation strategy by combining population diversity regularization constraints. The strategy distance calculation and population update module is used to calculate the strategy distance between the new state perturbation strategy and each parent perturbation strategy in the parent perturbation strategy set. When the strategy distance is greater than a preset distance threshold, the new state perturbation strategy is added to the perturbation strategy population pool; otherwise, the new state perturbation strategy is eliminated to obtain the updated perturbation strategy population pool. The population iteration loop control module is used to repeatedly execute the process from the parent policy sampling module to the policy distance calculation and population update module based on the updated perturbation policy population pool, to complete multiple rounds of population iteration until the preset training termination condition is met, and obtain the trained robust multi-agent policy.

Images