A policy gradient-based distributed multi-agent system optimal control method

Abstract

The application relates to a kind of distributed multi-agent system optimal control methods based on policy gradient, it is characterized in that, comprising the following steps: based on the performance index of discrete time nonlinear multi-agent system, corresponding state-action value function is constructed;Determine an initial control policy in the allowable control set;According to multiple state-action value functions and current control policy, policy evaluation is carried out;According to the policy evaluation result, combined with the method of gradient descent, policy improvement is carried out, the policy of next round iteration is obtained, returns to carry out policy evaluation, until the optimal control policy of convergence is learned, wherein, iterative optimization process is realized based on Actor-Critic structure and experience replay mechanism, critic network approximation control policy is used, and actor network approximation state-action value function is used;Distributed multi-agent system is controlled based on optimal control policy.Compared with the prior art, the application has the advantages of not depending on model, real-time, data utilization rate and the like.

Description

Technical Field

[0001] This invention relates to the intersection and integration of artificial intelligence and optimal control, and in particular to an optimal control method for distributed multi-agent systems based on policy gradients. Background Technology

[0002] For agent control, considering the constraint of controller input, M. Abu Khalaf and F.L. Lewis first introduced a non-quadratic performance index function into the value function to solve the optimal control problem of nonlinear systems under input saturation, but this iterative algorithm is based on system model information. Zhang H also discussed the case with input delay. Qin J solved the optimal synchronization problem of linear systems under input saturation by establishing the HJB equation containing a non-quadratic input energy term in a coupled form. Li C considered the cooperative control problem of multiple agents under input saturation for Euler-Lagrange systems. Yang T studied the optimal consistency problem with input constraints in the discrete case, where the network topology is fixed and undirected. Su H designed a coordinated tracking algorithm to solve the tracking control problem of linear multi-agent systems constrained by actuator amplitude saturation and with additional input uncertainties.

[0003] Inspired by biological learning mechanisms, Adaptive Dynamic Programming (ADP) methods can simulate biological feedback learning. By collecting offline or online input-output data from the system, ADP utilizes reinforcement learning to design approximate optimal controllers for complex systems when the model is unknown. Policy gradient adaptive dynamic programming (PGADP) is an important policy iteration-based method. However, existing methods only consider solutions for a single agent. Multi-agent systems often possess better distributed data and information processing capabilities than single-agent systems. In engineering practice and scientific research, many problems require more performance metrics to describe the system's objectives. Results using policy gradient-based methods to solve multi-objective optimal control problems are scarce. Summary of the Invention

[0004] The purpose of this invention is to provide an optimal control method for distributed multi-agent systems based on policy gradients. A non-quadratic term is introduced into the value function of each agent to represent the handling of input saturation, thereby transforming the input-constrained situation into a general situation without input constraints. Based on a data-driven policy iteration algorithm, the method achieves optimization of local performance indicators and state synchronization of agents. It does not require a known system model and can directly learn the controller policy from real-time interactive data, resulting in strong real-time performance.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] An optimal control method for a distributed multi-agent system based on policy gradient includes the following steps:

[0007] S1. Based on the performance index of discrete-time nonlinear multi-agent system, construct the corresponding state-action value function, where the state-action value function of each agent includes the non-quadratic term of itself and the non-quadratic term of its neighbors.

[0008] S2. Determine an initial control policy within the permissible control set;

[0009] S3. Evaluate the strategy based on multiple state-action value functions and the current control strategy;

[0010] S4. Based on the policy evaluation results of S3, the policy is improved by combining the gradient descent method to obtain the policy for the next iteration. The process is repeated until the convergent optimal control policy is learned. The iterative optimization process of S2-S4 is based on the Actor-Critic structure and the experience replay mechanism. The critic network is used to approximate the control policy, and the actor network is used to approximate the state-action value function.

[0011] S5. Controlling distributed multi-agent systems based on optimal control strategies.

[0012] The discrete-time nonlinear system has N+1 nodes, including N followers and 1 leader. The dynamic equation of the i-th follower is represented as x. i (k+1)=f(x i (k))+g i (x i (k))u i (k), i = 1, 2, ..., N, where... It is the state variable of the i-th follower. It is the control policy input of the i-th follower, satisfying f and g are nonlinear functions with appropriate dimensions. are constants, representing the lower and upper bounds of the saturation constraints on the system input, respectively.

[0013] For follower i, the local neighborhood consistency error is defined as:

[0014]

[0015] in, N i Represents all the neighbors of agent i, a ij Let b be the non-negative adjacency matrix element between agent i and agent j. ix0 is a non-negative constant fixed gain, and x0 represents the system consistency target state.

[0016] The performance metrics are determined based on the control input and the local neighborhood consistency error:

[0017]

[0018] Among them, u- i (k) represents the set of control inputs of all neighbors of follower i {u j (k),j∈N i}, r i (e i (t),u i (t),u -i Let (t) be the utility function, expressed by the following equation:

[0019]

[0020] Among them, Q i Let Q be the weight matrix. i >0. In the absence of input constraints, Y i (u i Y(t) can be chosen as a quadratic function; however, due to the input saturation constraint in the multi-agent system, Y... i (u i (t) is the non-quadratic function introduced by Y itself. j (u j (t) is the non-quadratic function of the introduced neighbor. Due to the local error e i (k) is influenced by both its own and its neighboring nodes' information, therefore its utility function is also related to u. -i (k) is relevant.

[0021] The non-quadratic function is expressed as:

[0022]

[0023]

[0024] Among them, R ii =diag{r i 1 ,r i 2 ,...,r i n}, It is a positive definite diagonal matrix. τ i and τ j It is a constant; Φ(·) is a continuously differentiable monotone bounded function, and Φ(0)=0.

[0025] The state value function of agent i is determined based on performance metrics:

[0026]

[0027] Among them, V i Let i be the state value function of agent i.

[0028] The purpose of this invention is to design a control strategy that satisfies the condition of control input saturation. i (k) drives all agents to achieve state synchronization while minimizing the state-action value function.

[0029] In S2, the permissible control set consists of continuous control strategies that make the performance index bounded.

[0030] The state-action value function of agent i is expressed as:

[0031]

[0032] Among them, Q i Let Q be the state-action value function of agent i. i (0,0)=0,

[0033] According to the Bellman optimality principle, the state value function satisfies the Bellman equation:

[0034] V i (e i (k))=r i (e i (t),u i (t),u -i (t))+V i (e i (k+1))

[0035] Therefore, the Hamiltonian function of agent i is defined as:

[0036]

[0037] Iterating through the state-action value function using the Hamiltonian function yields the optimal state-action value function as follows:

[0038]

[0039] Because of e i (k) is driven by information from itself and its neighbors, so the optimal consistency problem of this invention can be regarded as finding a Nash equilibrium.

[0040] In S2, let the iteration index m = 0. This represents the current control strategy.

[0041] The strategy improvement is based on the strategy evaluation results and control strategy of the current iteration to obtain the control strategy for the next iteration, specifically expressed as follows:

[0042]

[0043] Where α represents the learning rate, This represents the gradient of the state-action value function.

[0044] The optimal solution is found by approximating the control policy and the state-action value function using a finite-dimensional set of basis functions, as shown below:

[0045]

[0046]

[0047] in, Let W represent an L-dimensional activation function vector. i (·)={W1(·),W2(·),...,W L (·)} T Represents an L-dimensional weight vector. This represents the approximation state-action value function. This indicates an approximation control strategy.

[0048] The iterative optimization process based on the Actor-Critic structure and experience replay mechanism is as follows:

[0049] Let the collected offline dataset be represented as:

[0050] Among them, M i This represents the size of the dataset. This represents the data collected for each frame, where This represents the current local neighborhood consistency error. This represents strategy The local neighborhood consistency error at the next moment under the control of ;

[0051] remember This represents the weight of agent i in the Critic network. The weights of the Actor network representing the control policy of agent i are calculated, along with the initial weights for each network.

[0052] Select the initial Obtain the initial control strategy Let the iteration index m = 0;

[0053] Policy evaluation: Combining offline datasets and real-time changing online data frames Calculate the weights of the critic network At the same time, control strategy Applied to practical systems to acquire online data

[0054] Strategy improvement: Based on the calculations made during the strategy evaluation process. and offline datasets and online data Update the Actor network weights for the control policy

[0055] Let m = m + 1, and return to the policy evaluation step until the weights converge.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] (1) This invention uses the action-state value function Q instead of the commonly used state value function V to evaluate the strategy, separating the actions taken at each moment so as to learn better from real-time interactive data.

[0058] (2) This invention introduces non-quadratic functions to deal with the problem of input saturation. Each agent's value function contains its own non-quadratic terms as well as the non-quadratic terms of its neighbors.

[0059] (3) Based on the experience playback mechanism, this invention makes full use of online and offline data in the process of strategy learning, breaks the correlation between data, and improves the efficiency of data use.

[0060] (4) This invention designs a model-free optimal consensus algorithm based on policy gradient under input constraints. It does not rely on a model and can directly learn the controller policy from real-time interactive data, which has strong real-time performance. Attached Figure Description

[0061] Figure 1 This is a flowchart of the method of the present invention;

[0062] Figure 2 This is a schematic diagram of the communication topology of a multi-agent system in one embodiment;

[0063] Figure 3 This is a schematic diagram of the state trajectory of an agent in one embodiment.

[0064] Figure 4 This is a schematic diagram of local neighborhood consistency error in one embodiment.

[0065] Figure 5This is a schematic diagram of the Critic weight change of agent 1 in one embodiment.

[0066] Figure 6 This is a schematic diagram of the Critic weight change of agent 2 in one embodiment.

[0067] Figure 7 This is a schematic diagram illustrating the change in the Critic weight of agent 3 in one embodiment. Detailed Implementation

[0068] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0069] This embodiment provides a multi-objective optimal control method based on policy gradient, such as... Figure 1 As shown, it includes the following steps:

[0070] S1. For discrete-time nonlinear multi-agent systems with input constraints, construct the state-action valence function.

[0071] A general discrete-time nonlinear system can be represented as:

[0072] x i (k+1)=f(x i (k))+g i (x i (k))u i (k), i = 1, 2, ..., N

[0073] In this context, node 0 represents the leader, and there are three followers: nodes 1, 2, and 3. Figure 2 As shown.

[0074] The dynamic equation of follower i can be expressed as:

[0075]

[0076] in It is the state variable of the i-th follower. It is the control policy input of the i-th follower.

[0077] Define the state-action value function of agent i as follows:

[0078]

[0079] Among them, Q i (0,0)=0.

[0080] Because multi-agent systems have input saturation constraints, Y i (u i (t) can be expressed as a non-quadratic functional, as follows:

[0081]

[0082]

[0083] in It is a positive definite diagonal matrix. τ i and τ j It is a constant. Φ(·) is a continuously differentiable monotone bounded function, and Φ(0)=0.

[0084] S2. Select an initial control policy within the permissible control set.

[0085] An admissible control set consists of continuous control strategies that make performance indices bounded; such control strategies are also called admissible control strategies.

[0086] S3. Evaluate the strategy based on multiple state-action value functions and the current control strategy.

[0087] The calculation process for strategy evaluation is as follows:

[0088]

[0089] Here, the superscript m represents the number of iterations; when selecting the initial strategy in S3, m is set to 0. i (m) This represents the current control strategy.

[0090] S4. Based on the policy evaluation results of the previous step, the policy is improved by combining the gradient descent method to obtain the policy for the next iteration. Then, we continue to return to S3 until a convergent controller policy is finally learned.

[0091] The process of improving specific strategies is represented as follows:

[0092]

[0093] Where, α i The learning rate is represented by α1 = 0.08, α2 = 0.01, and α3 = 0.04, respectively, in this embodiment.

[0094] The policy boosting process utilizes the policy gradient method, which only requires updating the policy in one step, eliminating the need to search for the policy of the current value function in each iteration. This avoids unnecessary computation.

[0095] In this embodiment, the iterations of S2-S4 are based on the Actor-Critic structure and implemented using a neural network.

[0096] In the implementation of the specific algorithm, a critic network is used to approximate the control strategy, and an actor network is used to approximate the state-action value function to complete the algorithm implementation of the neural network.

[0097] The control policy and the state-action value function can be approximated using a finite set of basis functions, as follows:

[0098]

[0099]

[0100] in W represents an L-dimensional activation function vector. i (·)={W1(·),W2(·),...,W L (·)} T This indicates that there are a total of L-dimensional weight vectors.

[0101] Specifically, the iterative optimization process includes the following steps:

[0102] Step 1) Let the collected offline dataset be represented as:

[0103]

[0104] Among them, M i This represents the size of the dataset. This represents the data collected for each frame, where This represents the current local neighborhood consistency error. This represents strategy The local neighborhood consistency error at the next moment under the control of [the system].

[0105] 2) Record This represents the weight of agent i in the Critic network. The weights of the Actor network representing the control policy of agent i are used to calculate the initial weights of each critic network.

[0106] The weight calculation method is as follows:

[0107]

[0108]

[0109]

[0110] 3) Select an initial control strategy within the permissible control set. And set the iteration count m = 0.

[0111] 4) Policy evaluation: Combining offline datasets and real-time changing online data frames Calculate the weights of the critic network At the same time, control strategy Used in practical systems to acquire online data.

[0112] The weight update method is as follows:

[0113]

[0114]

[0115]

[0116]

[0117] 5) Strategy Improvement: Based on the calculations made during the strategy evaluation process. and offline datasets and online data To update the Actor network weights for the control policy

[0118]

[0119]

[0120]

[0121]

[0122] 6) Let m = m + 1, return to step 4), until the weights converge.

[0123] In this embodiment, as Figure 3 As shown, the agent's state information changes over time and reaches consistency after 20 iterations. The local neighborhood consistency error trajectory for each agent is shown below. Figure 4 As can be seen from the figure, the local neighborhood consistency error gradually approaches 0, and basically reaches 0 after 20 iterations.

[0124] Weighted convergence graph as follows Figure 5-7 As shown, they eventually converge to:

[0125] ρ1=[0.2662,1.2994,-0.3593,0.0296,0.0143,-0.1667]T

[0126] ρ2=[-0.0090,0.2388,1.5908,-0.0757,0.5233,-0.2177] T

[0127] ρ3=[1.3279,-0.9258,-0.1036,1.5743,-0.4400,-0.0469,0.0074,0.4595,-0.0037,0.0102] T

[0128] S5. Controlling distributed multi-agent systems based on optimal control strategies.

[0129] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An optimal control method for a distributed multi-agent system based on policy gradient, characterized in that, Includes the following steps: S1. Based on the performance index of discrete-time nonlinear multi-agent system, construct the corresponding state-action value function, where the state-action value function of each agent includes the non-quadratic term of itself and the non-quadratic term of its neighbors. S2. Determine an initial control policy within the permissible control set; S3. Evaluate the strategy based on multiple state-action value functions and the current control strategy; S4. Based on the policy evaluation results of S3, the policy is improved by combining the gradient descent method to obtain the policy for the next iteration. The process is repeated until the convergent optimal control policy is learned. The iterative optimization process of S2-S4 is based on the Actor-Critic structure and the experience replay mechanism. The critic network is used to approximate the control policy, and the actor network is used to approximate the state-action value function. S5. Controlling distributed multi-agent systems based on optimal control strategies; The performance metrics are determined based on the control input and the local neighborhood consistency error: in, Represents followers The set of control inputs of all neighbors , The utility function is expressed by the following formula: in, This is the weight matrix. , To introduce its own non-quadratic function, For the non-quadratic function of the introduced neighbor; The intelligent agent The state-action value function is expressed as: in, For intelligent agents i State-action value function , According to the Bellman optimality principle, the state value function satisfies the Bellman equation: Thus defining an intelligent agent The Hamiltonian function is: Iterating through the state-action value function using the Hamiltonian function yields the optimal state-action value function as follows: because Driven by information from itself and its neighbors, the optimal consistency problem can be viewed as finding a Nash equilibrium. The iterative optimization process based on the Actor-Critic structure and experience replay mechanism is as follows: Let the collected offline dataset be represented as: in, This represents the size of the dataset. This represents the data collected for each frame, where This represents the current local neighborhood consistency error. This represents strategy The local neighborhood consistency error at the next moment under the control of ; remember Representing intelligent agents The corresponding weights of the Critic network, Representing intelligent agents The weights of the Actor network for controlling the policy are calculated, along with the initial weights for each network. Select the initial Obtain the initial control strategy Let the iterative index ; Policy evaluation: Combining offline datasets and real-time changing online data frames Calculate the weights of the critic network. At the same time, control strategies Applied to practical systems to acquire online data ; Strategy improvement: Based on the calculations made during the strategy evaluation process. and offline datasets and online data Update the Actor network weights for the control policy ; make Return to the policy evaluation step until the weights converge.

2. The optimal control method for a distributed multi-agent system based on policy gradient according to claim 1, characterized in that, The discrete-time nonlinear system has a total of Nodes, including One follower and one leader, the first The dynamic equation for a follower is expressed as: in, It is the first The state variables of each follower It is the first The control strategy input of each follower satisfies , f , g For a nonlinear function with appropriate dimension, , are constants, representing the lower and upper bounds of the saturation constraints on the system input, respectively.

3. The optimal control method for a distributed multi-agent system based on policy gradient as described in claim 2, characterized in that, For followers In this regard, the local neighborhood consistency error is defined as: in, , It represents all intelligent agents. Neighbors a ij For intelligent agents i and intelligent agents j Non-negative adjacency matrix elements between them b i A fixed gain that is a non-negative constant. x 0 indicates the system consistency target state.

4. The optimal control method for a distributed multi-agent system based on policy gradient according to claim 1, characterized in that, The non-quadratic function is expressed as: in, It is a positive definite diagonal matrix. , and It is a constant; It is a continuously differentiable, monotone, and bounded function, and .

5. The optimal control method for a distributed multi-agent system based on policy gradient according to claim 4, characterized in that, The intelligent agent i The state value function is determined based on performance metrics: in, For intelligent agents i The state value function.

6. The optimal control method for a distributed multi-agent system based on policy gradient according to claim 1, characterized in that, The strategy improvement is based on the strategy evaluation results and control strategy of the current iteration to obtain the control strategy for the next iteration, specifically expressed as follows: in, Represents the learning rate. This represents the gradient of the state-action value function.

7. The optimal control method for a distributed multi-agent system based on policy gradient as described in claim 6, characterized in that, The optimal solution is found by approximating the control policy and the state-action value function using a finite-dimensional set of basis functions, as shown below: in, express dimensional activation function vector, express dimensional weight vector, This represents the approximation state-action value function. This indicates an approximation control strategy.

Images