An optimal consistency control method for intelligent unmanned swarm systems based on adaptive dynamic programming

Abstract

The present invention relates to the field of multi-agent system control, and in particular to an optimal consistency control method for an intelligent unmanned cluster system based on adaptive dynamic programming, comprising: determining the topological structure of the multi-agent system and the dynamic equations of the agents, wherein the system matrix of the agents is unknown; the system topology diagram contains a directed spanning tree, and the root node of the directed spanning tree is the leader; the local tracking error of the agent and the update expression of the local tracking error are defined according to the system state, thereby converting the consistency problem into a tracking control problem; defining a performance index function for evaluating the current control strategy, and optimizing the control strategy according to the current performance index function; collecting information of the system during the evolution process, and designing an adaptive dynamic programming algorithm for approximating the control strategy and performance index of the agent, and updating the agent state according to the control strategy, thereby achieving optimal consistency.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent collaborative control, and specifically relates to an optimal consistency control method for an intelligent unmanned cluster system based on adaptive dynamic programming. Background Art

[0002] In nature, there is a common phenomenon of biological clustering, which refers to the fact that some organisms usually carry out group activities according to certain rules. For example, bees build nests; fish forage for food. Inspired by this biological cluster behavior, relevant experts and scholars have applied the consistency of multi-agent systems (MASs) to the collaborative control of complex systems, such as smart grids, sensor networks, and the collaborative work of drone swarms. Consistency refers to the fact that each agent in the multi-agent system dynamically adjusts its own behavior according to the control protocol set in advance by exchanging information with its own local agents to reach a common state.

[0003] In real life, the resources and costs required by the system to achieve a specific control purpose are limited. Considering the economy and practicality, the system must be able to complete the control task in the most economical way, otherwise the theoretical results will be difficult to apply to the real industrial field. In addition, in practical applications, the system information of the agent is difficult to obtain. Therefore, studying the optimal consistency control of multi-agent systems under unknown models has great application value.

[0004] In the process of solving the optimal consistency control, the traditional approach is based on two types of algorithm frameworks of reinforcement learning, namely, value iteration algorithm and policy iteration algorithm. However, both types of algorithms have their own shortcomings. The policy iteration algorithm requires an initial admissible control policy, but this policy is usually difficult to obtain. Although the value iteration algorithm does not require an initial admissible control policy, its convergence speed will be very slow, which is not desirable in practical applications.

[0005] At present, most of the work using data-driven methods to study the optimal consistency control of multi-agent systems tends to use the data measured by the system only once, that is, to update the current control strategy based on the data generated by the current iteration. However, in practical applications, the data measured by the real system is very valuable and can be used as training data for other similar systems. Therefore, compared with ordinary data-driven methods, by introducing the experience replay mechanism to collect the data generated by the agent during the iteration process, the data can be repeatedly used to improve data efficiency and be more effective when data acquisition is difficult. And by randomly selecting training data, the correlation between data can be broken, thereby obtaining better training results.

[0006] Most of the research works mentioned above use a single neural network structure (evaluation network) to train the agent. The performance indicator function estimated by a separate Critic network is prone to instability when chasing the changing cumulative utility function. By introducing a target network, which is relatively fixed over a period of time, the update of the evaluation network can be converted into a problem similar to supervised learning, thereby increasing the stability of the learning process and better meeting the needs of actual application scenarios. Therefore, how to design an optimal consistency control for an intelligent unmanned cluster system based on adaptive dynamic programming is an urgent problem to be solved. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes an optimal consistency control method for an intelligent unmanned cluster system based on adaptive dynamic programming, comprising:

[0008] S1: The agents in the multi-agent system are divided into two categories: one is the leader agent, which will not receive information from neighboring agents, and the other is the follower agent, which will adjust its behavior according to the information received from neighboring agents;

[0009] The information sent by each intelligent agent includes its own state information and control strategy;

[0010] S2: Obtain the state information between the neighbor follower agent and the leader agent through the agent's own sensors, and calculate the local tracking error of the agent based on the state information;

[0011] S3: Set up an experience pool for each agent to store its current error, next moment error and control strategy;

[0012] S4: Construct an action state value function for evaluating the current control strategy of the agent based on the information sent by the agent and the local tracking error between agents;

[0013] S5: Design an adaptive dynamic programming algorithm to calculate the action state value function used to evaluate the quality of the current control strategy of the intelligent agent, and iteratively update the control strategy of each intelligent agent by gradient updating;

[0014] S6: Use neural networks to approximate the control strategy and action state value function respectively, set the number of algorithm iterations and repeat the process of iteratively updating the action state value function and control strategy in S5. When the action state value function and the control strategy converge to a sufficiently small value, the algorithm is executed and all agents obtain the optimal control strategy.

[0015] Beneficial effects of the present invention:

[0016] 1. Using data-based adaptive dynamic programming to enable discrete-time multi-agent systems to achieve optimal consensus;

[0017] 2. The experience replay mechanism is used to store the information generated by the agent during the interaction process, including current and past state information and the current control strategy. The experience replay mechanism can not only reuse the data generated by the system multiple times to improve data efficiency, but also break the correlation between data and improve training effects;

[0018] 3. The adaptive dynamic programming algorithm used by the multi-agent system combines the advantages of the value iteration algorithm and the policy iteration algorithm. It realizes the combination of the two types of algorithms through a balance parameter. This algorithm converges faster than the value iteration algorithm and does not require an initial admissible control strategy, effectively overcoming the defects of the value iteration algorithm and the policy iteration algorithm. Based on the adaptive dynamic programming algorithm, the control strategy can be updated in a "soft update" manner, which consumes less computational cost and ensures a more stable training process.

[0019] 4. The action-state value function is adopted. The intelligent agent has the characteristics of different policy learning in the learning process. Its behavior strategy can interact with the external environment, collect data and put it into the experience pool. The target strategy then selects data from the experience pool for neural network training and learning, which greatly improves the stability and efficiency of the training process.

[0020] 5. Adaptive parameter adjustment rules are used to adjust the adaptive parameter ρ l , ρ l Starting from 0 and increasing to 1, the proposed adaptive dynamic programming algorithm can first implement the value iteration algorithm, find the admissible control strategy, and then transition to the policy iteration algorithm. The adaptive dynamic programming algorithm does not require the difficult-to-obtain initial admissible control strategy, and it accelerates the convergence speed compared to the value iteration algorithm, which has a wider range of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a system control flow chart of a preferred embodiment provided by the present invention;

[0022] Figure 2 is a system topology diagram of an embodiment of the present invention;

[0023] Figure 3 is an evolution diagram of the three-dimensional state information of an intelligent agent according to an embodiment of the present invention;

[0024] Figure 4 is an evolution diagram of the agent error vector of an embodiment of the present invention;

[0025] Figure 5 It is a two-dimensional state information evolution diagram of an intelligent agent according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] An optimal consistency control method for intelligent unmanned cluster systems based on adaptive dynamic programming, such as Figure 1 As shown, including:

[0028] S1: The agents in the multi-agent system are divided into two categories: one is the leader agent, which will not receive information from neighboring agents, and the other is the follower agent, which will adjust its behavior according to the information received from neighboring agents;

[0029] The information sent by each intelligent agent includes its own state information and control strategy;

[0030] S2: Obtain the state information between the neighbor follower agent and the leader agent through the agent's own sensors, and calculate the local tracking error of the agent based on the state information;

[0031] S3: Set up an experience pool for each agent to store its current error, next moment error and control strategy;

[0032] S4: Construct an action state value function for evaluating the current control strategy of the agent based on the information sent by the agent and the local tracking error between agents;

[0033] S5: Design an adaptive dynamic programming algorithm to calculate the action state value function used to evaluate the quality of the current control strategy of the intelligent agent, and iteratively update the control strategy of each intelligent agent by gradient updating;

[0034] S6: Use neural networks to approximate the control strategy and action state value function respectively, set the number of algorithm iterations and repeat the process of iteratively updating the action state value function and control strategy in S5. When the action state value function and the control strategy converge to a sufficiently small value, the algorithm is executed and all agents obtain the optimal control strategy.

[0035] The leader-follower dynamics equations include:

[0036] x i (k+1)=Ax i (k)+B i (k),i=1,2,...,N

[0037] x 0 (k+1)=Ax 0 (k)

[0038] Among them, A∈R p×p and B i ∈R p×q is the system matrix of the agent and is completely unknown, where R p×p represents the value range of the system matrix A, R refers to the real number domain, and p, q refer to the dimensions of the matrix. i (k)∈R p and u i (k)∈R q are the state and control strategy of the ith agent at the kth moment; x 0 (k)∈R p is the leader’s state information, and also serves as the consistent target state of the follower agent. The information sent by each agent includes its own state information and control strategy. i (k)∈R p and u i (k)∈R q They are the state information and control strategy of the i-th agent at the k-th moment, where k represents the update time of the agent.

[0039] It is assumed that the agents will not be affected by the environment during the communication process, which will cause the communication topology to change. According to the information interaction between agents in the multi-agent system, the topology of the system is determined, and it is ensured that the topology contains a directed spanning tree.

[0040] Calculate the local tracking error of the agent based on the state information, including:

[0041]

[0042] Among them, e i (k) represents the local tracking error of agent i at the kth moment, b i Represents the containment gain, if b i = 1 means the agent is directly connected to the leader, otherwise b i =0; a ij represents the connection weight between the ith agent and the jth agent, x j (k) represents the state vector of the jth agent at the kth moment, represents the set of neighboring agents of the ith agent, x i (k) represents the state information of the i-th agent at the k-th moment, x 0 (k) represents the status information of the leader at the kth moment.

[0043] The experience pool in S3 has a capacity limit. When the capacity of the experience pool reaches a certain level, one or more sets of data with the longest storage time will be deleted, and the newly generated data will be stored.

[0044] The action state value function used to evaluate the quality of the current control strategy of the intelligent agent includes:

[0045] Q i (e i (k),μ)=c i (e i (k),μ)+τQ i (e i (k+1),u i (e i (k+1)))

[0046] Among them, Q i (e i (k),μ) represents the action state value function used to evaluate the quality of the current control strategy of the intelligent agent, e i (k) represents the local tracking error of agent i at the kth moment, μ represents the information sent by the agent, represents the utility function used to evaluate the control strategy of the agent at the kth moment, u i (k) represents the control strategy of agent i at the kth moment, T represents the transposition operation, Q ii ≥0,R ii ≥0 indicates a positive weight matrix, τ∈[0,1] indicates the discount factor in reinforcement learning, e i (k+1) represents the local tracking error of agent i at the next moment.

[0047] The adaptive dynamic programming algorithm: uses the adaptive dynamic parameter to adjust the dynamic factor ρ of the adaptive dynamic programming algorithm l , dynamic factor ρ l From small to large changes, initially, the dynamic factor ρ l Starting from close to 0, the adaptive dynamic programming algorithm finds an admissible control strategy through value iteration, and then as the dynamic factor ρ l Gradually increases to approach 1, and finds the optimal control strategy based on the admissible control strategy; the process is expressed as:

[0048] ρ l =tanh(bIn(a+l))

[0049] Where b represents the first constant, a represents the second constant, l represents the iteration index, a>1 and b>0, and the dynamic factor ρ l is a monotonically increasing function from 0 to 1.

[0050] Design an adaptive dynamic programming algorithm to calculate the action state value function used to evaluate the quality of the agent's current control strategy, and iteratively update the control strategy of each agent by gradient updating, including:

[0051] Initialize the iteration index l = 0 and initialize an arbitrary control strategy for all agents

[0052] Calculate the action state value function:

[0053]

[0054] Among them, Q i l () represents the action state value function integrated with the iteration index, e i (k) represents the local tracking error of agent i at the kth moment, μ represents the information sent by the agent,

[0055] represents the utility function used to evaluate the control strategy of the agent at the kth moment, τ represents the discount factor in reinforcement learning, and ρ l represents the dynamic factor, e i (k+1) represents the local tracking error of agent i at the next moment, and l represents the iteration index;

[0056] Update the control policy using the action-state value function:

[0057]

[0058] Among them, e i (k) represents the local tracking error of agent i at the kth moment, l represents the iteration index, μ represents the information sent by the agent, τ represents the discount factor in reinforcement learning, and ρ l represents the dynamic factor, e i (k+1) represents the local tracking error of agent i at the next moment, r i () represents the utility function of the ith agent, Represents the action-state value function that incorporates the iteration index;

[0059] As the iteration proceeds, if the action-state value function and control strategies If all can converge to the optimal value, the learning process of the agent is completed.

[0060] A neural network is used to approximate the control strategy and the action state value function respectively, and the number of algorithm iterations is set and the process of iteratively updating the action state value function and the control strategy in S5 is repeatedly executed, including:

[0061] Neural networks are used to approximate performance indicator functions and control strategies respectively, and the Actor-Critic neural network framework is used to solve the optimal control strategy; the Actor neural network is used to approximate the optimal control strategy, and the Critic neural network is used to approximate the performance indicator function;

[0062] The expression of Actor neural network is:

[0063]

[0064] in, represents the control strategy estimated by the Actor neural network, represents the first connection weight of the Actor neural network, T represents the transposition operation, σ ai () represents the activation function of the neural network actor, represents the second connection weight of the Actor neural network, θ ai (k) represents the adjustment matrix of the neural network actor;

[0065] The expression of the Critic neural network is:

[0066]

[0067] in, represents the action state value function estimated by the Critic neural network, e i (k) represents the local tracking error of agent i at the kth moment, μ represents the information sent by the agent, represents the first connection weight of the Critic neural network, T represents the transposition operation, σ ci () represents the activation function of the neural network Critic, represents the second connection weight of the Critic neural network, θ ci (k) represents the adjustment matrix of the neural network Critic.

[0068] When the performance indicator function and the control strategy converge to a sufficiently small value, the algorithm is executed and the multi-agent system reaches optimal consistency:

[0069] lim k→∞ ‖e(k)‖=0

[0070] in, k represents time, e(k) represents the global tracking error, ||e(k)|| represents the norm of the global tracking error, e N (k) indicates the N local tracking error, T represents the transposition operation, R Nprepresents a real field matrix.

[0071] This example considers a multi-agent system consisting of N follower agents and 1 leader agent. The relationship topology of the multi-agent system can be represented by a directed weighted graph G = (V, E, A), where each agent is a node of the undirected weighted graph G = (V, E, A), where V = {v 1 ,v 2 ,...,v N+1} represents the set of nodes, E = {a ij =(v i ,v j ):v i ,v j ∈V,i≠j}, A-[a ij ] represents the adjacency matrix, where the matrix element a ij represents the connection weight between agent nodes i and j. If there is a connection between nodes i and j, then a ij > 0; if there is no connection between nodes i and j, then a ij =0, specify a ij = 0 means that the system has no self-loop. The nodes connected to node i are the neighbor nodes of node i. The neighbor nodes of node i are represented by the set N i = {v j ∈V|(v j ,v i )∈E}. For follower agent i, its in-degree is expressed as Then the in-degree matrix of the system node is D = diag{d 1 ,...,d N}. The topological Laplace matrix of the system is L = DA = [l ij ]∈R N×N , where l ij =-a ij ,i≠j,l ii =∑ i≠j a ij .

[0072] This example uses Figure 2 The experimental topology is a multi-agent system consisting of 4 nodes, where node 0 is the leader agent and the other nodes are follower agents. There are direct edges between the leader agent and nodes 1 and 2. From the topology, it can be found that node 0 can indirectly reach all other follower agents, so the topology contains a directed spanning tree. The model of the system is as follows:

[0073] x 0 (k+1)=Ax 0 (k)

[0074] x i (k+1)=Ax i (k)+B i u i (k),i=1,2,3,4

[0075] in

[0076]

[0077]

[0078]

[0079] In a topological graph, the non-zero edge weight is a 32 =1, the leader's adjacency matrix is ​​B=diag{1,1,0}. Randomly initialize the leader's state information between [-1,1], and set the learning rates of the Actor neural network and the Critic neural network to α ai =α ci =0.03, and the initial weights of the neural network are all taken from random numbers in the range [-1,1]. The activation functions of the Critic neural network are σ c1 =tanh([e 11 (k) e 21 (k) u 1 (k)] T ),σ c2 =tanh([e 21 (k) e 22 (k) u 2 (k)] T ),σ c3 =tanh([e 31 (k) e 32 (k) u 3 (k) u 2 (k)]). The activation function of the Actor neural network is σ ai =[e i1 (k) e i2 (k)], and all neural networks were trained at least 300 times.

[0080] like Figure 3 As shown, the change process of the three-dimensional state information of all intelligent agents is demonstrated. It is not difficult to see that the states of all intelligent agents eventually reach a consensus with the leader, proving the feasibility of the algorithm. Figure 4 The evolution process of the tracking error of all agents is given, and it can be seen that they can all converge to 0 in the end, that is, all follower agents reach a consensus on the state information of the leader agent. Figure 5The evolution process of the two-dimensional state information of the intelligent agent is given. It can be seen that all intelligent agents are able to track the state of the leader, and finally all intelligent agents reach a consensus.

[0081] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An optimal consistency control method for an intelligent unmanned cluster system based on adaptive dynamic programming, characterized in that: include: S1: The agents in the multi-agent system are divided into two categories: one is the leader agent, which will not receive information from neighboring agents, and the other is the follower agent, which will adjust its behavior according to the information received from neighboring agents; The information sent by each intelligent agent includes its own state information and control strategy; S2: Obtain the state information between the neighbor follower agent and the leader agent through the agent's own sensors, and calculate the local tracking error of the agent based on the state information; S3: Set up an experience pool for each agent to store its current error, next moment error and control strategy; S4: Construct an action state value function for evaluating the current control strategy of the agent based on the information sent by the agent and the local tracking error between agents; S5: Design an adaptive dynamic programming algorithm to calculate the action state value function used to evaluate the quality of the current control strategy of the intelligent agent, and iteratively update the control strategy of each intelligent agent by gradient updating; The adaptive dynamic programming algorithm: uses the adaptive dynamic parameter to adjust the dynamic factor ρ of the adaptive dynamic programming algorithm l , dynamic factor ρ l From small to large changes, initially, the dynamic factor ρ l Starting from close to 0, the adaptive dynamic programming algorithm finds an admissible control strategy through value iteration, and then as the dynamic factor ρ l Gradually increases to approach 1, and finds the optimal control strategy based on the admissible control strategy; the process is expressed as: ρ l =tanh(bIn(a+l)) Where b represents the first constant, a represents the second constant, l represents the iteration index, a>1 and b>0, and the dynamic factor ρ l is a monotonically increasing function from 0 to 1; Design an adaptive dynamic programming algorithm to calculate the action state value function used to evaluate the quality of the agent's current control strategy, and iteratively update the control strategy of each agent by gradient updating, including: Initialize the iteration index l = 0 and initialize an arbitrary control strategy for all agents Calculate the action state value function: in, represents the action state value function incorporating the iteration index, e i (k) represents the local tracking error of agent i at the kth moment, μ represents the information sent by the agent, represents the utility function used to evaluate the control strategy of the agent at the kth moment, τ represents the discount factor in reinforcement learning, and e i (k+1) represents the local tracking error of agent i at the next moment; Update the control policy using the action-state value function: Among them, r i ( ) represents the utility function of the ith agent; As the iteration proceeds, if the action-state value function and control strategies If all can converge to the optimal value, the learning process of the agent is completed; S6: Use neural networks to approximate the control strategy and action state value function respectively, set the number of algorithm iterations and repeat the process of iteratively updating the action state value function and control strategy in S5. When the action state value function and the control strategy converge to a sufficiently small value, the algorithm is executed and all agents obtain the optimal control strategy.

2. According to claim 1, the optimal consistency control method of an intelligent unmanned cluster system based on adaptive dynamic programming is characterized in that: Calculate the local tracking error of the agent based on the state information, including: Among them, e i (k) represents the local tracking error of agent i at the kth moment, b i Represents the containment gain, if b i = 1 means the agent is directly connected to the leader, otherwise b i =0; a ij represents the connection weight between the ith agent and the jth agent, x j (k) represents the state vector of the jth agent at the kth moment, represents the set of neighboring agents of the ith agent, x i (k) represents the state information of the i-th agent at the k-th moment, and x0(k) represents the state information of the leader at the k-th moment.

3. According to claim 1, the optimal consistency control method of an intelligent unmanned cluster system based on adaptive dynamic programming is characterized in that: The experience pool in S3 has a capacity limit. When the capacity of the experience pool reaches a certain level, one or more sets of data with the longest storage time will be deleted, and the newly generated data will be stored.

4. According to claim 1, the optimal consistency control method of an intelligent unmanned cluster system based on adaptive dynamic programming is characterized in that: The action state value function used to evaluate the quality of the current control strategy of the intelligent agent includes: Q i (e i (k),μ)=c i (e i (k),μ)+τQ i (e i (k+1),u i (e i (k+1))) Among them, Q i (e i (k),μ) represents the action state value function used to evaluate the quality of the current control strategy of the intelligent agent, e i (k) represents the local tracking error of agent i at the kth moment, μ represents the information sent by the agent, represents the utility function used to evaluate the control strategy of the agent at the kth moment, u i (k) represents the control strategy of agent i at the kth moment, T represents the transposition operation, Q ii ≥0,R ii ≥0 indicates a positive weight matrix, τ∈[0,1] indicates the discount factor in reinforcement learning, e i (k+1) represents the local tracking error of agent i at the next moment.

5. According to claim 1, the optimal consistency control method of an intelligent unmanned cluster system based on adaptive dynamic programming is characterized in that: A neural network is used to approximate the control strategy and the action state value function respectively, and the number of algorithm iterations is set and the process of iteratively updating the action state value function and the control strategy in S5 is repeatedly executed, including: Neural networks are used to approximate performance indicator functions and control strategies respectively, and the Actor-Critic neural network framework is used to solve the optimal control strategy; the Actor neural network is used to approximate the optimal control strategy, and the Critic neural network is used to approximate the performance indicator function; The expression of Actor neural network is: in, represents the control strategy estimated by the Actor neural network, represents the first connection weight of the Actor neural network, T represents the transposition operation, σ ai () represents the activation function of the neural network actor, represents the second connection weight of the Actor neural network, θ ai (k) represents the adjustment matrix of the neural network actor; The expression of the Critic neural network is: in, represents the action state value function estimated by the Critic neural network, e i (k) represents the local tracking error of agent i at the kth moment, μ represents the information sent by the agent, represents the first connection weight of the Critic neural network, T represents the transposition operation, σ ci ( ) represents the activation function of the neural network Critic, represents the second connection weight of the Critic neural network, θ ci (k) represents the adjustment matrix of the neural network Critic.

6. The optimal consistency control method of an intelligent unmanned cluster system based on adaptive dynamic programming according to claim 1 is characterized in that: When the performance indicator function and the control strategy converge to a sufficiently small value, the algorithm is executed and the multi-agent system reaches optimal consistency: glue k→∞ ||e(k)||=0 Where k represents time, e(k) represents the global tracking error, and ||e(k)|| represents the norm of the global tracking error. e N (k) represents the Nth local tracking error, T represents the transposition operation, R Np represents a real field matrix.

Images