A multi-agent cooperative control method based on hybrid policy iteration

Abstract

The application provides a multi-agent cooperative control method based on a mixed policy iteration, which can be applied to the fields of adaptive dynamic programming, reinforcement learning and multi-agent system technology. The method comprises the following steps: based on the Bellman optimality principle, the optimal cooperative control strategy of each agent is obtained by using the optimal cooperative control model and the optimal cost function of each agent in the multi-agent system; the optimal cost function and the optimal cooperative control strategy of each agent are iteratively executed by using the mixed policy iteration algorithm; in the process of iteratively executing the approximation operation and the updating operation, the real-time evaluation operation is performed on the allowable control strategy learning stage and the optimal cooperative control strategy exploration stage by using the evaluation neural network, so that the real-time cooperative control strategy is obtained; and the inter-agent cooperative control operation of the multi-agent system is performed by using the real-time cooperative control strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of adaptive dynamic programming, reinforcement learning and multi-agent system, and in particular to a multi-agent collaborative control method based on mixed policy iteration. BACKGROUND

[0002] A multi-agent system is a system composed of multiple agents with autonomy, which can perceive the environment and take actions, such as a car-road-cloud multi-terminal cooperative vehicle system, an autonomous vehicle, a multi-type sorting robot applied to the field of warehouse logistics, a multi-type assembly robot applied to intelligent manufacturing, etc. These agents achieve system-level goals through interaction, cooperation or competition. Compared with single-agent systems, multi-agent systems have great advantages in computing power, decision-making ability and robustness, and have received extensive attention in academic and industrial fields. The collaborative control problem is a key problem in the field of multi-agent systems, which involves information exchange, task allocation, behavior coordination and consensus achievement among multiple agents, and has high complexity and challenge. Most of the existing multi-agent system collaborative control methods are based on traditional consensus control protocols, and the agents lack autonomous learning ability in the collaborative process, making it difficult to meet the individual optimality requirement.

[0003] In order to improve the intelligent level of multi-agent systems, existing technologies have begun to use adaptive dynamic programming learning control methods to solve multi-agent system collaborative control problems involving optimization goals. However, the existing learning control methods based on adaptive dynamic programming still face the cold start problem and technical problems such as slow convergence speed and high computational complexity. SUMMARY

[0004] In view of the above problems, the present application provides a multi-agent collaborative control method based on mixed policy iteration, which is used to at least solve one of the above problems.

[0005] According to a first aspect of the present application, a multi-agent collaborative control method based on mixed policy iteration is provided, comprising: based on the Bellman optimality principle, using the optimal collaborative control model and the optimal cost function of each agent in the multi-agent system, obtaining the optimal collaborative control strategy of each agent; using a mixed policy iteration algorithm to iteratively perform approximation and update operations on the optimal cost function and the optimal collaborative control strategy of each agent, wherein the mixed policy iteration algorithm includes a permissible control policy learning phase and an optimal collaborative control policy exploration phase; during the iterative execution of the approximation and update operations, using an evaluation neural network to perform real-time evaluation operations on the permissible control policy learning phase and the optimal collaborative control policy exploration phase, obtaining a real-time collaborative control strategy; using the real-time collaborative control strategy to perform collaborative control operations between agents in the multi-agent system.

[0006] According to the embodiment of the present application, the optimal cooperative control strategy of each agent is obtained based on the Bellman optimality principle, the optimal cooperative control model of each agent and the optimal cost function of each agent, and the method comprises the following steps: modeling each agent according to the attribute information of each agent to obtain a dynamic model of each agent; constructing an optimal cooperative control model of each agent by using the attribute information of the multi-agent system based on the leader-follower structure and the dynamic model of each agent; setting a quadratic optimal cost function for controlling the cost of each agent by using the optimal cooperative control model of each agent and the attribute information of the neighboring agents of each agent; and obtaining the optimal cooperative control strategy of each agent by using the Bellman optimality principle based on the quadratic optimal cost function of each agent.

[0007] According to the embodiment of the present application, the modeling of each agent according to the attribute information of each agent to obtain a dynamic model of each agent comprises the following steps: modeling each agent according to the internal dynamic information of each agent, a control input matrix, an optimal cooperative control strategy to be solved and the number of agents in the multi-agent system to obtain a dynamic model of each agent.

[0008] According to the embodiment of the present application, the constructing of an optimal cooperative control model of each agent by using the attribute information of the multi-agent system based on the leader-follower structure and the dynamic model of each agent comprises the following steps: obtaining a local neighborhood cooperative error matrix of each agent, and constructing an optimal cooperative control model of each agent according to the internal dynamic information of the multi-agent system and the dynamic model and the local neighborhood cooperative error matrix of each agent.

[0009] According to the embodiment of the present application, the iterative execution of the approximation operation and the update operation on the optimal cost function and the optimal cooperative control strategy of each agent by using the hybrid policy iteration algorithm comprises the following steps: setting a permissible control policy learning stage represented by a value update table and a stop condition of the permissible control policy learning stage; in the permissible control policy learning stage, based on the stop condition, the iterative execution of the approximation operation and the update operation on the optimal cost function and the optimal cooperative control strategy of each agent is performed to obtain a current value function and a current permissible control policy, wherein the current value function is a current approximation result of the optimal cost function, and the current permissible control policy is a current approximation result of the optimal cooperative control strategy.

[0010] According to the embodiment of the present application, the iteration of the approximation operation and the update operation of the optimal cost function and the optimal cooperative control strategy of each agent by using the hybrid policy iteration algorithm further comprises: constructing an optimal control strategy exploration stage with a policy evaluation operation and a policy improvement operation; in the policy evaluation operation, based on a preset evaluation condition, a current value function is solved by performing the policy evaluation operation on the current allowable control strategy iteration to obtain an updated value function; in the policy improvement operation, the policy improvement operation is performed on the current allowable control strategy iteration based on the updated value function to obtain an updated allowable control strategy.

[0011] According to the embodiment of the present application, the setting of the allowable control strategy learning stage characterized by value update and the stop condition of the allowable control strategy learning stage comprises: replacing the neighborhood cooperative error function of each agent with a preset neighborhood cooperative error approximation function, wherein the value range of the preset neighborhood cooperative error approximation function is greater than the value range of the neighborhood cooperative error function; determining a value update process of the value function by using the preset neighborhood cooperative error approximation function, the deterministic sequence and the arbitrary initial value function; and determining the stop condition according to the preset neighborhood cooperative error approximation function, the value range of the neighborhood cooperative error function, the deterministic sequence and the difference of the value function of the adjacent value update process.

[0012] According to the embodiment of the present application, in the iteration of the approximation operation and the update operation, the real-time evaluation operation of the allowable control strategy learning stage and the optimal cooperative control strategy exploration stage is performed by using the evaluation neural network to obtain a real-time cooperative control strategy, which comprises: in the iteration of the approximation operation and the update operation of the allowable control strategy learning stage, the allowable control strategy learning stage is reconstructed by using the evaluation neural network, and the real-time update operation of the current value function and the real-time evaluation operation of the current allowable control strategy are performed by using the evaluation neural network; in the iteration of the approximation operation and the update operation of the optimal cooperative control strategy exploration stage, the optimal cooperative control strategy exploration stage is reconstructed by using the evaluation neural network, and the real-time update operation of the updated current value function and the real-time evaluation operation of the updated allowable control strategy are performed by using the approximation Hamilton equation and the evaluation neural network to obtain the real-time cooperative control strategy.

[0013] According to the embodiment of the present application, the real-time updating operation on the current value function and the real-time evaluation operation on the current allowable control strategy by using the evaluation neural network include: constructing the evaluation neural network by using a three-layer feedforward neural network; representing the current value function by using the weights and the activation function of the evaluation neural network, and representing the gradient function of the current value function by using the partial derivative of the weights and the activation function of the evaluation neural network; determining a first learning error function of the evaluation neural network by using the deterministic sequence, the weights of the evaluation neural network and the value function of the last iteration round in the real-time updating process of the current value function; reconstructing the current allowable control strategy based on the partial derivative of the weights and the activation function of the evaluation neural network; reconstructing the weights of the evaluation neural network by using the activation function of the evaluation neural network and the first learning error function based on the first normalized gradient descent algorithm and the first minimization target function, to obtain a weight updating rule in the allowable control strategy learning stage; and performing the real-time updating operation on the current value function and the real-time evaluation operation on the current allowable control strategy by using the weight updating rule in the allowable control strategy learning stage, to obtain the updated value function and the updated allowable control strategy.

[0014] According to the embodiment of the present application, the real-time updating operation on the updated current value function and the real-time evaluation operation on the updated allowable control strategy by using the approximate Hamilton equation and the evaluation neural network to obtain the real-time collaborative control strategy include: reconstructing a second learning error function by using the approximate Hamilton equation and the partial derivative of the activation function of the evaluation neural network; reconstructing the updated allowable control strategy by using the weights and the partial derivative of the activation function of the evaluation neural network; reconstructing the weights of the evaluation neural network by using the partial derivative of the activation function of the evaluation neural network and the second learning error function based on the second normalized gradient descent algorithm and the second minimization target function, to obtain a weight updating rule in the optimal collaborative control strategy exploration stage; and performing the real-time updating operation on the updated value function and the real-time evaluation operation on the updated allowable control strategy by using the weight updating rule in the optimal collaborative control strategy exploration stage, to obtain the real-time collaborative control strategy.

[0015] The multi-agent collaborative control method based on the hybrid policy iteration provided by the present application solves the cold start problem, slow convergence speed and low computational efficiency of the multi-agent collaborative control scheme in the prior art by using the hybrid policy iteration algorithm and the evaluation neural network. Meanwhile, the iteration in the allowable control strategy learning stage and the iteration in the optimal collaborative control strategy exploration stage of the present application not only ensure the stability of the collaborative control strategy updating, but also balance the convergence speed and the global optimality. In addition, the real-time evaluation capability of the collaborative control strategy by the evaluation neural network significantly improves the adaptability to the dynamic environment. BRIEF DESCRIPTION OF DRAWINGS

[0016] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A flow chart of a multi-agent collaborative control method based on hybrid policy iteration according to an embodiment of the present application is shown.

[0018] Figure 2 A multi-agent communication topology diagram according to an embodiment of the present application is shown.

[0019] Figure 3 A comparison result of a hybrid policy iteration algorithm and a policy iteration algorithm learning process according to an embodiment of the present application is shown.

[0020] Figure 4 A test phase system state curve diagram based on a hybrid policy iteration algorithm according to an embodiment of the present application is shown.

[0021] Figure 5 A reliability test diagram of a multi-agent collaborative control method based on hybrid policy iteration in a multi-pendulum system according to an embodiment of the present application is shown.

[0022] Figure 6 A multi-pendulum system learning phase evaluation neural network weight change process diagram according to an embodiment of the present application is shown.

[0023] Figure 7 A multi-pendulum system test phase system state curve diagram according to an embodiment of the present application is shown.

[0024] Figure 8 A block diagram of an electronic device suitable for implementing a multi-agent collaborative control method based on hybrid policy iteration according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary of the present application, and is intended to provide a thorough description for implementations of the present application, and to convey the best understanding of the embodiments of the present application. Accordingly, those skilled in the art will appreciate that the present application can be practiced with

[0026] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present application. As used herein, the term "include", "comprise" and the like is meant to be inclusive of the stated features, steps, operations and / or components, but not exclusive of other features, steps, operations and / or components.

[0027] All terms used herein, including technical and scientific terms, have the meaning that is commonly understood by one of ordinary skill in the art unless otherwise defined. It should further be noted that the use of any terms herein should not be interpreted to imply any limitation on the scope of the disclosure. For example, and where appropriate, the singular forms of terms herein can be interpreted to include the plural form of said terms.

[0028] In the case of using expressions such as "at least one of A, B, and C", etc., it generally means that the "at least one of A, B, and C" should be interpreted to include one or intact of A, or one or intact of B, or one or intact of C, or any combination of A, B, and C. The expression "at least one of A, B, and C" should be interpreted to include one or intact of A, or one or intact of B, or one or intact of C, or any combination of A, B, and C.

[0029] The existing multi-agent system cooperative control method based on adaptive dynamic programming includes two typical iterative learning control algorithms, value iteration and policy iteration. Among them, the value iteration algorithm starts from an arbitrary given initial value function, and includes two iterative processes of policy update and value update. In contrast, the starting condition of the policy iteration algorithm is more stringent, which requires a permissible control policy to start the iterative process, and then repeats the policy evaluation and policy improvement iterative processes. The value iteration and policy iteration algorithms have the disadvantages of slow convergence speed and high computational complexity, respectively. In order to achieve a compromise between computational complexity and convergence speed, a generalized policy iteration algorithm is proposed, which effectively improves the learning efficiency of the algorithm. However, both the policy iteration algorithm and the generalized policy iteration algorithm require an initial permissible control policy to start the iterative process, which is usually difficult to obtain. Although the value iteration algorithm does not require prior knowledge of the initial permissible control, it often requires more iteration times, i.e. its convergence speed is slow. In general, the cold start problem of policy iteration and generalized policy iteration and the slow convergence speed problem of value iteration affect the learning efficiency of the iterative process, greatly limiting the application of existing iterative learning algorithms in multi-agent system cooperative control problems.

[0030] Therefore, based on the existing technology, how to provide an efficient multi-agent iterative learning algorithm to avoid the cold start problem and the slow convergence speed problem is of great practical significance for expanding the application prospect of adaptive dynamic programming technology in the multi-agent field.

[0031] The present application provides a multi-agent cooperative control method based on hybrid policy iteration, which is based on the hybrid policy iteration algorithm and can solve the cold start problem of the multi-agent system and solve the multi-agent system cooperative control problem. At the same time, the present application uses the Bellman optimal method to ensure the optimality of the cooperative control policy, uses the hybrid policy iteration algorithm to solve the cold start and slow convergence speed problems of the traditional iterative algorithm, and uses the neural network structure to realize the online implementation of the iterative learning algorithm and obtain the real-time cooperative control policy

[0032] Figure 1 A flow chart of the multi-agent cooperative control method based on hybrid policy iteration according to an embodiment of the present application is shown.

[0033] As shown in Figure 1 The multi-agent cooperative control method based on hybrid policy iteration of this embodiment includes operations S110-S140.

[0034] In operation S110, based on the Bellman optimality principle, the optimal cooperative control strategy of each agent is obtained by using the optimal cooperative control model and the optimal cost function of each agent in the multi-agent system.

[0035] The operation S110 is used to obtain the optimal cost function and the optimal cooperative control model of each agent: the multi-agent system is a typical complex network system, and the corresponding cooperative control problem aims to achieve the consistency of all agents. In this process, each agent achieves cooperative tasks through communication with neighbor agents. The cooperative error of each agent is determined by the state of the agent and the neighbor agents with which it communicates; therefore, based on the form of the cooperative error, a single agent dynamic model (or system dynamic model) is established, and further based on the form of the neighborhood cooperative error, a neighborhood cooperative error system dynamic is constructed, and then a cooperative control model of the multi-agent system is established. From the perspective of each agent, the optimal cost function of the agent is set by considering the convergence of the cooperative error and the control cost of the neighbor agents including the agent. Each agent makes cooperative control decisions by minimizing the designed cost function, and according to the Bellman optimality principle, the optimal cooperative control strategy of the cooperative control model of the multi-agent system is obtained.

[0036] According to the cooperative control requirements of the multi-agent system, the present application can select a quadratic energy type cost function, and then obtain the required optimal cooperative control strategy. For different control requirements, the cost function needs to be changed specifically, for example, considering the case of control input saturation, the relevant limiting function can be added to the control cost item of the cost function.

[0037] In operation S120, the optimal cost function and the optimal cooperative control strategy of each agent are iteratively executed approximation operation and update operation by using a hybrid policy iteration algorithm, wherein the hybrid policy iteration algorithm includes a permissible control policy learning stage and an optimal cooperative control policy exploration stage.

[0038] The application combines the traditional policy iteration and value iteration learning algorithm, and designs a hybrid policy iteration algorithm (or hybrid policy iteration learning algorithm) including a permissible control policy learning stage and an optimal cooperative control policy exploration stage. First, based on the value update characteristics of the value iteration algorithm, a permissible control policy learning stage characterized by a value update process is designed by introducing a new cooperative error function, which is used to obtain the permissible control policy of each agent; each agent starts the iteration process from the initial value function, avoiding the restriction of the initial permissible control start condition of the traditional policy iteration algorithm. Further, each agent based on the obtained permissible control policy, through the repeated execution of the policy evaluation and policy improvement two steps in the optimal cooperative control policy exploration stage, realizes the step-by-step approximation of the optimal cooperative control policy, thereby ensuring the good convergence speed of the iteration learning algorithm. Each agent starts from the initial value function, obtains the permissible control policy through the permissible control policy learning stage, and then realizes the good approximation of the optimal cooperative control policy through the optimal cooperative control policy exploration stage.

[0039] In operation S130, in the process of iteratively performing the approximation operation and the update operation, the evaluation neural network is used to perform real-time evaluation operation on the permissible control policy learning stage and the optimal cooperative control policy exploration stage to obtain a real-time cooperative control policy.

[0040] Based on the hybrid policy iteration algorithm, the evaluation neural network is constructed to obtain the real-time cooperative control policy. For the multi-agent system control model, the evaluation neural network is constructed based on the multi-layer feedforward neural network to approximate the cost function of the iteration learning process, thereby ensuring the online implementation of the hybrid policy iteration algorithm. In the permissible policy exploration stage, the value function update process of each agent is characterized by the value update step, and the value function update process of the optimal cooperative control policy exploration stage is characterized by the policy evaluation step. Based on the evaluation neural network update process of the two learning stages, the weight adaptive update law of the evaluation neural network is obtained by using the normalized gradient descent algorithm. Further, the output value of the evaluation neural network is used to calculate the cooperative control policy. Since the value function update process of each agent in the permissible control policy learning and optimal cooperative control policy exploration stages is characterized by different expressions, the weight update law of the evaluation neural network of each agent is different.

[0041] The adaptive dynamic programming technology based on the hybrid policy iteration algorithm designed by the application can also be implemented by using a behavior-evaluation neural network structure, in which case the control policy is directly given by the behavior network instead of indirectly calculated by the evaluation neural network. Compared with the single evaluation neural network structure, the single evaluation neural network structure is easier to implement and has less computational burden.

[0042] In operation S140, the real-time cooperative control policy is used to perform the cooperative control operation between agents of the multi-agent system.

[0043] The obtained mixed policy iteration based cooperative control strategy is loaded into the multi-agent system.

[0044] The mixed policy iteration based multi-agent cooperative control method provided by the application solves the cold start problem, slow convergence speed and low calculation efficiency of the existing multi-agent cooperative control scheme by using the mixed policy iteration algorithm and the evaluation neural network.

[0045] According to the embodiment of the application, the above-mentioned obtaining of the optimal cooperative control strategy of each agent based on the Bellman optimality principle comprises: modeling each agent according to the attribute information of each agent to obtain a dynamic model of each agent; constructing an optimal cooperative control model of each agent by using the attribute information of the multi-agent system based on the leader-follower structure and the dynamic model of each agent; setting a quadratic optimal cost function for controlling the cost for each agent by using the optimal cooperative control model of each agent and the attribute information of the neighboring agents of each agent; and obtaining the optimal cooperative control strategy of each agent by using the Bellman optimality principle based on the quadratic optimal cost function of each agent.

[0046] According to the embodiment of the application, the above-mentioned modeling of each agent according to the attribute information of each agent to obtain a dynamic model of each agent comprises: modeling each agent according to the internal dynamic information of each agent, a control input matrix, an optimal cooperative control strategy to be solved, and the number of agents in the multi-agent system to obtain a dynamic model of each agent.

[0047] According to the embodiment of the application, the above-mentioned constructing of the optimal cooperative control model of each agent by using the attribute information of the multi-agent system based on the leader-follower structure and the dynamic model of each agent comprises: obtaining a local neighborhood cooperative error matrix of each agent, and constructing the optimal cooperative control model of each agent according to the internal dynamic information of the multi-agent system and the dynamic model and the local neighborhood cooperative error matrix of each agent.

[0048] The following detailed description, through specific implementation methods, further illustrates the process of constructing the optimal cost function for each agent in the multi-agent system involved in the above embodiments of the present invention and the process of obtaining the optimal cooperative control strategy.

[0049] First, in order to further construct the cooperative control model of the multi-agent system, we must first establish the system dynamic model of a single agent, as shown in formula (1):

[0050] (1).

[0051] in, Index for the number of agents. The total number of agents; Represents intelligent agents The system dynamic model, Represents intelligent agents The system status; It is an intelligent agent and Relevant internal system dynamic information, It is an intelligent agent and The relevant control input matrix; It is an intelligent agent The control input. Note that the multi-agent system involved in this invention is a leader-follower structure, when... The term "leader" indicates that the agent is the leader and the others are followers. The ultimate goal of multi-agent collaborative control based on the leader-follower structure is to achieve consistency between the state of all followers and the state of the leader.

[0052] Secondly, based on the neighborhood cooperative error form, a neighborhood error system dynamic is constructed, and then a cooperative control model for the multi-agent system is established: In the multi-agent system, each agent can only communicate with its neighboring agents. During the cooperative process, each agent obtains the corresponding local neighborhood cooperative error by communicating with its neighboring agents, thereby adjusting the control strategy to make the local neighborhood cooperative error converge, and finally realizing the cooperation of the entire multi-agent system. According to formula (2), a cooperative control model is established from the perspective of each agent:

[0053] (2).

[0054] in, Represents intelligent agents The collaborative control model or error system Represents intelligent agents Cooperative error, Represents intelligent agents The system dynamic model, Represents intelligent agents The system status, In the leader's state, A dynamic model of the system representing the leader. It is an intelligent agent A collection of neighboring intelligent agents, Adjacency matrix No. line, number Column elements, reachability matrix The Middle One element, It is an intelligent agent The internal dynamic information of the error system, Laplace matrix The Middle line, number The elements of the column. For the agent. In other words, It is a cooperative error, also known as a local neighborhood error.

[0055] Then, the cost function for each agent is designed. To achieve good cooperative performance with minimal control cost, the agent is defined by comprehensively considering the cooperative error convergence and the control costs of neighboring agents, including the current agent. (or the first) (a number of agents), and the cost function at time t. ,in The intelligent agent at time t The cooperative error, i.e. The real-time value. Specifically, the cost function is shown in formula (3):

[0056] (3).

[0057] in, For intelligent agents With cooperative error The relevant function, called the system operating cost, needs to satisfy the positive definite condition, i.e. ; and Representing intelligent agents respectively and intelligent agents Controlling costs, among which and These are positive definite and symmetric first and second constant matrices, used to adjust the component of control cost in the cost function. More specifically, Represents intelligent agents Control costs in intelligent agents Cost function The proportion of in, Representing neighboring intelligent agents Control costs in intelligent agents Cost function The proportion of; Denotes the transpose of the first constant matrix. Denotes the transpose of the first constant matrix. Represents intelligent agents Transpose of the control policy (policy vector); For the adjacency matrix, the first... line, number Column elements; It is an intelligent agent A collection of neighboring intelligent agents, This is the control strategy of agent j. It can be seen that this cost function includes the control costs of all neighboring agents. By minimizing this cost function, each agent can minimize its control cost while ensuring local neighborhood error (i.e., cooperative error). The convergence of the functions ultimately enables the collaborative task of the entire multi-agent system.

[0058] Finally, the optimal cooperative control strategy is obtained. During the cooperative process, each agent aims to minimize its own cost function, thus obtaining the agent's optimal control strategy. With cooperative error Related optimal cost function This ensures that the entire multi-agent system achieves the fastest collaborative effect with minimal control cost. (In the agent...) Optimal cost function Based on this, and according to the Bellman optimality principle, the intelligent agent is obtained. With cooperative error Related optimal cooperative control strategies As shown in formula (4):

[0059] (4).

[0060] in, It is an intelligent agent Optimal cost function (i.e., the optimal cost function at time t) regarding The partial derivative or gradient of (i.e., the cooperative error at time t). Denotes the inverse of the first constant matrix. Represents intelligent agents control input matrix The optimal cooperative control strategy for the multi-agent system constructed using the transpose of the matrix and formula (4) is: ,in represents the first agent and the coordination error the optimal coordination control strategy, represents the second agent and the coordination error the optimal coordination control strategy, represents the Nth agent and the coordination error the optimal coordination control strategy.

[0061] After obtaining the optimal control model, a hybrid policy iteration algorithm can be designed to realize learning of the admissible control policy and approximation of the optimal coordination strategy.

[0062] According to the embodiments of the present application, the approximation and updating operations of the optimal cost function and the optimal coordination control strategy of each agent by using the hybrid policy iteration algorithm include: setting an admissible control policy learning stage represented by value updating and a stop condition of the admissible control policy learning stage; in the admissible control policy learning stage, based on the stop condition, the approximation and updating operations of the optimal cost function and the optimal coordination control strategy of each agent are performed to obtain a current value function and a current admissible control policy, wherein the current value function is a current approximation result of the optimal cost function, and the current admissible control policy is a current approximation result of the optimal coordination control strategy.

[0063] According to the embodiments of the present application, the approximation and updating operations of the optimal cost function and the optimal coordination control strategy of each agent by using the hybrid policy iteration algorithm further include: constructing an optimal control policy exploration stage with a policy evaluation operation and a policy improvement operation; in the policy evaluation operation, based on a preset evaluation condition, the current value function is solved by performing the policy evaluation operation on the current admissible control policy iteration to obtain an updated value function; in the policy improvement operation, the current admissible control policy iteration is performed by using the updated value function to obtain an updated admissible control policy.

[0064] According to the embodiments of the present application, the setting of the admissible control policy learning stage represented by value updating and the stop condition of the admissible control policy learning stage includes: replacing the neighborhood coordination error function of each agent with a preset neighborhood coordination error approximation function, wherein the value range of the preset neighborhood coordination error approximation function is greater than the value range of the neighborhood coordination error function; determining a value updating process of the value function by using the preset neighborhood coordination error approximation function, a deterministic sequence and an arbitrary initial value function; and determining the stop condition according to the preset neighborhood coordination error approximation function, the value range of the neighborhood coordination error function, the deterministic sequence and the difference of the value functions in the adjacent value updating processes.

[0065] The hybrid policy iteration algorithm provided by the above embodiments of the present application will be further described in detail through specific embodiments.

[0066] First, a learning phase for the admissible control policy, characterized by value updates, is designed. Each agent starts with an arbitrary initial value function. Start iterative updates, among which Represents intelligent agents At the initial iteration step (i.e., step 0), and the cooperative error The relevant initial value function, Let represent the set of non-negative definite functions. Its learning objective is to obtain a value function corresponding to an admissible control policy. Once the admissible control policy is obtained, the agent's first phase stops, and iterative learning proceeds in the next phase using the obtained admissible control policy. The deterministic sequence shown below is selected. As shown in formula (5):

[0067] (5).

[0068] in It is an intelligent agent A positive real number chosen in the k-th iteration step Where k is the iteration step index, from 1 to 1. . This represents the sum of the sequences. This indicates that the sequence tends towards at iteration step k. The limits of time.

[0069] Furthermore, intelligent agents Selection function Replace the original neighborhood cooperative error correlation function And satisfy ,function Represents intelligent agents With cooperative error The relevant actual system operating costs. Then, during the allowable control policy learning phase, the learning process of each agent, i.e., the value function update process, can be expressed by formula (6):

[0070] (6).

[0071] Among them, in formula (6) For indexing intelligent agents, This serves as an index for the iterative process. For intelligent agents In the The value function at the next iteration For intelligent agents In the The value function at the next iteration For intelligent agents exist In the next iteration, the value function is related to the cooperative error. gradient, It is an intelligent agent exist Control strategy during the next iteration Is it the intelligent agent j in The transpose of the control strategy in the next iteration. Other related symbols ( This is consistent with the formula mentioned above.

[0072] Secondly, design conditions that allow the control strategy to stop learning. (Agent) The learning process of the permissive control strategy stops when the iterative index satisfies the following inequality, as shown in formula (7):

[0073] (7).

[0074] in The meaning is consistent with the aforementioned formula, and the intelligent agent In the The value function at the next iteration satisfy ,in Denotes the set of positive definite functions, with the symbol... This indicates belonging to. The agent at this point... The flag for exiting the iteration is: i.e., intelligent agent The exploration of the permissive control policy ends at iteration k. (Agent) exist The iterative value function corresponding to the step is The control strategy corresponding to this value function To allow control strategies, where It is an intelligent agent The indicator for exiting the iteration. At this point, the agent... The first phase of learning has ended, and the permissive control strategy has been obtained. It is used to initiate the next stage of the learning process.

[0075] Then, an optimal control policy exploration phase, namely the second phase, is constructed, consisting of two steps: policy evaluation and policy improvement. For the agent... The resulting permissive control strategy Starting from the current k steps (subsequent iterations satisfy the following formula), we can determine the next step (k steps). ) control strategy Perform policy evaluation and solve the agent problem. The value function at k iteration steps As shown in formula (8):

[0076] (8).

[0077] in Represents intelligent agents The Hamiltonian function, which is related to the cooperative error and the iterative control strategy at step k and the value function at step k gradient These three factors are related. More specifically, the above strategy evaluation steps can be further expressed by formula (9):

[0078] (9).

[0079] Some mathematical symbols in formula (9) The meaning is the same as that of the formula mentioned above, so it will not be repeated here. Represents intelligent agents In the The control cost in a Hamiltonian function, and Represents intelligent agents In the Control costs in a Hamiltonian function; Represents intelligent agents Iterative control strategy at step k transpose, Represents intelligent agents Iterative control strategy at step k transpose, Represents intelligent agents gradient of the value function at step k The transpose of .

[0080] In the strategy enhancement step, based on intelligent agents Value function at the k-th iteration step The control strategy is further updated as shown in formula (10):

[0081] (10).

[0082] in It is an intelligent agent In the The control strategy obtained through iterative steps It is a value function Regarding cooperative error gradient, Denotes the inverse of the first constant matrix. It is an intelligent agent control input matrix The transpose of , and the other symbols have the same meaning as in the previous formula, and will not be repeated here.

[0083] According to the embodiment of the present application, in the process of iteratively performing the approximation operation and the update operation, the evaluation neural network is used to perform real-time evaluation operation in the allowable control policy learning stage and the optimal collaborative control policy exploration stage, and a real-time collaborative control policy is obtained. In the process of iteratively performing the approximation operation and the update operation in the allowable control policy learning stage, the evaluation neural network is used to reconstruct the allowable control policy learning stage, and the evaluation neural network is used to perform real-time update operation on the current value function and real-time evaluation operation on the current allowable control policy. In the process of iteratively performing the approximation operation and the update operation in the optimal collaborative control policy exploration stage, the evaluation neural network is used to reconstruct the optimal collaborative control policy exploration stage, and the approximation Hamilton equation and the evaluation neural network are used to perform real-time update operation on the updated current value function and real-time evaluation operation on the updated allowable control policy, thereby obtaining the real-time collaborative control policy.

[0084] According to the embodiment of the present application, the evaluation neural network is used to perform real-time update operation on the current value function and real-time evaluation operation on the current allowable control policy, which includes the following steps. A three-layer feedforward neural network is used to construct the evaluation neural network. The weights and the activation function of the evaluation neural network are used to represent the current value function, and the partial derivative of the weights and the activation function of the evaluation neural network is used to represent the gradient function of the current value function. In the process of real-time updating of the current value function, the first learning error function of the evaluation neural network is determined by using the deterministic sequence, the weights of the evaluation neural network, and the value function of the last iteration round. The current allowable control policy is reconstructed based on the partial derivative of the weights and the activation function of the evaluation neural network. The weights of the evaluation neural network are reconstructed by using the activation function of the evaluation neural network and the first learning error function based on the first normalized gradient descent algorithm and the first minimization target function, and the weight update rule of the allowable control policy learning stage is obtained. The real-time update operation on the current value function and the real-time evaluation operation on the current allowable control policy are performed by using the weight update rule of the allowable control policy learning stage, and the updated value function and the updated allowable control policy are obtained.

[0085] According to an embodiment of the present invention, the above-described method of performing real-time update operations on the updated current value function and real-time evaluation operations on the updated admissible control strategy using the approximate Hamiltonian equation and evaluation neural network to obtain a real-time cooperative control strategy includes: reconstructing a second learning error function using the partial derivatives of the activation function of the approximate Hamiltonian equation and evaluation neural network; reconstructing the updated admissible control strategy using the weights and partial derivatives of the activation function of the evaluation neural network; based on a second normalized gradient descent algorithm and a second minimization objective function, reconstructing the weights of the evaluation neural network using the partial derivatives of the activation function and the second learning error function to obtain a weight update rule for the optimal cooperative control strategy exploration phase; and performing a real-time update operation on the updated value function using the weight update rule for the optimal cooperative control strategy exploration phase and performing a real-time evaluation operation on the updated admissible control strategy to obtain a real-time cooperative control strategy.

[0086] The evaluation neural network provided by the present invention will be further described in detail below through specific embodiments.

[0087] First, an evaluation neural network is constructed using a three-layer feedforward network. In this specific implementation, a three-layer feedforward neural network is used to construct the evaluation neural network to approximate the optimal cost function online. At this point, the agent... Optimal value function and its gradient (in The optimal value of a variable (and so on) can be expressed by formulas (11) and (12):

[0088] (11),

[0089] (12).

[0090] in, It is an intelligent agent At time t, and the cooperative error The relevant evaluation of neural network activation functions, It is an activation function gradient, Represents the gradient of the activation function transpose; It is an intelligent agent At time t, and the cooperative error The relevant evaluation of the reconstruction error of the neural network, It is a reconstruction error The gradient; For intelligent agents To evaluate the ideal weights of a neural network, It is the transpose of the ideal weights. the number of neurons. In addition, the symbol represents a mathematical meaning belonging to, the set of real vectors of dimension .

[0091] Since the ideal weight of the critic neural network is unknown, the estimated weight is used to approximate the iterative value function and its gradient (where represents the approximation of a certain variable, and the following are similar and will not be repeated), at this time, the approximate output value of the critic neural network is , which is represented by the critic neural network weight and the activation function , as shown in equations (13) and (14):

[0092] (13),

[0093] (14).

[0094] where, is the transpose of the estimated weight of the agent ; represents the activation function of the critic neural network of the agent , which is a simplified representation of , simplified as , represents the gradient of ; is the approximate value function of the agent about the collaborative error ; is the approximate value function of the agent about . Further, the critic neural network weight estimation error can be represented as . By introducing the critic neural network, the collaborative control strategy of the agent during the iteration process can be represented by equation (15)

[0095] (15).

[0096] where represents the inverse matrix of the first constant matrix, represents the transpose matrix of the control input matrix of the agent . is the transpose of the value function gradient Other mathematical symbols are described in the foregoing equations.

[0097] Second, the evaluation neural network is used to reconstruct the control policy learning phase learning process. Let be the first value update process, the agent corresponding to the evaluation neural network estimate weight, that is, the current evaluation neural network estimate weight At this time, the corresponding iterative value function and its gradient can be represented by equations (16) and (17):

[0098] (16),

[0099] (17).

[0100] wherein is the agent about the collaborative error in the kth iteration step, the approximate value function is the gradient about the collaborative error ; is the agent the first value update process, the estimated weight transpose; other mathematical symbols are described in the foregoing equations.

[0101] The evaluation neural network learning error in the value update process can be represented by (wherein subscript refers to the first stage), as shown in equation (18):

[0102] (18).

[0103] wherein is the agent in the last time, that is, the value update process, the value function obtained, which can be regarded as a constant in the current value update process, is the weight of the evaluation neural network in the last value update process (i.e. iteration step), denotes its transpose. denotes the gradient about the collaborative error , and denotes its transpose. is the agent corresponding to the value function The cooperative control policy at the kth iteration step, is its transpose; is the critic neural network of the agent The cooperative control policy at the kth iteration step, is its transpose. Other symbols are defined in the foregoing. Next, the critic neural network structure is evaluated, which can be further written in the form shown in equation (19):

[0104] (19).

[0105] Then, the corresponding critic neural network weight update law based on the first normalized gradient descent algorithm is designed to minimize the objective function where denotes the objective function of the first stage. Then, the weight update law of the allowable control policy learning stage is

[0106] (20).

[0107] where, denotes the critic network estimated weight of the agent , and denotes its derivative; is the learning rate of the critic neural network of the agent , and is used for normalization; denotes the mathematical operation of partial differentiation.

[0108] Then, the optimal control policy exploration stage learning process is reconstructed based on the critic neural network, i.e., the second stage. Based on the critic neural network estimated weight , the learning error represented by the approximate Hamiltonian function (where subscript refers to the second stage) can be represented by equation (21):

[0109] (21).

[0110] where, denotes the approximate Hamiltonian function of the agent , which is an approximation of the Hamiltonian function in equation (8); is the follower agent based on the cooperative control policy obtained by the critic neural network at the kth iteration step, and other mathematical symbols are described in the foregoing equations. Further, the iterative cooperative control policy is shown in equation (22):

[0111] (twenty two).

[0112] In addition, the neighboring intelligent agent Iterative control strategy in the k-th iteration step It is also given in a similar form. The objective function for this learning phase is further defined as follows: ,in Let represent the objective function for the second stage. To minimize the second minimization objective function... The corresponding second normalized gradient descent algorithm is designed to update the evaluation neural network, and the update rule is shown in formula (23):

[0113] (twenty three).

[0114] in, Represents intelligent agents The regression vector for the agent In other words, It is an activation function gradient, This is an iterative cooperative control strategy based on the evaluation neural network obtained at the k-th iteration step. The th in the reachability matrix One element, It is an intelligent agent The internal dynamic information of the error system, The first in the Laplace matrix line, number Column elements, It is an intelligent agent With state The relevant control input matrix; Represents the transpose of the regression vector; For the first The evaluation neural network learning rate for each follower agent Used to achieve normalization. The remaining mathematical symbols have been described in formula (20) above and will not be explained again.

[0115] After the mixed policy iteration algorithm and the evaluation neural network are designed, the obtained collaborative control strategy is applied to the multi-agent system: the mixed policy iteration algorithm is loaded into the multi-agent system. In some embodiments, the mixed policy iteration algorithm can be loaded into the multi-agent system in the form of a computer program and mathematical symbols, at which time each agent in the multi-agent system starts from an arbitrary initial value function, and the evaluation neural network weight is updated continuously through the allowable control policy learning stage until the allowable control policy is obtained; further, based on the allowable control policy, the optimal collaborative strategy is explored, and the evaluation neural network weight is updated continuously through the policy evaluation and policy improvement cycle until convergence. The collaborative control strategy is obtained and the multi-agent collaborative control is realized. Under the action of the mixed policy iteration algorithm, the evaluation neural network weight of each agent is continuously updated. Based on the evaluation neural network, the real-time collaborative control strategy is obtained and loaded into the multi-agent system, and finally the multi-agent collaborative control is realized.

[0116] Figure 2 A multi-agent communication topology diagram according to an embodiment of the application is shown.

[0117] Figure 3 A comparison result of the mixed policy iteration algorithm and the policy iteration algorithm learning process according to an embodiment of the application is shown.

[0118] Figure 4 A system state curve diagram of the test stage system based on the mixed policy iteration algorithm according to an embodiment of the application is shown.

[0119] Figure 5 A reliability test diagram of the multi-agent collaborative control method based on the mixed policy iteration according to an embodiment of the application in a multi-pendulum system is shown.

[0120] Figure 6 A multi-pendulum system learning stage evaluation neural network weight change process diagram according to an embodiment of the application is shown.

[0121] Figure 7 A multi-pendulum system test stage system state curve diagram according to an embodiment of the application is shown.

[0122] The multi-agent method provided by the application is further described and verified in detail below through specific experiments and in combination with the drawings.

[0123] Specific experiment 1 relates to a nonlinear continuous-time multi-agent system, as shown below:

[0124] First, Figure 2The communication topology diagram of the multi-agent system according to the embodiment of the present application is shown, and it can be seen that the multi-agent cooperative control method based on mixed policy iteration provided by the present application is applied to a multi-agent system composed of a single leader (i.e., a leader agent) and five followers (i.e., follower agents), wherein 0 represents the leader agent, and 1-5 represent the follower agents. The corresponding Laplacian matrix and the reachable matrix are shown in the following formula (24):

[0125] (24),

[0126] wherein diag represents that the matrix is a diagonal matrix, and specifically, each element of the reachable matrix (or ) is a diagonal element, i.e. .

[0127] Based on the nonlinear continuous-time multi-agent system shown in Figure 2 , the designed cooperative control method finally realizes the consistency of the states of all followers and the leader. Then, the operations S110-S140 shown in Figure 1 are continued to be performed.

[0128] According to the operation S110, first, the system dynamics of a single agent are established, and the nonlinear multi-agent dynamics are expressed by the following formula (25):

[0129] (25).

[0130] wherein . is the state vector of each follower agent, wherein is the first state component of the agent and is the second state component of the agent . The initial states of the five followers are , , , and , respectively. In addition, the dynamics of the leader agent (referred to as the leader) are expressed by the following formula (26):

[0131] (26).

[0132] wherein is the state vector of the leader, wherein is the first state component of the leader and ​is the second state component of the leader. The initial state vector of the leader is chosen as . Then the cooperative control model is established according to formula (2). For all follower agents, the configuration of the related parameters in the cost function is as follows: , and , where is the first component of the cooperative error of the agent and is the second component of the cooperative error of the agent . According to operation S120, the following is selected: and

[0133] , where k represents the iteration index. According to operation S130, in the present embodiment, 3 nodes are configured in the hidden layer of the evaluation neural network of each agent, and the activation function of the agent

[0134] and the gradient of the agent with respect to the cooperative error are set as shown in formula (27): (27).

[0135] The learning rate of the evaluation neural network of each follower agent is set as . Further, in order to ensure that the initial value function of the hybrid policy iteration algorithm satisfies the initial condition, the initial weight of the evaluation neural network is randomly selected between 0 and 2. Finally, the initial weights of the evaluation neural networks of the follower agents are randomly selected as

[0136] , , , , , , , respectively. Then the corresponding initial value functions are 2.45, 1.38, 3.77, 3.28 and 8.22 respectively, all of which satisfy the requirement of the hybrid policy iteration algorithm for the initial value function, i.e. all initial value functions are non-negative.

[0137] According to operation S140, the nonlinear continuous-time multi-agent system calculates the cooperative control strategy in the learning phase of the allowable control strategy according to formula (19), and calculates the cooperative control strategy in the exploration phase of the optimal cooperative control strategy according to formula (22). Each follower agent realizes the iterative learning process based on the hybrid policy iteration algorithm by constructing the evaluation neural network, and obtains the real-time cooperative control strategy by introducing the evaluation neural network, thereby completing the cooperative control of the multi-agent system.

[0138] Further, the designed hybrid policy iteration algorithm is compared with the traditional policy iteration algorithm and the value iteration algorithm under the same initial weight condition of the evaluation neural network to verify the advantages of the designed hybrid policy iteration algorithm in initial starting condition and convergence speed, i.e., to prove the beneficial effect of the designed hybrid policy iteration algorithm on improving learning efficiency.

[0139] Figure 3 is the comparison result of the hybrid policy iteration algorithm and the policy iteration algorithm learning process according to this embodiment, wherein, Figure 3 agent1~agent5 in agent1~agent5 represent follower agent1~follower agent5, represents the norm of the approximate coordination error, represents the coordination error is the approximate estimated value in the learning process, represents the agent represents the norm of the evaluation network estimated weight, represents the ideal evaluation weight is the approximate estimated value in the learning process, represents the norm operation; Figure 3 (a) and Figure 3 (b) in (b) are respectively the convergence processes of the evaluation neural network weight and the coordination error in the learning process of the designed hybrid policy iteration algorithm. Figure 3 (c) and Figure 3 (d) in (d) are respectively the convergence processes of the evaluation neural network weight and the coordination error in the learning process of the traditional policy iteration algorithm. It can be seen that, under the same initial weight, the network weight of each agent and the coordination error in the learning process of the hybrid policy iteration algorithm converge smoothly, and the optimal coordination control strategy is obtained at the iteration index (i.e., the agent obtains the allowable control strategy at the iteration step) obtains the respective allowable control strategy. In contrast, based on the traditional policy iteration algorithm, the coordination error of each follower agent in the training stage finally diverges, and the weight update process of the evaluation neural network also stops at the same time, which proves that the control strategy corresponding to the initial weight of the evaluation neural network selected by the above hybrid policy iteration algorithm is not allowable. Therefore, the comparison result indicates that the hybrid policy iteration algorithm successfully obtains the allowable control strategy of each agent in the allowable strategy exploration stage, provides the necessary starting condition for the optimal coordination control strategy exploration stage, and finally realizes the convergence of the system state and the evaluation neural network weight. Figure 4 is the system state curve in the test stage based on the hybrid policy iteration algorithm according to this embodiment, and it can be seen that the state change trends of the agents are similar, Figure 4 represents the agent ​The first state component Represents intelligent agents The second state component, where ;from Figure 4 As can be seen, convergence is achieved in approximately 15 seconds, demonstrating the excellent collaborative performance of the multi-agent cooperative control method based on hybrid strategy iteration provided by this invention.

[0140] Furthermore, by introducing the algorithm's CPU time, the advantages of the hybrid policy iteration algorithm provided in this invention in terms of convergence speed are compared with those of the traditional policy iteration and value iteration algorithms. Table 1 shows the comparison results. It can be seen that among the three iterative learning algorithms, the policy iteration algorithm has the shortest CPU time and the fastest convergence speed. In contrast, the hybrid policy iteration algorithm, due to the presence of a permissive policy exploration phase, has a slightly slower convergence speed than policy iteration, exhibiting a slightly longer CPU time. However, it avoids the limitations of permissive control policies, greatly reducing the difficulty of algorithm application. Compared with the hybrid policy iteration algorithm and policy iteration, the traditional value iteration has the slowest convergence speed. Therefore, in general, the hybrid policy iteration algorithm, while ensuring a faster convergence speed, avoids the limitations of permissive control policies, exhibiting higher learning efficiency, as shown in Table 1.

[0141] Table 1: Comparison of three learning algorithms in multi-agent cooperative control problems

[0142] Iterative learning method Hybrid policy iteration algorithm Policy iteration Value iteration CPU time (s) 0.3937 0.3770 0.6919

[0143] Based on Experiment 1, Experiment 2 considers a multi-pendulum system, such as... Figure 5 As shown, this is to verify the reliability of the multi-agent cooperative control method based on a hybrid strategy iterative algorithm provided by the present invention in a practical system, wherein, Figure 5 In For the mass of each pendulum, The length of the pendulum, It is the acceleration due to gravity. It is the product of mass and gravitational acceleration, that is, the weight of each pendulum. For each follower pendulum, the control torque received is and has . The angle between the pendulum and the vertical direction.

[0144] First, the communication topology of the multi-agent system in this embodiment is consistent with that of Experiment 1, see... Figure 6 The correlation matrix is ​​shown in formula (24), and then, according to Figure 1 The operations shown, S110-S140, continue to be executed.

[0145] According to operation S110, the system dynamics of a single agent are first established. The pendulum system dynamics are represented by the following linearized model, as shown in formula (28):

[0146] (28).

[0147] in, For the mass of each pendulum, The length of the pendulum is 1 meter. It is the acceleration due to gravity. For each follower pendulum, the control torque received (assuming a DC motor is used) is given, and there is . The angle between the pendulum and the vertical direction. Represents angular velocity This represents angular acceleration. For the first... A pendulum, choose its state vector. for The first state component It's an angle. The second state component It is angular velocity Then the pendulum model can be rewritten in the form shown in formula (28):

[0148] (29).

[0149] in, Furthermore, the leader's initial state is chosen as follows: The initial states of the five follower agents are as follows: , , , , For intelligent agents In this regard, the parameters in the cost function are: , as well as ,in It is an intelligent agent Cooperative error The first component is It is an intelligent agent Cooperative error The second component.

[0150] According to operation S120, select and , where k represents the iteration index.

[0151] According to operation S130, in this specific experiment, two nodes are configured in the hidden layer of the evaluation neural network, and the agent... activation function And its relation to cooperative error gradient The settings are as shown in formula (30):

[0152] (30).

[0153] The learning rate of the evaluation neural network for each follower agent is set to... Furthermore, to ensure that the initial value function of the hybrid strategy iterative algorithm satisfies the initial conditions, the initial weights of the evaluation neural network are randomly selected between 0 and 10.

[0154] According to operation S140, the nonlinear continuous-time multi-agent system calculates the cooperative control strategy in the allowable control strategy learning phase according to formula (19), and calculates the cooperative control strategy in the optimal cooperative control strategy exploration phase according to formula (22). Each follower agent realizes the learning process of the hybrid strategy iterative algorithm by constructing an evaluation neural network, and obtains the real-time cooperative control strategy by introducing the evaluation neural network, thereby completing the multi-agent cooperative control.

[0155] Figure 6 This embodiment illustrates the evaluation of neural network weight changes during the learning phase of a multi-pendulum system. It shows how each follower agent (i.e., ...) can be observed. Figure 6 The weights of the evaluation neural network (agent1, agent2, agent3, agent4, agent5) converge in about 40 seconds. Represents intelligent agents Evaluate the norm of the network's estimated weights. Indicates the ideal evaluation weight Approximate estimates during the learning process This represents the norm operation. It also represents the iteration index corresponding to when each follower agent discovers an admissible control policy. (i.e., the first) The iterative steps for each agent to obtain the permissible control policy are as follows: That is, each intelligent agent quickly obtains the permissible control policy, thereby carrying out the subsequent iterative learning process. Figure 7 This embodiment illustrates the system state curves during the testing phase of a multi-pendulum system, where leader represents the leader agent, and agents1 to 5 represent follower agents 1 to 5. Represents intelligent agents The first state component Represents intelligent agents The second state component, where It can be seen that the states of each follower agent finally reach an agreement with the leader agent at about time 25 seconds to 30 seconds, and the corresponding neighborhood coordination error also converges, which embodies good coordination effect.

[0156] It can be obviously seen that the mixed policy iteration algorithm designed in the application can reduce the start condition limitation of the iteration algorithm, improve the convergence speed and show good learning efficiency while ensuring good coordination effect.

[0157] In combination with the results of the specific experiment 1 and the specific experiment 2, the multi-agent coordination control method based on the mixed policy iteration provided in the embodiments of the application realizes the beneficial effect of anti-cold start by using the mixed policy iteration algorithm and the evaluation neural network.

[0158] Figure 8 A block diagram of an electronic device suitable for implementing the multi-agent coordination control method based on the mixed policy iteration according to the embodiments of the application is shown.

[0159] As shown in Figure 8 The electronic device 800 according to the embodiments of the application includes a processor 801 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 802 or loaded from a storage portion 808 into a random access memory (RAM) 803. The processor 801 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and the like. The processor 801 can also include an on-board memory for cache use. The processor 801 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to the embodiments of the application.

[0160] In the RAM 803, various programs and data required for the operation of the electronic device 800 are stored. The processor 801, the ROM 802 and the RAM 803 are connected to each other through a bus 804. The processor 801 performs various operations of the method processes according to the embodiments of the application by executing the programs in the ROM 802 and / or the RAM 803. It should be noted that the programs can also be stored in one or more memories other than the ROM 802 and the RAM 803. The processor 801 can also perform various operations of the method processes according to the embodiments of the application by executing the programs stored in the one or more memories.

[0161] According to an embodiment of the present application, the electronic device 800 can further include an input / output (I / O) interface 805 that is also connected to the bus 804. The electronic device 800 can further include one or more of the following components connected to the input / output (I / O) interface 805: an input part 806 including a keyboard, a mouse, etc.; an output part 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 808 including a hard disk, etc.; and a communication part 809 including a network interface card such as a LAN card, a modem, etc. The communication part 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output (I / O) interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 810 as necessary, so that a computer program read out therefrom is installed in the storage part 808 as necessary.

[0162] The present application also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present application.

[0163] According to an embodiment of the present application, the computer readable storage medium can be a non-volatile computer readable storage medium, which can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present application, the computer readable storage medium can include the ROM 802 and / or the RAM 803 described above, and / or one or more memories other than the ROM 802 and the RAM 803.

[0164] The computer program product of the present application can be a computer program product comprising a computer readable storage medium and a computer program mechanism embedded in the computer readable storage medium. Such computer program product can further include a computer readable storage medium and program means for causing a processor or other programmable processing apparatus to function in a particular manner, such that the computer program mechanism that can be executed by such a processor or processing apparatus cause the processor or processing apparatus to implement the functions of the system, method and computer program product of the present application.

[0165] It will be appreciated by persons skilled in the art that features of the various embodiments described herein can be combined and / or substituted into each other's embodiments, even if this is not explicitly stated in the present disclosure. In particular, features of the various embodiments described herein can be combined and / or substituted into each other's embodiments without departing from the spirit and teachings of the present application. All such combinations and / or substitutions are within the scope of the present application.

[0166] The above-described embodiments of the application have been described in order to allow a clear and consistent understanding of the application. Subsequently, those embodiments can be modified or combined into other embodiments in order to obtain a patentable range. Although the present application has been described with reference to the embodiments illustrated in the drawings, it is understood that the present application should not be limited to the embodiments and can be implemented in other ways. The present application is not limited to the above-described embodiments and can be implemented in other ways.

Claims

1. A multi-agent cooperative control method based on hybrid policy iteration, characterized in that, The method comprises: Based on the Bellman optimality principle, the optimal cooperative control model and the optimal cost function of each agent in the multi-agent system are used to obtain the optimal cooperative control strategy of each agent; The optimal cost function and the optimal cooperative control strategy of each agent are iteratively executed by using the mixed policy iteration algorithm, wherein the mixed policy iteration algorithm comprises a permissible control policy learning stage and an optimal cooperative control policy exploration stage; During the iterative execution of the approximation operation and the update operation, the permissible control policy learning stage and the optimal cooperative control policy exploration stage are executed in real time by using the evaluation neural network to obtain a real-time cooperative control strategy; The real-time cooperative control strategy is used to perform cooperative control operation between agents in the multi-agent system.

2. The method of claim 1, wherein, Based on the Bellman optimality principle, the optimal cooperative control model and the optimal cost function of each agent in the multi-agent system are used to obtain the optimal cooperative control strategy of each agent, which comprises: Modeling each agent according to the attribute information of each agent to obtain a dynamic model of each agent; Using the attribute information of the multi-agent system based on the leader-following structure and the dynamic model of each agent to construct an optimal cooperative control model of each agent; Setting a quadratic optimal cost function for each agent for controlling the cost by using the optimal cooperative control model of each agent and the attribute information of the neighboring agents of each agent; Based on the quadratic optimal cost function of each agent, the Bellman optimality principle is used to obtain the optimal cooperative control strategy of each agent.

3. The method of claim 2, wherein, Modeling each agent according to the attribute information of each agent to obtain a dynamic model of each agent comprises: Modeling each agent according to the system internal dynamic information of each agent, the control input matrix, the optimal cooperative control strategy to be solved, and the number of agents in the multi-agent system to obtain a dynamic model of each agent.

4. The method of claim 2, wherein, Using the attribute information of the multi-agent system based on the leader-following structure and the dynamic model of each agent to construct an optimal cooperative control model of each agent comprises: Obtaining a local neighborhood cooperative error matrix of each agent, and constructing an optimal cooperative control model of each agent according to the internal dynamic information of the multi-agent system and the dynamic model and the local neighborhood cooperative error matrix of each agent.

5. The method of claim 1, wherein, Iteratively executing the approximation operation and the update operation of the optimal cost function and the optimal cooperative control strategy of each agent by using the mixed policy iteration algorithm comprises: Setting a permissible control policy learning stage represented by a value update table and a stop condition of the permissible control policy learning stage; In the allowable control policy learning stage, based on the stop condition, the optimal cost function and the optimal collaborative control policy of each agent are iteratively executed approximation operation and update operation to obtain a current value function and a current allowable control policy, wherein the current value function is a current approximation result of the optimal cost function, and the current allowable control policy is a current approximation result of the optimal collaborative control policy.

6. The method of claim 5, wherein, Further comprising: An optimal control policy exploration stage with a policy evaluation operation and a policy improvement operation is constructed. In the policy evaluation operation, based on a preset evaluation condition, the current value function is solved by iteratively executing the policy evaluation operation on the current allowable control policy to obtain an updated value function. In the policy improvement operation, based on the updated value function, the policy improvement operation is iteratively executed on the current allowable control policy to obtain an updated allowable control policy.

7. The method of claim 5, wherein, The allowable control policy learning stage characterized by value updates and the stop condition of the allowable control policy learning stage include: A preset neighborhood collaborative error approximation function is used to replace the neighborhood collaborative error function of each agent, wherein the value range of the preset neighborhood collaborative error approximation function is greater than the value range of the neighborhood collaborative error function; A value update process of the value function is determined by using the preset neighborhood collaborative error approximation function, a deterministic sequence and an arbitrary initial value function; The stop condition is determined according to the preset neighborhood collaborative error approximation function, the value range of the neighborhood collaborative error function, the deterministic sequence and the difference of the value functions of adjacent value update processes.

8. The method of claim 5, wherein, In the process of iteratively executing approximation operation and update operation, an evaluation neural network is used to perform real-time evaluation operation on the allowable control policy learning stage and the optimal collaborative control policy exploration stage to obtain a real-time collaborative control policy, including: In the process of iteratively executing approximation operation and update operation in the allowable control policy learning stage, the allowable control policy learning stage is reconstructed by using the evaluation neural network, and real-time update operation is performed on the current value function and real-time evaluation operation is performed on the current allowable control policy by using the evaluation neural network; In the process of iteratively executing approximation operation and update operation in the optimal collaborative control policy exploration stage, the optimal collaborative control policy exploration stage is reconstructed by using the evaluation neural network, and real-time update operation is performed on the updated current value function and real-time evaluation operation is performed on the updated allowable control policy by using the approximation Hamilton equation and the evaluation neural network to obtain the real-time collaborative control policy.

9. The method of claim 8, wherein, Real-time update operation is performed on the current value function and real-time evaluation operation is performed on the current allowable control policy by using the evaluation neural network, including: A three-layer feedforward neural network is used to construct the evaluation neural network; The current value function is represented by the weights and activation functions of the evaluation neural network, and the gradient function of the current value function is represented by the partial derivatives of the weights and activation functions of the evaluation neural network; In the real-time updating process of the current value function, a first learning error function of the evaluation neural network is determined by using a deterministic sequence, weights of the evaluation neural network, and a value function of a last iteration round; The current admissible control strategy is reconstructed based on the weights of the evaluation neural network and a partial derivative of an activation function; The weights of the evaluation neural network are reconstructed based on a first normalized gradient descent algorithm and a first minimization target function by using the activation function of the evaluation neural network and the first learning error function, to obtain a weight updating rule of the admissible control strategy learning stage; The current value function is subjected to real-time updating operation and the current admissible control strategy is subjected to real-time evaluation operation by using the weight updating rule of the admissible control strategy learning stage, to obtain an updated value function and an updated admissible control strategy.

10. The method of claim 8, wherein, The updated current value function is subjected to real-time updating operation and the updated admissible control strategy is subjected to real-time evaluation operation by using the approximate Hamilton equation and the evaluation neural network, to obtain the real-time collaborative control strategy, including: A second learning error function is reconstructed by using the approximate Hamilton equation and the partial derivative of the activation function of the evaluation neural network; The updated admissible control strategy is reconstructed by using the weights of the evaluation neural network and the partial derivative of the activation function; The weights of the evaluation neural network are reconstructed based on a second normalized gradient descent algorithm and a second minimization target function by using the partial derivative of the activation function of the evaluation neural network and the second learning error function, to obtain a weight updating rule of the optimal collaborative control strategy exploration stage; The updated value function is subjected to real-time updating operation and the updated admissible control strategy is subjected to real-time evaluation operation by using the weight updating rule of the optimal collaborative control strategy exploration stage, to obtain the real-time collaborative control strategy.

Images