Reinforcement Learning Formation Method for Multi-Agent Input Delay Systems

Abstract

The present invention relates to a reinforcement learning formation method for a multi-agent input time-delay system, including: establishing a communication topology graph among first-order multi-agent systems; establishing a tracking error between the remaining agents and the leader agent in the system; designing a formation error for each agent; introducing new system variables, and then obtaining a dynamic model equation of a new time-delay-free multi-agent system; establishing the tracking error in the system and the formation error of the time-delay-free system; considering a controller, when it makes the time-delay-free system achieve formation consensus, the original multi-agent system can also achieve formation consensus; establishing performance index functions corresponding to the two systems to make the two systems reach the optimal condition together; establishing the HJB equation; establishing an optimal controller based on reinforcement learning by using an actor-critic network structure method; and designing the update laws of the actor and critic networks. This method can reduce the calculation time while ensuring formation consensus and optimality.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-agent formation control, and particularly relates to a reinforcement learning formation method for a multi-agent input time-delay system. Background Technique

[0002] The formation control of multi-agent systems has attracted the attention of many scholars due to its wide applications. The main goal of multi-agent system formation control is to make all agents move according to a set geometric shape through a designed control protocol and complete certain tasks based on this shape. As a typical group behavior, formation is one of the common and prominent phenomena in actual scenarios and plays an important role in aspects such as underwater vehicles, unmanned surface vehicles, and unmanned aerial vehicles. As a group control method, the leader-follower method has been widely used due to its simple structure and easy implementation. Briefly speaking, the leader-follower method only needs to set one agent as the leader and specify its trajectory, and then design a controller to make all agents as followers and track the leader's trajectory at a certain distance, so as to achieve the formation movement of the multi-agent system.

[0003] The optimal problem of agents is one of the common problems in multi-agent control research. In the multi-agent formation problem, it is expected to design an optimal formation controller to optimize the system performance, that is, to make all agents maintain a certain formation and achieve the minimum performance index. Solving the Hamilton-Jacobi-Bellman (HJB) equation is a common method to obtain the optimal formation control strategy. However, it is difficult to obtain the analytical solution of the coupled HJB equation. Since the HJB equation often contains inherent non-linear terms and unknown dynamic terms, these complex terms make it difficult to solve the analytical solution of the HJB equation. Therefore, it is difficult to obtain the desired optimal control strategy by solving the HJB equation.

[0004] The reinforcement learning method is considered an effective means to solve the above difficulties. In a multi-agent system, agents interact with the environment and use reinforcement learning for evaluation and feedback, and finally can maximize the reward or achieve a specific goal. In recent years, multi-agent control based on reinforcement learning has been widely studied, especially the reinforcement learning method based on actor-critic. However, although most of the optimal control methods based on the reinforcement learning method effectively solve the problem that the HJB equation is difficult to solve, the update rules designed by the optimal control method are complex, which contain a large number of parameters that need to be updated, resulting in a long overall calculation time.

[0005] In engineering applications, the dynamic behavior of agents is complex and affected by external factors such as disturbances and time delays. During the task execution of a multi-agent system, due to factors such as the calculation and execution of control inputs and signal transmission, time delay problems often occur in the system. The time delay problem is relatively widespread and can be divided into state time delay, input time delay, communication time delay, etc. For example, an adaptive controller can be designed based on a specific Lyapunov function and neural network theory to offset the influence of state time delay in a multi-agent system; for the communication time delay problem, a distributed time-varying formation and trajectory tracking control protocol based on neighborhood interaction information can be adopted. Generally speaking, there are various types of time delays in a multi-agent system, and corresponding solutions need to be adopted according to different types to offset the influence of time delays. Summary of the Invention

[0006] The object of the present invention is to provide a reinforcement learning formation method for a multi-agent input time delay system, which can reduce the calculation time while ensuring formation consistency and optimality.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a reinforcement learning formation method for a multi-agent input time delay system, including the following steps:

[0008] Step 1: For a first-order multi-agent system with control input time delay, adopt graph theory to establish a communication topology graph between the first-order multi-agent systems; the communication topology graph is an undirected graph, that is, an agent can obtain information of adjacent agents and can send its own information to adjacent agents; in addition, specify an agent in the multi-agent system as the leader agent and set a specified moving trajectory for it, and the remaining agents follow the leader agent at a certain formation distance to achieve the formation movement goal.

[0009] Step 2: According to the communication topology graph established in Step 1, establish the tracking error between the remaining agents and the leader agent in the system, that is, the tracking error is equal to the position of the leader agent minus the position of agent i; at the same time, based on the tracking error between each agent, combined with the set distance difference between the leader and the remaining agents, design the formation error of each agent in the system.

[0010] Step 3: According to the model conversion method, introduce a new system variable and establish its connection with the position variable of the multi-agent system with control input time delay, and take the derivative of the new system variable to obtain a dynamic model equation of a new multi-agent system without time delay.

[0011] Step 4: For the multi-agent system without time delay obtained in Step 3, establish the tracking error in the system according to the communication topology graph established in Step 1, and adopt the distance difference set in Step 2 to establish the formation error of the system without time delay.

[0012] Step 5: For the multi-agent system with control input delay and the delay-free multi-agent system established in Step 3, consider a controller. Prove that the controller can make the delay-free multi-agent system established in Step 3 achieve formation consensus, and when the controller makes the delay-free system achieve formation consensus, the original multi-agent system with control input delay can also achieve formation consensus;

[0013] Step 6: For the system with control input delay and the delay-free multi-agent system established in Step 3, consider the optimal control theory, establish the performance index functions corresponding to the two systems, and obtain the conditions for the two systems to jointly reach the optimum through proof;

[0014] Step 7: According to the performance index function of the delay-free multi-agent system in Step 3 and the utility function in the performance index function, establish its corresponding Hamilton-Jacobi-Bellman equation, and take the partial derivative of this equation with respect to the control input to obtain the expression form of the control input with respect to the gradient of the performance index function;

[0015] Step 8: Separate the parameters of the gradient of the performance index function obtained in Step 7, obtain the separated form of the control input, and establish an optimal controller based on reinforcement learning using the actor-critic network structure method;

[0016] Step 9: Improve the method of obtaining the network update law by taking the derivative of the Bellman residual in the traditional method. Consider a simple positive function to be equivalent to the method of obtaining the update law with the Bellman residual, and design the update laws of the actor and critic networks according to the derivative of the simple positive function.

[0017] Furthermore, the multi-agent system model with input delay is as follows:

[0018]

[0019] where p i (t) represents the position of the i-th agent in the system; u i (t) represents the control of the i-th agent in the system; ρ i (·) represents an unknown nonlinear function, and it is assumed to be Lipschitz continuous; and both represent known constant matrices with appropriate dimensions;

[0020] The leader trajectory is set as follows:

[0021]

[0022] Among them, p0 represents the desired movement trajectory of the leader agent during formation movement; v0 represents the speed of the desired leader during formation movement.

[0023] Based on the system model and the leader's trajectory, define the formation error z of each follower agent pi as follows:

[0024]

[0025] Among them, ζ i is the position difference between the i-th follower and the leader, which is mainly used to maintain the formation during system movement; Λ i represents the adjacency matrix of the i-th follower; a ij is the value of the adjacency matrix in the i-th row and j-th column, which is used to describe the communication connection weight between the i-th follower agent and its neighbor agents; b i represents the communication connection weight between the leader agent and the i-th follower.

[0026] Furthermore, in step three, combined with the defined system model, introduce a new state model, and its related expressions are as follows:

[0027]

[0028] Derive the introduced new state model to obtain a new system model expression without control input delay as follows:

[0029]

[0030] Among them, is an unknown bounded nonlinear function.

[0031] Furthermore, in step four, combined with the defined new state model, define its own formation error z yi as follows:

[0032]

[0033] Furthermore, in step five, combined with the formation error of the defined new state model, consider the controller u i (t) = γ z z yi (t), and prove that the controller can make the original system and the converted model without input delay achieve formation consensus together;

[0034] The defined formation error z yi can be re-expressed as follows:

[0035]

[0036] Among them,

[0037] According to the rewritten formation error z yi and the defined controller u i (t), the following inequality is obtained:

[0038]

[0039] Among them, z yit (s):=z yit (t + s), -τ i ≤ s ≤ 0;

[0040] According to the above inequality, when the controller u i (t) makes the formation error z yi (t) of the delay-free system approach 0, it can also make the formation error z pi (t) of the original multi-agent system approach 0.

[0041] Furthermore, in step six, the performance index function of the original system with input delay is defined as follows:

[0042]

[0043] Among them, r pi represents the utility function of the system with input delay; Γ i represents a symmetric positive semi-definite matrix; represents the control vector at time [kT s , (k + 1)T s ;

[0044] In addition, the performance index function of the transformed delay-free system is defined as follows:

[0045]

[0046] Among them, r yi represents the utility function of the delay-free system; and represent matrices with appropriate dimensions;

[0047] When the appropriate conditions are met, the optimal performance can be achieved simultaneously between the two systems.

[0048] Furthermore, in step seven, the following equation can be solved:

[0049]

[0050] Taking the partial derivative of the above equation with respect to the control input, the relevant expression form of the controller is obtained:

[0051]

[0052] Among them,

[0053] Furthermore, for the obtained and expressions are separated to obtain the following expression forms:

[0054]

[0055]

[0056] Among them,

[0057] h in the above formula i and are actually unknown terms. An approximation is introduced based on the theory of fuzzy logic systems combined with the actor-critic reinforcement learning method to obtain the approximate gradient term and the actor-based control strategy as follows:

[0058]

[0059]

[0060] Among them, and represent the estimated matrix of the ideal fuzzy logic weight matrix; S hi and S i represent the fuzzy basis function vectors;

[0061] Furthermore, a positive function is designed, and its expression form is as follows:

[0062]

[0063] For the obtained and u i in the expression form of the actor network and the critic network design the update law as follows:

[0064]

[0065]

[0066] Derive the designed positive function to obtain the following expression:

[0067]

[0068] The designed update law of the actor-critic network can make the derivative of the defined positive function ψ i (t) approach 0. And the fact that the derivative approaches 0 is equivalent to the approximate Hamilton-Jacobi-Bellman equation approaching 0. The designed update law obtained in this way is not only simpler but also satisfies the original conditions.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] 1. For the multi-agent system with control input delay, the present invention adopts the method of model transformation, introduces new state variables to design a new time-delay-free system model, and considers control strategies to enable the two system models to achieve formation consensus and optimality.

[0071] 2. Through the fuzzy logic system, the network update law of the control strategy is redesigned. Compared with the network update law of the traditional method, the update law designed in the present invention contains fewer parameters, can reduce the overall calculation time of the system, and ensure that the formation error does not increase significantly. Description of the Drawings

[0072] Figure 1 is the block diagram of the traditional reinforcement learning structure method;

[0073] Figure 2 is the block diagram of the simplified reinforcement learning update law method of the embodiment of the present invention;

[0074] Figure 3 is the communication topology structure diagram of the multi-agent system of the embodiment of the present invention;

[0075] Figure 4 is the formation movement trajectory diagram of the multi-agent system with input delay of the embodiment of the present invention

[0076] Figure 5 is the identification error diagram of the identifier of the embodiment of the present invention;

[0077] Figure 6 is the parameter weight change diagram of the critic network of the embodiment of the present invention;

[0078] Figure 7 is the parameter weight change diagram of the actor network of the embodiment of the present invention;

[0079] Figure 8 is the formation movement trajectory diagram of the time-delay-free system by two methods of the embodiment of the present invention;

[0080] Figure 9 is the formation error diagram of two methods of the embodiment of the present invention;

[0081] Figure 10 is the calculation time comparison diagram of two methods of the embodiment of the present invention.

[0082] Figure 11 It is the flowchart of the method implementation in the embodiment of the present invention. Specific implementation manners

[0083] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0084] It should be noted that the following detailed descriptions are all exemplary and are intended to provide further descriptions of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present application belongs.

[0085] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary implementation manners according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "include" and / or "comprise" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or their combinations.

[0086] The present invention aims at a multi-agent system with input time delay, uses a model transformation method to transform the input time-delay model into a time-delay-free system, redesigns the update law based on reinforcement learning to achieve optimal formation control, and tests this method through simulation. As Figure 11 shown, this method includes the following steps:

[0087] Step 1: Consider the multi-agent system model with input time delay as follows:

[0088]

[0089] wherein, p i (t) represents the position of the i-th agent in the system; u i (t) represents the control of the i-th agent in the system; ρ i (·) represents an unknown non-linear function and is assumed to be Lipschitz continuous; and represent known constant matrices with appropriate dimensions.

[0090] Consider the trajectory of the leader agent as follows:

[0091]

[0092] wherein, p0 represents the desired moving trajectory of the leader agent in formation movement; v0 represents the desired speed of the leader in formation movement.

[0093] Step 2: Consider that the communication topology graph of the multi-agent system is an undirected graph, where Λ i represents the adjacency matrix of the i-th follower; a ij is the value of the adjacency matrix in the i-th row and j-th column, which is used to describe the communication connection weight between the i-th follower agent and its neighbor agents; b i represents the communication connection weight between the leader agent and the i-th follower.

[0094] Based on the communication topology graph and the system model and leader trajectory defined in Step 1, set the formation error as follows:

[0095]

[0096] where, ζ i is the position difference between the i-th follower and the leader, and its main role is to maintain the formation of the system during movement.

[0097] Step 3: According to the model transformation method, introduce a new system variable, and its related expressions are as follows:

[0098]

[0099] Taking the derivative of the introduced new state, a new system model expression without control input delay can be obtained as follows:

[0100]

[0101] where, is an unknown bounded nonlinear function.

[0102] Step 4: For the multi-agent system model without delay obtained in Step 3, establish the formation error of the system without delay as follows:

[0103]

[0104] Step 5: For the formation error z yi designed in Step 4, it can be re-expressed in the following form:

[0105]

[0106] where,

[0107] According to the rewritten formation error z yi , and the defined controller u i (t), the following inequality can be obtained:

[0108]

[0109] where, zyit (s) := z yit (t + s), -τ i ≤ s ≤ 0.

[0110] According to the above inequality, when the controller u i (t) makes the formation error z yi (t) of the time-delay-free system approach 0, it can also make the formation error z pi (t) of the original multi-agent system approach 0.

[0111] Step Six: For the system with control input delay, define the performance index function of the original system with input delay as follows:

[0112]

[0113] where r pi represents the utility function of the system with input delay; Γ i represents a symmetric positive semi-definite matrix; represents the control vector at time [kT s , (k + 1)T s .

[0114] In addition, the performance index function of the transformed time-delay-free system can be defined as follows:

[0115]

[0116] where r yi represents the utility function of the time-delay-free system; and represent matrices with appropriate dimensions.

[0117] When appropriate conditions are met, the optimal performance can be achieved simultaneously between the two systems.

[0118] Step Seven: Establish the corresponding Hamilton-Jacobi-Bellman equation for the time-delay-free multi-agent system as follows:

[0119]

[0120] Taking the partial derivative of the above equation with respect to the control input, the relevant expression form of the controller is obtained:

[0121]

[0122] where

[0123] Step Eight: Separate the parameters of the gradient of the performance index function obtained in Step Seven and obtain the separated form of the control input as follows:

[0124]

[0125]

[0126] Among them,

[0127] h in the above formula i and are actually unknown terms, and it is necessary to introduce an approximation based on the theory of fuzzy logic systems combined with the actor-critic reinforcement learning method to obtain the approximate gradient term and the actor-based control strategy as follows:

[0128]

[0129]

[0130] Among them, and represent the estimated matrix of the ideal fuzzy logic weight matrix; S hi and S i represent the fuzzy basis function vectors.

[0131] Step Nine: Consider the equivalence of a simple positive function and the Bellman residual to obtain the update law method, and its form is as follows:

[0132]

[0133] For the expression of and u i obtained for claim 9 in the expression form of the actor network and the critic network design their update law expressions as follows:

[0134]

[0135]

[0136] Take the derivative of the designed positive function to obtain the following expression:

[0137]

[0138] It can be seen that the designed update laws of the actor and critic networks can make the derivative of the defined positive function ψ i (t) approach 0, and the derivative approaching 0 is equivalent to the approximate Hamilton-Jacobi-Bellman equation approaching 0. The obtained update law design is not only simpler but also satisfies the original conditions.

[0139] To prove that the reinforcement learning formation method for a multi-agent input time-delay system designed in this example can achieve formation control behavior for a multi-agent system with control input time-delay, two simulation experiments are given here. In Experiment 1, the expression form of the multi-agent system with control input time-delay is as follows:

[0140]

[0141] Among them, μi = -4.2, 5.4, -2.8, 7.2; And the initial positions of the four follower agents are set as x i (0) = [5, 5] T , [-5, 5] T , [5, -5] T , [-5, -5] T .

[0142] The model of the leader agent is designed as follows:

[0143]

[0144] The initial position of the leader agent among them is [0, 0] T .

[0145] For the formation control of the multi-agent system, the communication topology needs to be considered. The matrix Λ i expresses the connection weights between the leader and the followers, as follows:

[0146]

[0147] The matrix B is the connection matrix expressing the followers among the agents, as follows:

[0148] B = diag{1, 0, 0, 0} (24)

[0149] In Experiment 2, a multi-agent system without time-delay is considered to prove the advantage of the present invention in terms of computing time. Its system model is expressed as follows:

[0150]

[0151] Among them, ω i = -0.8, 0.4, -0.5, 0.2; β i1 = 0.4, 0.7, 0.2, 0.8; β i2 = 0.2, 0.9, 0.5, 0.7; And the initial positions of the four follower agents are set as x i (0) = [4, 4] T , [-4, 4]T , [4, -4] T , [-4, -4] T .

[0152] The model of the designed leader agent is shown as follows:

[0153]

[0154] The initial position of the leader agent among them is [0, 0] T .

[0155] Figure 1 This is the block diagram of the traditional reinforcement learning structure method, Figure 2 This is the block diagram of the simplified reinforcement learning update law method in this embodiment. The difference between the two lies in the obtaining of the update law. Figure 1 The process of obtaining the update law of the traditional method shown is directly obtained from the approximate performance index function without simplification, while Figure 2 the simplified method of the present invention shown solves the reinforcement learning update law from the positive function of the equivalent performance index function. Figure 3 Shows the communication topology diagram of the multi-agent system in this embodiment. The communication between agents in this embodiment is carried out according to this diagram. Figure 4 This is the formation movement trajectory diagram of the multi-agent system with input time delay in this embodiment. It can be seen that the four follower agents can follow the leader's trajectory well for movement. Figure 5 This is the identification error diagram of the identifier in the embodiment of the present invention. It can be seen from the figure that the identifier can estimate the information change of the system model well. Figure 6 and Figure 7 are respectively the change diagrams of the parameter weights of the critic and actor networks in this embodiment. Figure 8 Represents the formation movement trajectory diagram of the two methods for the time-delay-free system in this embodiment. It can be seen that both the method proposed by the present invention and the traditional method with the unsimplified update law can enable the followers of the time-delay-free multi-agent system to follow the leader's trajectory for movement. Figure 9 This is the formation error diagram of the two methods in this embodiment. It can be seen that although the update law of the present invention is simplified in design, for the formation errors of some follower agents, the formation errors obtained by the method proposed by the present invention can even be smaller than those obtained by the method with the unsimplified traditional update law. Figure 10 This is the comparison diagram of the calculation time of the two methods in this embodiment. It can be directly seen from the figure that compared with the method with the unsimplified traditional update law, the calculation time of the method proposed by the present invention is significantly reduced.

[0156] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention in any other form. Any person skilled in the relevant art may use the technical content disclosed above to make changes or modifications into equivalent embodiments with equivalent changes. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the technical content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A reinforcement learning formation method for multi-agent input time-delay systems, characterized in that It includes the following steps: Step 1: For a first-order multi-agent system with control input delay, using graph theory, establish a communication topology graph among the first-order multi-agent systems; the communication topology graph is an undirected graph, that is, an agent can obtain information of adjacent agents and can send its own information to adjacent agents; In addition, specify an agent in the multi-agent system as the leader agent, and set a prescribed moving trajectory for it, and the remaining agents follow the leader agent at a certain formation distance to achieve the formation movement goal; Step 2: According to the communication topology graph established in Step 1, establish the tracking error between the remaining agents and the leader agent in the system, that is, the tracking error is equal to the position of the leader agent minus the position of agent i; at the same time, from the tracking error between each agent, combined with the set distance difference between the leader and the remaining agents, design the formation error of each agent in the system; Step 3: According to the model transformation method, introduce a new system variable, and establish its connection with the position variable of the multi-agent system with control input delay, and take the derivative of the new system variable to obtain a dynamic model equation of a new multi-agent system without delay; Step 4: For the multi-agent system without delay obtained in Step 3, according to the communication topology graph established in Step 1, establish the tracking error in the system, and use the distance difference set in Step 2 to establish the formation error of the system without delay; Step 5: For the multi-agent system with control input delay and the multi-agent system without delay established in Step 3, consider a controller, and prove that the controller can make the multi-agent system without delay established in Step 3 achieve formation consensus, and when the controller makes the system without delay achieve formation consensus, the original multi-agent system with control input delay can also achieve formation consensus; Step 6: For the system with control input delay and the multi-agent system without delay established in Step 3, consider the optimal control theory, establish the performance index functions corresponding to the two systems, and obtain the conditions under which the two systems can jointly reach the optimum through proof; Step 7: According to the performance index function of the multi-agent system without delay in Step 3 and the utility function in the performance index function, establish its corresponding Hamilton-Jacobi-Bellman equation, and take the partial derivative of this equation with respect to the control input to obtain the expression form of the control input with respect to the gradient phase of the performance index function; Step 8: Separate the parameters of the gradient phase of the performance index function obtained in Step 7, and obtain the separated form of the control input, and establish an optimal controller based on reinforcement learning using the actor-critic network structure method; Step 9: Improve the method of obtaining the network update law by taking the derivative of the Bellman residual in the traditional method, consider a simple positive function to be equivalent to the method of obtaining the update law with the Bellman residual, and design the update laws of the actor and critic networks according to the derivative of the simple positive function.

2. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 1, characterized in that The model of the multi-agent system with input delay is as follows: where p i (t) represents the position of the i-th agent in the system; u i (t) represents the control of the i-th agent in the system; ρ i (·) represents an unknown nonlinear function, and it is assumed to be Lipschitz continuous; and both represent known constant matrices with specified dimensions; The leader trajectory is set as follows: Among them, p0 represents the desired movement trajectory of the leader agent in formation movement; v0 represents the speed of the desired leader in formation movement. Define the formation error \(z\) of each follower agent by the system model and the leader's trajectory pi as follows: Among them, ζ i is the position difference between the i-th follower and the leader, which is mainly used to maintain the formation when the system moves; Λ i represents the adjacency matrix of the i-th follower; a ij is the value of the adjacency matrix in the i-th row and j-th column, which is used to describe the communication connection weight between the i-th follower agent and its neighbor agents; b i represents the communication connection weight between the leader agent and the i-th follower.

3. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 2, wherein In step three, in combination with the defined system model, a new state model is introduced, and its related expressions are as follows: Derivation is performed on the introduced new state model to obtain a new system model expression without control input delay as follows: wherein, is an unknown bounded non-linear function.

4. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 3, characterized in that In step 4, in combination with the defined new state model, define its own formation error z yi as follows:

5. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 4, characterized in that In step five, considering the formation error of the newly defined state model, the controller u i (t) = γ z z yi (t) is considered, and it is proved that the controller can make the original system and the transformed model without input time-delay reach formation consensus together; The defined formation error z yi is re-expressed as follows: Among them, Based on the rewritten formation error z yi and the defined controller u i (t), the following inequality is obtained: where z yit (s):=z yit (t + s), -τ i ≤ s ≤ 0; According to the above inequality, when the controller u i (t) makes the formation error z yi (t) of the time-delay-free system approach 0, it can also make the formation error z pi (t) of the original multi-agent system approach 0.

6. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 5, characterized in that In step six, the performance index function of the original system with input delay is defined as follows: where, r pi represents the utility function of the input time-delay system; Γ i represents a symmetric positive semi-definite matrix; represents the control vector at time [kT s , (k + 1)T s ; In addition, the performance index function of the converted system without delay is defined as follows: where r yi represents the utility function of the time-delay-free system; and represent matrices with specified dimensions; When the set conditions are met, the optimal performance can be achieved simultaneously between the two systems.

7. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 6, wherein In step seven, the following equation is solved: Taking the partial derivative of the above equation with respect to the control input, the related expression form of the controller is obtained: Among them, 8. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 7, characterized in that For the obtained and Separate the expressions to obtain the following expression forms: Among them, h in the above formula i and are actually unknown terms. An approximation is introduced based on the theory of fuzzy logic systems combined with the actor-critic reinforcement learning method to obtain the approximate gradient term and the actor-based control strategy as follows: Among them, and represent the estimated matrix of the ideal fuzzy logic weight matrix; S hi and S i represent the fuzzy basis function vectors.

9. The reinforcement learning formation method for a multi-agent input time-delay system according to claim 8, characterized in that Design a positive function, and its expression form is as follows: For the obtained one regarding and u i the actor network in the expression form and the critic network design its update law expression as follows: Taking the derivative of the designed positive function, the following expression is obtained: The designed update law of the actor-critic network enables the derivative of the defined positive function ψ i (t) to approach 0, and the derivative approaching 0 is equivalent to the approximate Hamilton-Jacobi-Bellman equation approaching 0.

Images