A Multi-Agent Consensus Reinforcement Learning Method and System Based on Improved Q-Function

Abstract

This invention relates to a multi-agent consensus reinforcement learning method and system based on an improved Q-function. First, a dynamic model of a linear discrete-time heterogeneous multi-agent system is established. Second, a reliability factor ρ is introduced to compromise between two learning modes: non-policy Q-learning and policy Q-learning. A distributed control protocol based on the improved Q-function is designed. Finally, a Critic neural network is used to learn the optimal control protocol from data generated by the system's dynamic model. Consistency control of the heterogeneous multi-agent system is then performed according to this optimal control protocol. This invention proposes an improved Q-function-based heterogeneous multi-agent consensus reinforcement learning control method, enhancing the self-learning capability of reinforcement learning, making agents more sensitive to environmental changes, accelerating learning speed, improving learning reliability and efficiency, and simultaneously achieving consensus in the heterogeneous multi-agent system in an optimal manner.

Description

Technical Field

[0001] This invention belongs to the field of multi-agent consensus technology, specifically relating to a multi-agent consensus reinforcement learning method and system based on an improved Q-function. Background Technology

[0002] As a type of artificial intelligence, reinforcement learning has demonstrated great power and potential in various practical applications such as intelligent robots, drones, and power grids, because it can make autonomous decisions through interaction with unknown environments in order to maximize cumulative rewards.

[0003] In practical applications, both model-based and model-free reinforcement learning methods are forms of self-learning. Model-based reinforcement learning relies on model dynamics. Since the dynamic models of most systems cannot be determined perfectly accurately, systems are often nonlinear and contain disturbances, agents are coupled with each other, and communication delays between agents make consistent control of multi-agent systems more difficult. Model-dependent reinforcement learning methods lack rapid perception of environmental changes and often exhibit model errors. Model-free reinforcement learning requires abundant data with sufficient exploratory value, but the sparsity of data in practical applications poses a significant challenge to model-free reinforcement learning techniques.

[0004] Reinforcement learning-based heterogeneous multi-agent consensus control learning methods include value iteration, policy iteration, policy, and non-policy self-learning methods. The optimal consensus distributed control design of heterogeneous multi-agent systems relies solely on policy reinforcement learning methods with model dynamics or non-policy reinforcement learning methods with data information. However, accurate models of real-world systems are difficult to obtain, or the known model dynamics contain model errors. On the other hand, the sparsity of data in practical applications makes reinforcement learning techniques that do not rely on models have low learning efficiency and poor self-learning performance. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a multi-agent consensus reinforcement learning method based on an improved Q-function, comprising the following steps:

[0006] Step 1: Establish a dynamic model of the linear discrete-time heterogeneous multi-agent system;

[0007] Step 2: Define the leader-follower consistency error equation for the heterogeneous multi-agent system, and introduce a reliability factor ρ to compromise between two learning modes: non-policy Q-learning and policy Q-learning. Design a distributed control protocol based on the improved Q function.

[0008] Step 3: Use the Critic neural network to learn the optimal control protocol from the data generated by the system dynamic model, and perform consistent control on the heterogeneous multi-agent system according to the optimal control protocol.

[0009] Step 1 is as follows:

[0010] Consider a linear discrete-time heterogeneous multi-agent system with N followers and one leader. The dynamics model of follower i is as follows:

[0011] x i (k+1)=A i x i (k)+B i u i (k) (1)

[0012] Where, x i (k) and u i (k) represent the state and control input of the i-th follower at time k (k = 1, 2, ...), respectively. i and B i Let these represent the system matrix and control matrix of the i-th follower, respectively;

[0013] The dynamic model of the leader (i.e., the reference trajectory in a heterogeneous multi-agent system) is as follows:

[0014] x0(k+1)=A0x0(k) (2)

[0015] Where x0(k) represents the state of the leader at time k (k = 1, 2, ...), and A0 represents the system matrix of the leader.

[0016] Step 2 defines the leader-follower consistency error equation for the heterogeneous multi-agent system as follows:

[0017] Define the local neighbor synchronization error of the i-th follower as:

[0018]

[0019] Where, δ i (k) represents the local neighbor synchronization error of the i-th follower, a ib x represents the weight between follower i and follower b. b (k) represents the state of the b-th follower at time k (k = 1, 2, ...), g i (g i ≥0) represents the fixed gain between the i-th follower and the leader, N i This represents the number of neighboring nodes of follower i in the system topology graph. This represents the sum of all topological relationships from follower b to follower i in the system topology graph;

[0020] The global synchronization error of a heterogeneous multi-agent system is defined as:

[0021] ξ(k)=x(k)- x 0(k) (4)

[0022] Where ξ(k) represents the global synchronization error of the heterogeneous multi-agent system, and x(k) represents the global state of the N followers at time k. T represents the vector transpose. x 0(k) represents the global state of the leader corresponding to each follower at time k.

[0023] The global synchronization error ξ(k) and the global neighbor synchronization error δ(k) have the following relationship:

[0024]

[0025] Where L represents the Laplace matrix of the system topology graph, I n Let g represent an n-dimensional identity matrix, g = diag{g i}∈R N×N This represents a diagonal matrix with a fixed gain. It represents the Kronecker product.

[0026] For any follower i, its performance metric is defined as:

[0027]

[0028] in,

[0029]

[0030] Among them, J i U represents the performance metric of follower i. ik U represents the control protocol of the i-th follower at time k. -ik ={u b |b∈N i} represents the control protocol of the neighbor of the i-th follower at time k, u b This represents the control protocol of the b-th follower. U represents the sum of all moments corresponding to follower i. i Let δ represent the effect function of the i-th follower. ik Let Q represent the local neighbor synchronization error of the i-th follower at time k. ii ≥0,R ii >0,R ib >0 indicates a symmetric matrix with appropriate dimensions, u bkThis represents the control protocol of the b-th follower at time k.

[0031] Step 2 introduces a trade-off between two learning modes: non-policy Q-learning and policy Q-learning, based on a reliability factor ρ. The resulting distributed control protocol, designed using the improved Q-function, is as follows:

[0032] Defining a reliability factor ρ as a trade-off between non-policy Q-learning and policy Q-learning modes, the improved Q-function is obtained as follows:

[0033]

[0034] The distributed control protocol based on the improved Q-function is as follows:

[0035]

[0036] Where, ε ik It is caused by δ ik and δ -ik The global synchronization error of the i-th follower at time k. This represents the data generated by the target policy of the i-th follower at time k. Let α represent the target policy of the i-th follower at time k, α represent the learning rate, and μ represent the control policy. express hour right The gradient is:

[0037]

[0038] Step 3 includes the following steps:

[0039] Step 3.1: Provide a behavior control strategy for follower i, and collect the state of the i-th follower at time k+1 and the local neighbor synchronization error;

[0040] Step 3.2: Initialize the Critic network weights of follower i to enable the control protocol of the heterogeneous multi-agent system;

[0041] Step 3.3: Obtain the neural network approximation error of follower i based on the collected state of the i-th follower at time k+1 and the local neighbor synchronization error;

[0042] Step 3.4: Train the weights of the Critic network for follower i based on the approximation error of the neural network for follower i;

[0043] Step 3.5: Determine whether the Critic network weights of follower i after training have reached the threshold. If yes, proceed to the next step; otherwise, return to step 3.3.

[0044] Step 3.6: Use the consensus control protocol corresponding to the Critic network weights of the trained follower i as the optimal control protocol.

[0045] Step 3.4 involves training the Critic network weights for follower i, using the following formula:

[0046]

[0047] in, This represents the updated Critic network weights. μ represents the weights of the Critic neural network. ic This represents the learning rate of the neural network. Let represent the improved Q-function obtained by the i-th follower through a trade-off between the two reinforcement learning modes using a reliability factor ρ. This represents the difference between the activation function at the previous time step and the next time step. The column vector formed by these.

[0048] The system based on the improved Q-function multi-agent consensus reinforcement learning method uses the improved Q-function multi-agent consensus reinforcement learning method as described in any one of claims 1-6 to perform optimal consensus control of heterogeneous multi-agent systems.

[0049] Furthermore, the system also includes a storage device that stores instructions and data for implementing the multi-agent consensus reinforcement learning method based on the improved Q-function as described in any one of claims 1-6.

[0050] Advantages of this invention:

[0051] This invention relates to a multi-agent consensus reinforcement learning method and system based on an improved Q-function. By compromising between non-policy reinforcement learning and policy reinforcement learning modes through a reliability factor ρ, it proposes an improved Q-function heterogeneous multi-agent consensus reinforcement learning control method. This method enhances the self-learning capability of reinforcement learning, makes agents more sensitive to environmental changes, learns faster, improves learning reliability, and increases learning efficiency. At the same time, it enables the heterogeneous multi-agent system to achieve consensus in an optimal manner. Attached Figure Description

[0052] Figure 1 This is a flowchart of a multi-agent consensus reinforcement learning method based on an improved Q-function according to an embodiment of the present invention.

[0053] Figure 2 This is a design framework diagram of a multi-agent consensus reinforcement learning method based on an improved Q-function according to an embodiment of the present invention;

[0054] Figure 3This is a communication topology diagram of a heterogeneous multi-agent system with one leader and four followers, according to an embodiment of the present invention.

[0055] Figure 4 This is a weight graph of the follower Critic neural network for implementing Algorithm 2 in one embodiment of the present invention;

[0056] Figure 5 This is a leader and follower state trajectory diagram of Algorithm 2 in one embodiment of the present invention;

[0057] Figure 6 This is a follower local neighbor error graph of Algorithm 2 in one embodiment of the present invention;

[0058] Figure 7 This is a state trajectory diagram of the follower and leader in an embodiment of the non-strategy Q-learning algorithm of the present invention;

[0059] Figure 8 This is a follower local neighbor error map for implementing a non-strategy Q-learning algorithm according to an embodiment of the present invention. Detailed Implementation

[0060] An embodiment of the present invention will be further described below with reference to the accompanying drawings.

[0061] In this embodiment of the invention, a multi-agent consensus reinforcement learning method based on an improved Q-function is described in the flowchart below. Figure 1 As shown, it includes the following steps:

[0062] Step 1: Establish a dynamic model of a linear discrete-time heterogeneous multi-agent system with N followers and one leader:

[0063] The dynamic model of follower i is:

[0064] x i (k+1)=A i x i (k)+B i u i (k) (1)

[0065] Where, x i (k) and u i (k) represent the state and control input of the i-th follower at time k (k = 1, 2, ...), respectively. i and B i Let these represent the system matrix and control matrix of the i-th follower, respectively;

[0066] The dynamic model of the leader (i.e., the reference trajectory in a heterogeneous multi-agent system) is as follows:

[0067] x0(k+1)=A0x0(k) (2)

[0068] Where x0(k) represents the state of the leader at time k (k = 1, 2, ...), and A0 represents the system matrix of the leader;

[0069] In this embodiment of the invention, each of the N followers can obtain its own state information at each time point and the state information of its connected neighboring smart agent j at each time point.

[0070] Step 2: Define the leader-follower consistency error equation for the heterogeneous multi-agent system, and introduce a reliability factor ρ to compromise between two learning modes: non-policy Q-learning and policy Q-learning. Design a distributed control protocol based on the improved Q function.

[0071] Based on the heterogeneous multi-agent dynamics model, the local neighbor synchronization error of the i-th follower is defined as:

[0072]

[0073] Where, δ i (k) represents the local neighbor synchronization error of the i-th follower, a ib x represents the weight between follower i and follower b. b (k) represents the state of the b-th follower at time k (k = 1, 2, ...), g i (g i ≥0) represents the fixed gain between the i-th follower and the leader, N i This represents the number of neighboring nodes of follower i in the system topology graph. This represents the sum of all topological relationships from follower b to follower i in the system topology graph;

[0074] Set δ -ik ={δ j |j∈N i} represents the local error of all neighboring followers of the i-th follower at time k, given by equation (3) and δ -ik The global synchronization error of a heterogeneous multi-agent system can be expressed as:

[0075] ξ(k)=x(k)- x 0(k) (4)

[0076] Where ξ(k) represents the global synchronization error of the heterogeneous multi-agent system, and x(k) represents the global state of the N followers at time k. T represents the vector transpose. x 0(k) represents the global state of the leader corresponding to each follower at time k.

[0077] Based on formulas (3) and (4), the global synchronization error ξ(k) and the global neighbor synchronization error δ(k) of the heterogeneous multi-agent system have the following relationship:

[0078]

[0079] Where L represents the Laplace matrix of the system topology graph, I n Let g represent an n-dimensional identity matrix, g = diag{g i}∈R N×N This represents a diagonal matrix with a fixed gain. It represents the Kronecker product.

[0080] For any follower i, its performance metric is defined as:

[0081]

[0082] in,

[0083]

[0084] Among them, J i U represents the performance metric of follower i. ik U represents the control protocol of the i-th follower at time k. -ik ={u b |b∈N i} represents the control protocol of the neighbor of the i-th follower at time k, u b This represents the control protocol of the b-th follower. U represents the sum of all moments corresponding to follower i. i Let δ represent the effect function of the i-th follower. ik Let Q represent the local neighbor synchronization error of the i-th follower at time k. ii ≥0,R ii >0,R ib >0 indicates a symmetric matrix with appropriate dimensions, u bk This represents the control protocol of the b-th follower at time k.

[0085] Based on existing reinforcement learning results, the value function and Q function based on the Bellman equation are defined as follows:

[0086]

[0087] and

[0088]

[0089] Where, ε ik It is caused by δ ikand δ -ik The global synchronization error of the i-th follower at time k, δ -ik Let δ represent the global synchronization error of the neighboring follower of the i-th follower at time k. is Let u represent the local neighbor error of the i-th follower from the current time to infinity. is Let u represent the control strategy of the i-th follower from the current time to infinity. -is This represents the control policy of all neighboring followers of the i-th follower from the current time to infinity. This represents the optimal control protocol for the i-th follower at time k, where * denotes the optimal value;

[0090] According to the necessary condition for optimality, we have:

[0091]

[0092] Where arg represents Q i u when taking the minimum value ik The value of Q i Describe the Q-function of the i-th follower;

[0093] Further define the Q-function iteration process based on non-policy reinforcement learning and policy reinforcement learning methods:

[0094] The iterative equation for the Q-function based on non-policy reinforcement learning methods is:

[0095]

[0096] The iterative equation for the Q-function based on policy reinforcement learning is:

[0097]

[0098] Where j represents the iteration index, This represents the target policy of the neighboring follower of the i-th follower at time k. This represents the target policy of the i-th follower at time k. This represents the data generated by the target policy of the i-th follower at time k.

[0099] Define a reliability factor ρ to compromise between the non-policy reinforcement learning Q-function and the policy reinforcement learning Q-function, resulting in the improved Q-function:

[0100]

[0101] The distributed control protocol based on the improved Q-function is as follows:

[0102]

[0103] Where α represents the learning rate and μ represents the control policy. express hour right The gradient is:

[0104]

[0105] Based on this, the optimal control policy is learned by making full use of the detectable data and using the proposed reinforcement learning method with an improved Q function.

[0106] Step 3: Using a Critic neural network, the optimal control protocol is learned from the data generated by the i-th follower (i.e., measurable data). Consistent control of the heterogeneous multi-agent system is then performed according to the optimal control protocol, including the following steps:

[0107] Step 3.1: Provide a behavior control strategy for follower i, and collect the state of the i-th follower at time k+1 and the local neighbor synchronization error;

[0108] Step 3.2: Initialize the Critic network weights of follower i to enable the control protocol of the heterogeneous multi-agent system;

[0109] Step 3.3: Obtain the neural network approximation error of follower i based on the collected state of the i-th follower at time k+1 and the local neighbor synchronization error;

[0110] Step 3.4: Train the weights of the Critic network for follower i based on the approximation error of the neural network for follower i;

[0111] Step 3.5: Determine whether the Critic network weights of follower i after training have reached the threshold. If yes, proceed to the next step; otherwise, return to step 3.3.

[0112] Step 3.6: Use the consensus control protocol corresponding to the Critic network weights of the trained follower i as the optimal control protocol.

[0113] In this embodiment of the invention, two algorithms are used to learn the optimal control protocol method for follower i:

[0114] Algorithm 1 is a self-learning algorithm with an improved Q-function:

[0115] Step 3.1: Define the initially permissible control protocol and permissible behavioral control strategies with detection noise u ik It acts on the system to generate and save data;

[0116] Step 3.2: Strategy Evaluation: For equations (11) and (12), calculate... and Then, the improved Q function is calculated according to equation (13).

[0117] Step 3.3: Policy Update: Update the control protocol according to formula (14);

[0118] Step 3.4: When (ε is a very small integer, i.e., a positive error limit, on the order of 10) -4 above)

[0119] Stop if the time is right; otherwise, return to step 3.2.

[0120] In this embodiment of the invention, for the above algorithm 1, only the improved Q function is approximated using a Critic neural network, and the weights of the Critic neural network are trained using gradient descent to evaluate the optimal consensus control protocol, as detailed below:

[0121] Constructing a Critic neural network for follower i:

[0122]

[0123] in, express The estimated value, This represents the improved Q-function of the i-th follower. The weights of the Critic neural network are:

[0124]

[0125] u using equation (16) ik Taking the partial derivative, we obtain an approximation of the control protocol:

[0126]

[0127] Right now:

[0128]

[0129] Further, there are:

[0130]

[0131] in, Represented by a symmetric matrix The reconstructed column vectors, i.e., the estimated weights of the Critic neural network, express The corresponding activation function, Zik This represents the vector consisting of the states of the i-th follower at time k.

[0132] set up The objective Q-function of the Critic neural network is:

[0133]

[0134] The neural network approximation error of follower i is calculated as follows:

[0135]

[0136] Its square approximation error for:

[0137]

[0138] in, Let represent the estimated value of the control policy of the i-th follower. This represents the estimated value of the neighbor follower control policy of the i-th follower. Let represent the estimated control policy of the neighboring followers of the b-th follower. This represents the difference between the activation function at the previous time step and the next time step. The column vector formed by U i (δ ik ,u ik ,u -ik )and Let these represent the non-strategy Q-effect function and the strategy Q-effect function, respectively. The improved Q-function for i followers at time k;

[0139] The gradient of the squared approximation error of the Critic neural network is calculated as follows:

[0140]

[0141] Finally, the weight update formula for the Critic neural network is obtained:

[0142]

[0143] Where, μ ic This represents the learning rate of the neural network. Let represent the improved Q-function obtained by the i-th follower through a trade-off between the two reinforcement learning modes using a reliability factor ρ.

[0144] In this embodiment of the invention, Algorithm 2 proposes how to find an approximately optimal consensus control protocol using only a Critic neural network;

[0145] Algorithm 2 is implemented using only the Critic neural network:

[0146] Step 3.1: Provide behavioral control strategies for follower i, and collect system data x using equations (1) and (3) respectively. i(k+1) The local neighbor error δ of all followers i i(k+1) ;

[0147] Step 3.2: Initialize the Critic network weights of follower i and give the permissible control law.

[0148] Step 3.3: Calculate using the collected data. The data includes data generated by behavior control strategies and data utilizing updated target strategies. Data collected online from followers;

[0149] Step 3.4: Based on the Critic network weight update formula (25), update the weights of the network. The right to conduct training;

[0150] Step 3.5, if Then stop iterating and... Substitute into equation (19) to obtain an approximate value of the optimal consensus control protocol; otherwise, return to step 3.3.

[0151] Step 3.6: Use the consensus control protocol corresponding to the Critic network weights of the trained follower i as the optimal control protocol.

[0152] In embodiments of the present invention, such as Figures 2-8 As shown, in order to more intuitively demonstrate the effectiveness of the multi-agent consensus reinforcement learning method based on the improved Q function proposed in this invention, MATLAB software is used to simulate and verify the method proposed in this invention.

[0153] The linear discrete-time state equation for follower i is:

[0154] x i (k+1)=A i x i (k)+B i u i (k) (26)

[0155] Where i = 1, ..., 4,

[0156] The leader's state equation is:

[0157] x0(k+1)=A0x0(k) (27)

[0158] in,

[0159] In this embodiment of the invention, Figure 4 This is the weight graph of the Critic neural network for follower i implementing Algorithm 2. Figure 5 This is the state trajectory diagram of the leader and follower i implementing Algorithm 2. Figure 6 It is the local neighbor error graph of follower i implementing Algorithm 2, from Figure 4 , Figure 5 and Figure 6 It can be seen that the multi-agent system proposed in this invention can keep up with the leader's movement trajectory very well. Figure 7 To implement the non-policy Q-learning algorithm, the state trajectory graphs of follower i and leader are shown. Figure 8 It is the local neighbor error map of follower i implemented using the non-policy Q-learning algorithm. From Figures 4-8 As can be seen from the simulation curves, when the weights gradually converge to a stable value, the state of the system also tends to be consistent. Moreover, when noise is added to the four followers between 200s and 400s, they also reach consistency after running. The simulation curves show that the method proposed in this invention can improve the self-learning ability of reinforcement learning, make the agent more sensitive to environmental changes, learn faster, enhance learning reliability, improve learning efficiency, and at the same time realize the consistency of heterogeneous multi-agent systems in an approximately optimal way.

[0160] In this embodiment of the invention, a multi-agent consensus reinforcement learning method system based on an improved Q-function is provided, which uses the multi-agent consensus reinforcement learning method based on an improved Q-function as described in any one of claims 1-6 to perform optimal consensus control of a heterogeneous multi-agent system.

Claims

1. A multi-agent consensus reinforcement learning method based on improved Q function, characterized in that: Includes the following steps: Step 1: Establish a dynamic model of the linear discrete-time heterogeneous multi-agent system; Consider a linear discrete-time heterogeneous multi-agent system with one follower and one leader, the follower has the dynamics model: (1) in, and They represent the first One follower The status and control input at any given time, and They represent the first The system matrix and control matrix of each follower; The leader's dynamic model, i.e., the reference trajectory in a heterogeneous multi-agent system, is as follows: (2) in, Representative Leader The state at any given moment, The system matrix representing the leader; Step 2: Define the leader-follower consistency error equation for the heterogeneous multi-agent system and introduce a reliability factor. A compromise is made between non-policy Q-learning and policy Q-learning modes, and a distributed control protocol based on an improved Q function is designed. Step 2 defines the leader-follower consistency error equation for the heterogeneous multi-agent system as follows: Definition of the first The local neighbor synchronization error of a follower is: (3) in, Indicates the first Local neighbor synchronization error of a follower Indicates follower With followers The weights between them Indicates the first One follower The state at any given moment, Indicates the first A fixed gain for each follower and leader. Represents the follower in the system topology diagram The number of neighboring nodes, Represents the follower in the system topology diagram To followers The sum of all topological relationships; The global synchronization error of a heterogeneous multi-agent system is defined as: (4) in, This represents the global synchronization error of a heterogeneous multi-agent system. express The global state of N followers at any given time. , This represents the transpose of a vector. express The global state of the leader corresponding to each follower at any given moment. ; Global synchronization error Synchronization error with global neighbors The following relationship exists between them: (5) in, The Laplace matrix represents the topology of the system. Represent a An identity matrix of dimension 1 This represents a diagonal matrix with a fixed gain. Indicates the Kronecker product; For any one of the followers Its performance metrics are defined as follows: (6) in, (7) in, Indicates follower The performance indicators Indicates the first One follower Timing control protocol Indicates the first A follower's neighbor Timing control protocol Indicates the first A follower's control protocol Indicates follower The sum of all corresponding moments, Indicates the first The effect function of each follower Indicates the first One follower Local neighbor synchronization error at time t, This represents a symmetric matrix with appropriate dimensions. Indicates the first One follower Timing control protocol; Step 2 introduces the reliability factor To compromise between non-policy Q-learning and policy Q-learning, a distributed control protocol based on an improved Q-function is designed as follows: Define a reliability factor By compromising between non-strategy Q-learning and strategy Q-learning modes, the improved Q-function is obtained as follows: (13) The distributed control protocol based on the improved Q-function is as follows: (14) in, It is by and The first of the composition One follower Global synchronization error at time t, Indicates the first One follower Data generated by the target strategy at any given time. Indicates the first One follower The goal strategy at any time Indicates the learning rate. Indicates control strategy , express hour right The gradient is: (15) Step 3: Use the Critic neural network to learn the optimal control protocol from the data generated by the system dynamic model, and perform consistent control on the heterogeneous multi-agent system according to the optimal control protocol.

2. The multi-agent consensus reinforcement learning method based on the improved Q-function according to claim 1, characterized in that: Step 3 includes the following steps: Step 3.1, for followers Provide behavior control strategies and collect data. One follower The state at any given time and the local neighbor synchronization error; Step 3.2: Initialize Followers The Critic network weights enable the implementation of control protocols for heterogeneous multi-agent systems. Step 3.3: Based on the collected data... One follower The state at time and the local neighbor synchronization error are used to obtain the follower. The approximation error of the neural network; Step 3.4, based on the followers The neural network approximation error for followers The Critic network weights are trained; Step 3.5: Determine the followers after training. If the weights of the Critic network have reached the threshold, proceed to the next step; otherwise, return to step 3.

3. Step 3.6: Train the followers The consensus control protocol corresponding to the Critic network weights is taken as the optimal control protocol.

3. The multi-agent consensus reinforcement learning method based on the improved Q-function according to claim 2, characterized in that: Step 3.4 describes the process for followers. The Critic network weights are trained using the following formula: (25) in, This represents the updated Critic network weights. Represents the weights of the Critic neural network. This represents the learning rate of the neural network. Indicates the first A follower through reliability factor An improved Q-function obtained by compromise between the two reinforcement learning modes. This represents the difference between the activation function at the previous time step and the next time step. The column vector formed by these.

4. A multi-agent consensus reinforcement learning method system based on an improved Q-function, characterized in that: The optimal consensus control of heterogeneous multi-agent systems is achieved using the multi-agent consensus reinforcement learning method based on the improved Q function as described in any one of claims 1-3.

Images