Multi-agent fixed time optimal consistency algorithm based on reinforcement learning

Abstract

The invention provides a multi-agent fixed time optimal consistency control method based on reinforcement learning, and relates to the technical field of multi-agent control. The method comprises the steps of determining a multi-agent system dynamic model, approximately solving optimal control through reinforcement learning, designing a double-event trigger function to reduce communication loss, and filling a control algorithm into multiple agents. The optimal control strategy is approached through interactive iteration of the actor neural network and the critic neural network, the control strategy can be learned without an accurate system model, communication resource loss is effectively reduced, and system operation efficiency is improved. The optimal consistency control in the fixed time is realized, and the system performance and reliability are improved. The method is suitable for the fields of intelligent transportation, intelligent power grids, robot cooperation, unmanned aerial vehicle cluster control and the like, and meets the requirements of actual application for the convergence speed of the system.

Description

Technical Field

[0001] The present invention relates to the field of multi-agent control technology, and in particular focuses on a multi-agent fixed-time optimal consistency control method based on reinforcement learning. Background Art

[0002] With the development of science and technology, multi-agent systems have shown great application potential in many fields such as intelligent transportation, smart grid, robot collaboration, and drone swarm control. However, multi-agent systems face many challenges in practical applications.

[0003] The dynamic model of a multi-agent system is usually complex and contains a large number of nonlinear factors. For example, in a robot collaboration scenario, the robot's motion is affected by a variety of nonlinear factors such as its own mechanical structure, environmental friction, and load changes. In addition, in many cases, there are unknown parts in the system, making it extremely difficult to accurately establish a system model. Traditional model-based control methods are limited in effectiveness when facing such complex unknown systems.

[0004] In a multi-agent system, agents need to communicate to coordinate their behaviors. In the traditional time-triggered communication mode, agents exchange information at fixed time intervals. However, when the system state changes slowly, frequent communication does not provide more valuable information, but consumes a lot of communication resources and energy. For example, in a large-scale smart grid system, many distributed energy resources and devices such as smart meters use time-triggered communication, which will generate a large amount of communication data, increase the system burden, and reduce operating efficiency.

[0005] In order to improve the performance and efficiency of multi-agent systems, a control method that can handle unknown parts of the system and reduce communication losses is needed. Reinforcement learning can learn the optimal control strategy without an accurate system model, providing a new approach to solving multi-agent system control problems. At the same time, meeting the requirements of practical applications for system convergence speed and achieving optimal consistency control within a fixed time have become research focuses, which will help the system achieve an ideal collaborative state within a specified time and improve overall performance and reliability. Summary of the invention

[0006] The present invention relates to the field of multi-agent control technology, and in particular focuses on a multi-agent fixed-time optimal consistency control method based on reinforcement learning.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] 1. A multi-agent fixed-time optimal consistency control method based on reinforcement learning, characterized in that it includes the following steps:

[0009] S1: Define the dynamics model of the multi-agent system, which contains N agents. The dynamics model of the i-th agent is

[0010] x i =F i (x 1i ,x 2i )+B i U i

[0011] where x i (t)∈R n ,U i (t)∈R n ,f i (x i (t))∈R n They represent the state of the agent, the control input, and possibly unknown nonlinear functions. The input amplitude is restricted to ‖U i ‖≤δ i , it is known that i is the upper bound of the input.

[0012] S2: Using reinforcement learning technology, the optimal control strategy is approximately solved through the input and output data of the system. In this process, the integral reinforcement learning method is used to update the weights of the neural network so that the neural network can adapt to the unknown characteristics of the system and gradually converge to the optimal control strategy.

[0013] S3: Design a dual event trigger function to control the communication and control of the system respectively. When the trigger conditions are met, communication and status update are performed to reduce the loss of communication and control resources.

[0014] S4: The above-mentioned unknown model identification strategy and new control algorithm are programmed into each multi-agent so that it can operate according to the designed rules.

[0015] Compared with the prior art, the present invention has the following significant advantages:

[0016] The two major innovations of the present invention improve the control efficiency and performance of the multi-agent system. First, a fixed-time optimal consistency control algorithm based on reinforcement learning is proposed. The algorithm approximates the optimal control strategy through the interactive iteration of the actor and critic neural networks, and can learn the control strategy without an accurate system model. Secondly, an event trigger mechanism is introduced to reduce the loss of communication resources and improve the system operation efficiency. By designing the optimal tracking algorithm and event trigger function, the present invention ensures that the system achieves the optimal performance while meeting the consistency, and is suitable for the fixed-time optimal consistency control of the multi-agent system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the multi-agent communication topology diagram in the first example

[0018] Figure 2is the speed error of each agent in the first example

[0019] Figure 3 The multi-agent communication topology diagram in the second example

[0020] Figure 4 The state diagram of the auxiliary system in the second example is

[0021] Figure 5 is the position state of each agent in the second example DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention clearer and easier to understand, the present invention is further described in detail below with reference to the accompanying drawings and implementation examples.

[0023] A multi-agent fixed-time optimal consistency control method based on reinforcement learning, characterized by comprising the following steps:

[0024] S1. Clarify the system dynamics model

[0025] Determine a multi-agent dynamics model based on reinforcement learning containing N agents, and the dynamics model of the i-th agent is:

[0026] x i =F i (x 1i ,x 2i )+B i U i (1)

[0027] where x i (t)∈R n ,U i (t)∈R n ,f i (x i (t))∈R n They represent the state of the agent, the control input, and possibly unknown nonlinear functions. The input amplitude is restricted to ‖U i ‖≤δ i , it is known that i is the upper bound of the input.

[0028] Since the control input amplitude is limited, the cost function is considered:

[0029]

[0030] Using stationary conditions Get the gradient of the optimal value function:

[0031]

[0032] Next, we use the actor-critic framework to approximate the optimal controller.

[0033] S2. Algorithm design based on reinforcement learning

[0034] Since the system dynamics are partially unknown, in order not to apply the system dynamics in the process of solving the HJB equation, the idea of ​​integral reinforcement learning is used to approximate the optimal control.

[0035] critic network design

[0036] Based on integral reinforcement learning, the optimal value function is approximated, the optimal value function is reconstructed through a neural network and its gradient is calculated:

[0037]

[0038] The Bellman error is calculated according to the formula:

[0039]

[0040] in

[0041] Defining the objective function based on Bellman error Using the gradient descent method, the update rate of the critic neural network is obtained according to the corresponding formula:

[0042]

[0043] Actor Network Design

[0044] The controller is constructed based on the critic neural network, where the controller is designed as follows:

[0045]

[0046] in right Estimates.

[0047] right Use gradient descent to reduce the update rate of the actor neural network Designed for:

[0048]

[0049] Where k3 is the parameter to be designed.

[0050] S3. Design of optimal consistency algorithm for dual event trigger mechanism

[0051] In the above, we discussed the problem of optimal fixed-time consensus for a class of nonlinear multi-agent systems with unknown dynamics. However, in practical applications, the communication resources of the system are often limited. Therefore, it is necessary to design a dual-event trigger mechanism to reduce communication loss.

[0052] An event triggering mechanism is designed in the optimal consensus algorithm. First, a first-order integrator auxiliary system is introduced for each agent, where the dynamics of the i-th multi-agent auxiliary system is expressed as follows:

[0053]

[0054] in represents the state of the system, τ i ∈R m Represents the input of the system

[0055] Design an optimal tracking algorithm so that the multi-agent system can track the auxiliary system z i The tracking error of the ith agent can be reformulated as follows:

[0056]

[0057] The performance indicators of the system are defined as follows:

[0058]

[0059] in

[0060] So the HJB equation can be expressed as follows:

[0061]

[0062] Based on the defined objective function, the update rate of the critic neural network is designed using the gradient descent method as follows:

[0063]

[0064] in k ic ,k1,k2 are the parameters to be designed.

[0065] S4. Injecting new algorithms into intelligent agents

[0066] In the experiment of multi-agent fixed-time optimal consensus control based on reinforcement learning, the above unknown model identification strategy and new control algorithm are programmed into each multi-agent. i (t)∈R,u i (t)∈R The new Laplace corresponding to the multi-agent communication topology in this embodiment is as follows:

[0067]

[0068] The nonlinear function of the system dynamics is defined as:

[0069] f i (x 1i ,x 2i ) = sin(x 1i )+x 1i x 2i ,i=1,2,3,4

[0070] Arbitrarily, the initial state of the system is set to: x1(0) = [1; 0], x2(0) = [-1; 1], x3(0) = [0; 1], x4(0) = [2; 3]. The upper bound of the control δ i =1; performance index weight matrix Q i = I. Both neural networks use 24 neurons and the integration interval T = 0.1s.

[0071] from Figure 2 It can be seen that the speed errors of each agent tend to be consistent after 5s, that is, the algorithm can achieve fixed-time optimal consistency control of the multi-agent system.

[0072] In the experiment of event-triggered multi-agent fixed-time optimal consensus control based on reinforcement learning, the above-mentioned unknown model identification strategy, event triggering strategy, and new control algorithm are programmed into each multi-agent. i (t)∈R,u i t()∈R The Laplace corresponding to the multi-agent communication topology in this embodiment is as follows:

[0073]

[0074] The nonlinear function of the system dynamics is defined as:

[0075] f i (x 1i ,x 2i )=(1+sin(x 1i )cos(x 2i )) 2 +vsin(x 2i ),i=1,2,3,4,5

[0076] Arbitrarily, the initial state of the system is set to: x1(0) = [0; 1], x2(0) = [5; 1], x3(0) = [-2.5; 2], x4(0) = [-2; 0.5], x5(0) = [2; 1] The upper bound δ of the control i =2; performance index weight matrix Q i=I, two neural networks use 24 neurons, and the integration interval T = 0.1s. The parameters in the event trigger mechanism are: c1 = 10 -4 ,c2=10 -2 ,c3=5.

[0077] Figure 4 The position error of each agent is given. Figure 5 The speed errors of each agent are given, and it can be seen that the speed errors of each agent tend to be consistent after 10s, and the error tends to 0. It is proved that the proposed new algorithm can achieve the expected goal and confirms the feasibility of the algorithm.

[0078] The above embodiments are only preferred implementation schemes of the present invention. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A multi-agent fixed-time optimal consensus method based on reinforcement learning, characterized in that: The following steps are involved: S1: Define the dynamics model of the multi-agent system, which contains N agents. The dynamics model of the i-th agent is x i =F i (x 1i ,x 2i )+B i U i where x i (t)∈R n , U i (t)∈R n ,f i (x i (t))∈R n They represent the state of the agent, the control input, and possibly unknown nonlinear functions. The input amplitude is restricted by ||U i ||≤δ i , it is known that i is the upper bound of the input. S2: Using reinforcement learning technology, the optimal control strategy is approximately solved through the input and output data of the system. In this process, the integral reinforcement learning method is used to update the weights of the neural network so that the neural network can adapt to the unknown characteristics of the system and gradually converge to the optimal control strategy. S3: Design a dual event trigger function to control the communication and control of the system respectively. When the trigger conditions are met, communication and status update are performed to reduce the loss of communication and control resources. S4: The above-mentioned unknown model identification strategy and new control algorithm are programmed into each multi-agent so that it can operate according to the designed rules.

2. According to the reinforcement learning-based multi-agent fixed-time optimal consistency control method of claim 1, it is characterized by: In S1: x i =F i (x 1i ,x 2i )+B i U i Defined as a dynamic model of a multi-agent system, where the input amplitude is constrained by ||U i ||≤δ i , it is known that i is the upper bound of the input.

3. According to the reinforcement learning-based multi-agent fixed-time optimal consistency control method of claim 1, it is characterized in that: In S2: For the value function network, the optimal value function is approximated based on integral reinforcement learning, and the optimal value function is reconstructed through the neural network. And calculate its gradient At the same time, the objective function is defined based on the value function error, and the Bellman error is calculated according to the formula:

4. Define the objective function based on Bellman error Using the gradient descent method, the update rate of the critic neural network is obtained according to the corresponding formula: The controller is constructed based on the critic neural network, where the controller is designed as follows: in Yes Estimates. right Use gradient descent to reduce the update rate of the actor neural network Designed for: Where k3 is the parameter to be designed.

5. According to the reinforcement learning-based multi-agent fixed-time optimal consistency control method of claim 1, it is characterized in that: In S3: Design an optimal tracking algorithm so that the multi-agent system can track the auxiliary system z i The tracking error of the ith agent can be reformulated as follows: The performance indicators of the system are defined as follows: in So the HJB equation can be expressed as follows:

6. Design an event trigger mechanism. On the basis of the auxiliary system, reasonably define the event trigger function, and based on the defined objective function, use the gradient descent method to design the update rate of the critic neural network as: in k ic ,k1, k2 are the parameters to be designed.

7. According to the reinforcement learning-based multi-agent fixed-time optimal consistency control method of claim 1, it is characterized in that: In S4: In the experiment of multi-agent fixed-time optimal consensus control based on reinforcement learning, the above-mentioned unknown model identification strategy and new control algorithm are programmed into each multi-agent.

Images