Data-driven optimal proportional cooperative control method for multi-robot system

Abstract

This invention discloses a data-driven optimal proportional cooperative control method for multi-robot systems, comprising the following steps: S1, establishing a multi-robot dynamics model described by a general linear system, and constructing the optimal proportional consistency control problem of the multi-robot system using the Bellman optimality principle; S2, designing an adaptive dynamic programming algorithm to calculate approximate solutions to the discrete-time Hamilton-Jacobi-Bellman equations based on generalized policy iteration; S3, building an evaluation-execution neural network to fit the iterative control law and performance index respectively; S4, deploying the evaluation-execution network controller designed in step S3 to the robot swarm. This invention's method, targeting multi-robot systems described by a general linear system, considers optimal control situations with specific task objectives and uses an evaluation-execution network to achieve online optimal cooperative control, making it applicable to the field of swarm cooperative control of multi-robot systems.

Description

Technical Field

[0001] This invention belongs to the field of collaborative control technology for multi-robot systems, and specifically relates to a data-driven optimal proportional collaborative control method for multi-robot systems. Background Technology

[0002] In recent years, cooperative distributed control of multi-robot systems has become a research hotspot in the control field due to its wide range of applications. Multi-robot cooperative control refers to designing control algorithms to enable multiple robots to communicate and work collaboratively to complete a specific task. However, in engineering applications, achieving cooperative behavior in multi-robot systems involves not only robot consistency but also minimizing energy consumption during the cooperative control process. Therefore, finding an effective solution to the optimal cooperative control problem of multi-robot systems is a pressing issue that needs to be addressed.

[0003] Currently, numerous studies have focused on the cooperative optimal control problem of multi-robot systems. However, on the one hand, most existing research has not considered that the motion states of robots with different tasks converge to an equilibrium state dependent on initial conditions, rather than a common value, which greatly limits practical applications. On the other hand, existing methods for solving optimal cooperative control problems often result in excessively large computational dimensions, making them difficult to solve. Therefore, research on data-driven control algorithms that can avoid the curse of dimensionality is of great significance. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a data-driven optimal proportional cooperative control method for multi-robot systems that takes into account the optimal control situation with specific task objectives, uses an evaluation-execution network to achieve online optimal cooperative control, and is applicable to the field of cluster cooperative control of multi-robot systems.

[0005] The objective of this invention is achieved through the following technical solution: a data-driven optimal proportional cooperative control method for multi-robot systems, the specific steps of which are as follows:

[0006] S1. Establish a multi-robot dynamics model described by a general linear system, and construct an optimal proportional consistency problem based on the task objective; the robot's dynamics model is as follows:

[0007] x i (k+1)=Ax i (k)+B i u i (k), i = 1, 2, ..., N (1)

[0008] Where x i (k)∈R nThis is the system state quantity of the i-th robot at time k, where n represents the dimension of the robot system state, and N represents the number of robots. It is the control input, m i This indicates the control input dimension, A∈R n×n It is a system matrix. It is the input matrix;

[0009] The communication network between multiple robot systems is abstracted as a directed graph. From a set of vertices An edge set and a weighted adjacency matrix Composition; where the critical element is non-negative a ij ≥0; if and only if (v j ,v i When )∈ε, a ij >0 means that robot i can receive information from robot j; otherwise, a ij =0; for a ii =0; node v i The neighbor set is represented as The in-degree matrix is ​​represented as a diagonal matrix. Then through Obtain the Laplace matrix

[0010] The proportional cooperative behavior of a multi-robot system refers to designing a corresponding distributed control law u for each robot i. i (k) is a formula that enables all robots to converge to different states according to a preset proportional coefficient and remain stable. This is called the achievement of proportional cooperative behavior, and the problem is characterized by the following mathematical representation:

[0011]

[0012] Where α i Let be the collaboration ratio value of the i-th robot;

[0013] Then, for each robot i, the proportional coordination error is defined as follows:

[0014]

[0015] set up Given the proportional coordination consistency error vector, we get:

[0016]

[0017] in

[0018] Based on formulas (1) and (3), the dynamic equation for the proportional consistency error ∈i(k) is obtained as follows:

[0019]

[0020] Each robot i needs to minimize the performance index function V. i (∈ i (k),u i (k),u (j) (k)), u (j) (k) is the set of control inputs to robot i's neighboring robots; the optimal control law will be designed next. To minimize the performance index function, while ensuring that the proportional consistency error of each robot ∈ [0, 1] as k approaches infinity. i (k)→0;

[0021] The local performance index function for each robot is defined as follows:

[0022]

[0023] Where c i (∈ i (k),u i (k),u (j) (k)) is the utility function; γ∈(0,1] is the discount factor; let V i (∈ i (k),u i (k),u (j) (k))=V i (∈ i (k));

[0024] According to the Bellman optimality principle, the optimal performance index function satisfies the following discrete-time Hamilton-Jacobi-Bellman equation:

[0025]

[0026] The optimal control law is:

[0027]

[0028] Optimal control law Substituting into the HJB equation of formula (7), we get:

[0029]

[0030] S2. Design an adaptive dynamic programming algorithm to calculate approximate solutions to the discrete-time Hamilton-Jacobi-Bellman equation based on generalized policy iteration;

[0031] S3. Construct an evaluation-execution neural network to fit the iterative control law and performance index respectively, and iteratively update the network weights according to the gradient descent algorithm;

[0032] S4. Deploy the evaluation-execution network controller designed in step S3 into the robot swarm.

[0033] The specific implementation method of step S2 is as follows: Define a utility function that comprehensively considers energy loss and system control process error as follows:

[0034]

[0035] Q ii >0,R ii >0,R ij >0, all are positive definite symmetric weight matrices;

[0036] set up and V i p (∈(k)) represent the iterative control law and the iterative performance index function, respectively; where p represents the iterative index and k represents the time step; let V i p,q (∈ i (k) is used as the sub-iteration performance index function, where the iteration index q = 0, 1, 2, ..., N j , where N j It is the preset maximum number of sub-iterations;

[0037] The iterative process of the adaptive generalized policy iterative algorithm is as follows:

[0038] S21. For the control law of p = 1, 2, ... The update rules are as follows:

[0039]

[0040] S22, Fixed Control Law The iterative performance index function V at step p i p (∈(k)) is calculated in the following two stages:

[0041] S221. For q = 1, 2, ..., N j :

[0042]

[0043] in,

[0044]

[0045] S222, the value function of the p-th sub-iteration is:

[0046]

[0047] The specific implementation method of step S3 is as follows: the evaluation network of robot i utilizes the error ∈ i Control input u i and its neighbors' control input u (j) The input to the network is used as the network input, and the output is the performance index function V. i An approximation of is expressed as:

[0048]

[0049] Where X ci (k) is derived from ∈ i (k),u i (k),u (j) The input vector Y of the critic network agent i composed of (k) ci It is the weight matrix, w ci It is the weight matrix of the hidden layer and output layer of the neural network; φ ci (·) is chosen as the activation function for the critic:

[0050]

[0051] Then, define the error function of the critic network:

[0052]

[0053] Therefore, the goal is to minimize the loss function:

[0054]

[0055] Using the gradient descent principle, the online update rule for the network weight parameters is designed as follows:

[0056]

[0057] Where, β c It is the network learning rate;

[0058] The execution network of robot i utilizes error ∈ i As the input to the network; the output is the control law u. i An approximation of is expressed as:

[0059]

[0060] Where X ai (k) is derived from ∈ iThe input vector of the actor network robot i composed of (k), Y ai It is the weight matrix, w ai It is the weight matrix of the hidden layer and the output layer of the neural network, φ ai (·) serves as the activation function constraint control signal for the actor;

[0061] Then, define the error function for the actor network:

[0062] ε ai =V i -U c (twenty one)

[0063] Among them, U c Let U be the cost function. c =0; therefore, the goal is to minimize the loss function:

[0064]

[0065] Using the gradient descent principle, the online update rule for the network weight parameters is designed as follows:

[0066]

[0067] Where, β a It is the network learning rate.

[0068] The beneficial effects of this invention are as follows: First, a multi-robot dynamics model is established. Based on the task objective, the optimal proportional consistency problem is identified. An adaptive dynamic programming algorithm is designed to calculate an appropriate approximate solution to the discrete-time HJB equation based on generalized policy iteration. On this basis, an evaluation-execution neural network is built to fit the iterative control law and performance index respectively. The network weights are iteratively updated using a gradient descent algorithm, thus realizing data-driven online optimal cooperative control of the multi-robot system. This invention's method is designed for multi-robot systems described by general linear systems. It considers the optimal cooperative control of the multi-robot system under specific task objectives, employs a data-driven approach to achieve online update control, and is independent of the specific system model, making it applicable to the field of cooperative control of multi-robot systems. Attached Figure Description

[0069] Figure 1 This is a flowchart of a data-driven optimal cooperative control method for a multi-robot system according to the present invention;

[0070] Figure 2 This is a schematic diagram of the communication topology of a multi-robot system given in an embodiment of the present invention;

[0071] Figure 3This is a schematic diagram illustrating the optimal proportional consistent cooperative behavior exhibited after deploying a data-driven adaptive dynamic programming control algorithm onto a multi-robot system in an embodiment of the present invention. Detailed Implementation

[0072] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0073] like Figure 1 As shown, the present invention provides a data-driven optimal proportional cooperative control method for a multi-robot system, the specific steps of which are as follows:

[0074] S1. Establish a multi-robot dynamics model described by a general linear system, and construct an optimal proportional consistency problem based on the task objective; the robot's dynamics model is as follows:

[0075] x i (k+1)=Ax i (k)+B i u i (k), i = 1, 2, ..., N (1)

[0076] Where x i (k)∈R n It is the system state variable of the i-th robot at time k, usually represented by the robot's spatial position coordinates x. i (k)=[X i (k),Y i (k)] T In the unmanned logistics industry, ground vehicles are used as robots. Their spatial coordinates are obtained by the position sensors and inertial measurement units carried by the vehicles themselves. These coordinates are used to describe the spatial position and motion of the ground vehicle robots, enabling localization, trajectory planning, and motion control of the vehicles. This allows for the cluster delivery of multiple robot systems in the unmanned logistics industry, autonomous obstacle avoidance, and efficient logistics transportation. 'n' represents the state dimension of the robot system, and 'N' represents the number of robots. It is the control input, m i This indicates the control input dimension, A∈R n×n It is a system matrix. It is the input matrix, where the system matrix and the input matrix are determined by the specific physical model of the object under study.

[0077] Multi-robot system communication structure such as Figure 2 As shown, each robot agent has the ability to send or receive information from its neighbors. The communication network between the multi-robot system is abstracted as a directed graph. From a set of vertices An edge set and a weighted adjacency matrix Composition; where the critical element is non-negative a ij ≥0; if and only if At that time, a ij >0 means that robot i can receive information from robot j; otherwise, a ij =0; for a ii =0; node v i The neighbor set is represented as The in-degree matrix is ​​represented as a diagonal matrix. Then through Obtain the Laplace matrix

[0078] The proportional cooperative behavior of a multi-robot system refers to designing a corresponding distributed control law u for each robot i. i (k) is a formula that enables all robots to converge to different states according to a preset proportional coefficient and remain stable. This is called the achievement of proportional cooperative behavior, and the problem is characterized by the following mathematical representation:

[0079]

[0080] Where α i Let be the collaboration ratio value of the i-th robot;

[0081] Then, for each robot i, the proportional coordination error is defined as follows:

[0082]

[0083] set up Given the proportional coordination consistency error vector, we get:

[0084]

[0085] in

[0086] Based on formulas (1) and (3), the proportional consistency error ∈ i The dynamic equation of (k) is:

[0087]

[0088] Each robot i needs to minimize the performance index function V. i (∈ i (k),u i (k),u (j) (k)), u (j) (k) is the set of control inputs to robot i's neighboring robots; the optimal control law will be designed next. To minimize the performance index function, while ensuring that the proportional consistency error of each robot ∈ [0, 1] as k approaches infinity. i (k)→0;

[0089] The local performance index function for each robot is defined as follows:

[0090]

[0091] Where c i ( i (k),u i (k),u (j) (k)) is the utility function; γ∈(0,1] is the discount factor; for convenience, let V be... i (∈ i (k),u i (k),u (j) (k))=V i (∈ i (k));

[0092] According to the Bellman optimality principle, the optimal performance index function satisfies the following discrete-time Hamilton-Jacobi-Bellman (HJB) equation:

[0093]

[0094] The optimal control law is:

[0095]

[0096] Optimal control law Substituting into the HJB equation of formula (7), we get:

[0097]

[0098] S2. Design an adaptive dynamic programming algorithm to calculate approximate solutions to the discrete-time Hamilton-Jacobi-Bellman equations based on generalized policy iteration; the specific implementation method is as follows: define a utility function that comprehensively considers energy loss and system control process error as follows:

[0099]

[0100] Q ii >0,R ii >0,R ij >0, all are positive definite symmetric weight matrices;

[0101] set up and V i p(∈(k)) represent the iterative control law and the iterative performance index function, respectively; where p represents the iterative index and k represents the time step; for ease of algorithm design, let V be denoted as V. i p,q (∈ i (k) is used as the sub-iteration performance index function, where the iteration index q = 0, 1, 2, ..., N j , where N j It is the preset maximum number of sub-iterations;

[0102] The iterative process of the adaptive generalized policy iterative algorithm is as follows:

[0103] S21. For the control law of p = 1, 2, ... The update rules are as follows:

[0104]

[0105] S22, Fixed Control Law The iterative performance index function V at step p i p (∈(k)) is calculated in the following two stages:

[0106] S221. For q = 1, 2, ..., N j :

[0107]

[0108] in,

[0109]

[0110] S222, the value function of the p-th sub-iteration is:

[0111]

[0112] S3. Based on the adaptive dynamic programming algorithm framework in step S2, build an evaluation-execution neural network to fit the iterative control law and performance index respectively, and iteratively update the network weights according to the gradient descent algorithm.

[0113] The specific implementation method is as follows: the evaluation network of robot i utilizes the error ∈ i Control input u i and its neighbors' control input u (j) The input to the network is used as the network input, and the output is the performance index function V. i An approximation of is expressed as:

[0114]

[0115] Where X ci (k) is derived from ∈i (k),u i (k),u (j) The input vector Y of the critic network agent i composed of (k) ci It is the weight matrix, w ci It is the weight matrix of the hidden layer and output layer of the neural network; φ ci (·) is chosen as the activation function for the critic:

[0116]

[0117] Then, define the error function of the critic network:

[0118]

[0119] Therefore, the goal is to minimize the loss function:

[0120]

[0121] Using the gradient descent principle, the online update rule for the network weight parameters is designed as follows:

[0122]

[0123] Where, β c It is the network learning rate;

[0124] The execution network of robot i utilizes error ∈ i As the input to the network; the output is the control law u. i An approximation of is expressed as:

[0125]

[0126] Where X ai (k) is derived from ∈ i The input vector of the actor network robot i composed of (k), Y ai It is the weight matrix, w ai It is the weight matrix of the hidden layer and the output layer of the neural network, φ ai (·) serves as the activation function constraint control signal for the actor;

[0127] Then, define the error function for the actor network:

[0128] ε ai =V i -U c (twenty one)

[0129] Among them, U c Let U be the cost function. c=0; therefore, the goal is to minimize the loss function:

[0130]

[0131] Using the gradient descent principle, the online update rule for the network weight parameters is designed as follows:

[0132]

[0133] Where, β a It is the network learning rate.

[0134] S4. Deploy the evaluation-execution network controller designed in step S3 into the robot swarm. This enables data-driven online optimal collaborative control of a multi-robot system under specific task objectives, with the final effect as follows: Figure 3 As shown.

[0135] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A data-driven optimal proportional cooperative control method for multi-robot systems, characterized in that, The specific steps are as follows: S1. Establish a multi-robot dynamics model described by a general linear system, and construct an optimal proportional consistency problem based on the task objective; the robot's dynamics model is as follows: （1）； in It is the system state variable of the i-th robot at time k. This represents the state dimension of the robot system, where N represents the number of robots. It is a control input. This indicates the control of the input dimension. It is a system matrix. It is the input matrix; The communication network between multiple robot systems is abstracted as a directed graph. , consisting of a set of vertices An edge set and a weighted adjacency matrix Composition; where the critical element is non-negative If and only if hour, Representative robot Able to receive robot Information; otherwise, ;for ;node The neighbor set is represented as The in-degree matrix is ​​represented as a diagonal matrix. Then through Obtain the Laplace matrix ; Proportional cooperative behavior of multi-robot systems refers to the behavior of each robot in a coordinated manner. Design the corresponding distributed control law. If all robots can converge to different states according to a preset proportional coefficient and remain stable, then proportional cooperative behavior is said to be achieved. This problem is characterized by the following mathematical representation: （2）； in For the first The collaboration ratio of each robot; Then, for each robot Define proportional coordination error: （3）； set up Given the proportional cooperative error vector, we get: （4）； in ; Based on formulas (1) and (3), the proportional cooperative error is obtained. The dynamic equation is: （5）； Each robot All need to minimize the performance metric function , It's a robot. The set of control inputs for neighboring robots; next, the optimal control law will be designed. To minimize the performance metric function, while ensuring that when As the value approaches infinity, the proportional consistency error of each robot ; The local performance index function for each robot is defined as follows: （6）； in It is a utility function; It is a discount factor; denoted as... ; According to the Bellman optimality principle, the optimal performance index function satisfies the following discrete-time Hamilton-Jacobi-Bellman equation: （7）； The optimal control law is: （8）； Optimal control law Substituting into the HJB equation of formula (7), we get: （9）； S2. Design an adaptive dynamic programming algorithm to calculate approximate solutions to the discrete-time Hamilton-Jacobi-Bellman equation based on generalized policy iteration; S3. Construct an evaluation-execution neural network to fit the iterative control law and performance index respectively, and iteratively update the network weights according to the gradient descent algorithm; S4. Deploy the evaluation-execution network controller designed in step S3 into the robot swarm.

2. The optimal proportional cooperative control method for a data-driven multi-robot system according to claim 1, characterized in that, The specific implementation method of step S2 is as follows: Define a utility function that comprehensively considers energy loss and system control process error as follows: （10）； in All are positive definite symmetric weight matrices; set up and Let represent the iterative control law and the iterative performance index function, respectively; where, Indicates the iteration index, Indicates the time step; (Note: The original text contains several grammatical errors and inconsistencies. A more accurate translation would require the full context.) As a sub-iteration performance index function, where the iteration index ,in It is the preset maximum number of sub-iterations; The iterative process of the adaptive generalized policy iterative algorithm is as follows: S21, Regarding Control Law The update rules are as follows: （11）； S22, Fixed Control Law , No. Iterative performance index function of step Calculated in the following two stages: S221, Regarding : （12）； in, （13）； S222, No. The value function of the second iteration is: （14）。 3. The optimal proportional cooperative control method for a data-driven multi-robot system according to claim 1, characterized in that, The specific implementation method of step S3 is as follows: robot Evaluation network utilizes error Control input Control inputs of its neighbors The input is used as the network input, and the output is a performance metric function. An approximation of is expressed as: （15）； in It is by The composition of the critic network intelligent agent The input vector, It is a weight matrix. It is the weight matrix of the hidden layer and the output layer of the neural network; The activation function chosen for the critic is: （16）； Then, define the error function of the critic network: （17）； Therefore, the goal is to minimize the loss function: （18）； Using the gradient descent principle, the following rules are designed for the online update of the weight matrices of the hidden and output layers of the evaluation network: （19）； in, It is the network learning rate; robot The execution network utilizes errors As the input to the network; the output is the control law. An approximation of is expressed as: （20）； in It is by Composed of actor network robots The input vector, It is a weight matrix. It is the weight matrix of the hidden layer and the output layer of the neural network. The activation function of the actor constrains the control signal; Then, define the error function for the actor network: （21）； in, It is the cost function, let Therefore, the goal is to minimize the loss function: （22）； Using the gradient descent principle, the online update rules for the weight matrices of the hidden and output layers of the neural network are designed as follows: （23）； in, It is the network learning rate.

Images