Distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based distributed robust inclusion control method for multiple unmanned harvesters, and belongs to the technical field of distributed robust inclusion control methods for multiple unmanned harvesters, and the method comprises the steps: taking each unmanned harvester as a communication node to construct a communication topology, carrying out the system modeling and problem definition of the multiple unmanned harvesters, and carrying out the control of the multiple unmanned harvesters. The method comprises the following steps: constructing kinematics models of a leader and a follower in a multi-unmanned harvester system, designing a discount value function of the following unmanned harvester, converting distributed robust control into an optimal control problem of a nominal object, setting distributed robust based on reinforcement learning and including a control protocol, and solving the optimal control problem of the nominal object. And constructing a reviewer neural network to approximately solve and couple a Hamilton-Jacobi-Bellman equation, and according to a proposed distributed robust control strategy, enabling all inclusion errors of the random unmanned harvester to be consistent and finally bounded.

Description

Technical Field

[0001] The present invention relates to a distributed robust inclusion control method for multiple unmanned harvesters, in particular to a distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning, and belongs to the technical field of distributed robust inclusion control methods for multiple unmanned harvesters. Background Art

[0002] In recent years, with the continuous advancement of agricultural modernization, the level of intelligence and automation in agricultural production has been significantly improved. As an important part of the agricultural unmanned system, the collaboration of multiple unmanned harvesters has shown great application potential in the field of agricultural harvesters. Compared with traditional manual harvesting or single-machine harvesting methods, multiple unmanned harvesters can significantly improve operating efficiency, increase the utilization rate of agricultural machinery, and improve the quality of agricultural machinery operations. Through automation and intelligent technology, unmanned harvesters can operate continuously in harsh environments, reduce labor costs, and reduce farmers' labor intensity. However, the collaborative control of multiple unmanned harvester systems faces many adjustments, such as the system's dynamic uncertainty, communication delays, interference from complex farmland environments, and energy consumption optimization. External factors will affect the stability and robustness of the system.

[0003] In traditional multi-agent system control research, certain results have been achieved in solving the leader-follower consensus problem of a single leader. However, these methods have limitations in dealing with the problem of multi-leader containment control and cannot fully meet the collaborative operation requirements of multiple unmanned harvester systems in complex field environments. At the same time, traditional control methods often find it difficult to ensure the stability and robustness of the system when faced with dynamic uncertainty of the system. Moreover, the existing methods of using reinforcement learning to solve the problem of collaborative control of multi-agent formations have algorithms that are highly sensitive to parameters, and different parameter settings may lead to different control effects.

[0004] In addition, in existing multi-UAV harvester systems, centralized control or simple distributed control strategies are usually adopted. The centralized control strategy concentrates the control decisions of all UAVs in a central controller. Although this approach can achieve global optimal control, in practical applications, its stability and robustness are often difficult to guarantee due to communication delays and high system complexity. Simple distributed control strategies can reduce system complexity, but due to the lack of intelligent decision-making capabilities, they are difficult to adapt to complex and changing environments. With the rapid development of artificial intelligence technology, reinforcement learning, as a method of learning optimal strategies through the interaction between intelligent agents and the environment, has received widespread attention in the field of UAV control. Reinforcement learning enables UAVs to make adaptive decisions according to environmental changes and mission objectives, and has strong robustness and adaptability. However, existing research on UAV harvester control methods based on reinforcement learning is mostly designed for a single UAV, which is difficult to directly apply to multi-UAV harvester systems. Therefore, a distributed robust inclusion control method for multi-UAV harvesters based on reinforcement learning is designed to solve the above problems. Summary of the Invention

[0005] The main purpose of the present invention is to provide a distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning.

[0006] The purpose of the present invention can be achieved by adopting the following technical solutions:

[0007] The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning includes the following steps:

[0008] Step 1: Build a communication topology using each unmanned harvester as a communication node;

[0009] Step 2: Multi-UAV harvester system modeling and problem definition;

[0010] Step 3: Construct the kinematic model of the leader and follower in the multi-UAV harvester system;

[0011] Step 4: Design a discount value function for the unmanned harvester and transform the distributed robust control into an optimal control problem for its nominal plant.

[0012] Step 5: Set up distributed robustness based on reinforcement learning and include control protocols;

[0013] Step 6: Construct a critic neural network to approximately solve the coupled Hamilton-Jacobi-Bellman equations;

[0014] Step 7: According to the proposed distributed robust control strategy, the inclusion errors of all follower unmanned harvesters are uniformly bounded.

[0015] Preferably, step three also includes defining an included error for following the unmanned harvester.

[0016] Preferably, step one further includes the following steps:

[0017] S11: Assume that a multi-UAV harvester system consists of M UAV harvester leaders and N UAV harvester followers;

[0018] S12: The set of followers and leaders are represented by S f and S l express, is the focus matrix of fixed payoffs from the mth UAV leader to the i-th UAV follower;

[0019] S13: If there is a connection between the mth leader and the nth follower, then m∈S l and otherwise,

[0020] S14: The undirected communication topology graph of the UAV harvester follower is represented as Λ g ={P,ξ,A}, where P = {p1,…,p N} is a set of nodes, ξ={(p i ,p j ):p i ,p j ∈P} is a set of edges, A=[a ij ] is the weighted adjacency matrix;

[0021] S15: If and only if the i-th unmanned harvester follower and the j-th unmanned harvester follower are directly connected, then (p i ,p j )∈ξ, and if (p i ,p j )∈ξ,then a ij >0; if Then a ij =0;

[0022] And for all i=1,…,N, then a ii =0;

[0023] S16: Let N i ={j:(p i ,p j )∈ξ,j≠i} is a neighbor set of the i-th follower, is the set of the ith follower and its neighbors, D=diag{v1,…,v N},in Λg The degree matrix of , L = DA is the Laplace matrix.

[0024] Preferably, step 2 further includes the following steps:

[0025] S21: Construct and initialize the kinematic models of the leader and follower in a multi-UAV harvester system;

[0026] S22: Consider a multi-drone system with M drone leaders and N drone followers. The system kinematics of each drone follower is described as:

[0027]

[0028] Among them, the coordinate system XOY is the global ground inertial coordinate;

[0029] The agricultural machinery coordinate system is xoy;

[0030] The yaw angle of the agricultural machinery is φ;

[0031] The front wheel steering angle is δ;

[0032] The rear axle speed is v;

[0033] The wheelbase of the unmanned harvester is L;

[0034] The above kinematic model is regarded as a control system with control variable u = δ and state variable X = (x, y, φ), which can be expressed as:

[0035]

[0036] where X i =[x i ,y i ,u i ] T ,u i =δ i ;

[0037] The system kinematics of the mth unmanned harvester leader is given by:

[0038]

[0039] in and is a differentiable function.

[0040] Preferably, the finite set S = {s1,…,s q The convex hull Co(S) of} is the smallest convex set containing all the points in S, that is:

[0041]

[0042] A distributed robust inclusion controller based on reinforcement learning is proposed to drive all following unmanned harvesters with dynamic uncertainty to converge to the convex hull spanned by the leader unmanned harvester, i.e.,

[0043] The nominal system of formula (2) can be expressed as:

[0044]

[0045] Preferably, the inclusion error of the following unmanned harvester is defined to measure the distance from the leading unmanned harvester group;

[0046] The inclusion error of the i-th following robot is defined as:

[0047]

[0048] Then, the kinematics including the error can be calculated as:

[0049]

[0050] In the formula

[0051] The inclusion error of the following UAV is defined as a key indicator, which measures the distance and position relationship between each following UAV and the leader UAV group;

[0052] The inclusion error reflects whether the follower UAV successfully converges and remains inside the convex hull spanned by the leader UAV.

[0053] Preferably, step three also includes designing a discount value function that follows the unmanned harvester and using Hamiltonian to solve the discounted optimal value function to obtain a coupled HJB equation, which is specifically:

[0054] The discount value function of the i-th following unmanned harvester is defined as:

[0055]

[0056] When 0<λ<1, the positive upper bound function Γ i (x i ) and the utility function U i (·) Designed to:

[0057]

[0058] where u (-i) ={u j |j∈N i} is the control input of the i-th neighbor following the unmanned harvester, and is a positive definite matrix;

[0059] The Hamiltonian of the i-th unmanned harvester is defined as:

[0060]

[0061] The discounted optimal value function of the i-th following unmanned harvester is:

[0062]

[0063] The HJB equation is satisfied:

[0064]

[0065] in is a set of acceptable controls. Then, the optimal inclusion control law is derived as:

[0066]

[0067] According to formula (8) and formula (9), we can get:

[0068]

[0069] Preferably, step 4 further includes converting the distributed robust inclusion control of the multi-unmanned harvester system into the optimal containment control of its nominal system;

[0070] Based on the obtained approximate optimal value function and approximate inclusion control law, the constraint conditions for all the inclusion errors following the unmanned harvester to be asymptotically stable or uniformly bounded are determined;

[0071] The equivalence of the problem transformation is proved by the following theorem;

[0072] set up is a continuously differentiable positive definite function, It's a K ∞ Class function;

[0073] Consider a multi-unmanned harvester system with dynamic uncertainty, whose error kinematics is given by Equation (7), and the optimal control law is given by Equation (9). If Q ii satisfy;

[0074]

[0075] Then, it is guaranteed that the inclusion error of the i-th following unmanned harvester is asymptotically stable;

[0076] Considering the unmanned harvester system with dynamic uncertainty, the kinematics including the error given by Equation (7) and the weights of the critical neural network updated by (20), if Qii satisfy;

[0077]

[0078] when When , the approximate inclusion control law (9) ensures that the inclusion error of each trailing robot is uniformly bounded, which means that all following UAVs enter the convex hull formed by the UAV leader.

[0079] Preferably, step five further includes defining a discounted optimal value function of the i-th following unmanned harvester as:

[0080]

[0081] in, is the ideal weight vector, is the activation function, h c is the number of hidden neurons, is the reconstruction error. Then, right The partial derivative of is given by:

[0082]

[0083] The approximate discounted value function is given by:

[0084]

[0085] in W ic The estimated value of , similarly, can be obtained;

[0086]

[0087] According to equations (9) and (14), the optimal inclusion control law is rewritten as:

[0088]

[0089] Then, the approximate inclusion control law is given by:

[0090]

[0091] According to equations (10) and (18), the approximate Hamiltonian is:

[0092]

[0093] set up To calculate the weight estimation error, we use the gradient descent algorithm and design the weight update rule for evaluating the neural network as follows:

[0094]

[0095] In the formula α c >0 is the learning rate,

[0096] An adaptive critic design method based on reinforcement learning is combined with the approximation capability of neural networks to construct a critic neural network to learn and optimize control strategies online.

[0097] Preferably, in step six, a multi-unmanned harvester system including six unmanned harvesters is used to verify the effectiveness of the developed distributed robust inclusion control scheme based on reinforcement learning;

[0098] The system consists of three leading and three following unmanned harvesters, which exchange information based on a predefined communication topology. By constructing a detailed system kinematic model and introducing dynamic uncertainty to simulate a real-world operating environment, the performance of the unmanned harvester system's control strategy was tested.

[0099] During the simulation, we focus on the dynamic changes of the included error of the unmanned harvester to verify whether it can converge to a small zero region as expected;

[0100] If, in the presence of system modeling uncertainty and external disturbances, the trajectories of all following UAVs successfully enter the convex hull formed by the leader UAV, and the inclusion error decreases rapidly and tends to be stable, it proves the efficiency and robustness of the developed control strategy in handling the collaborative operation of multiple UAV systems.

[0101] Beneficial technical effects of the present invention:

[0102] The reinforcement learning-based distributed robust inclusive control method for multiple unmanned harvesters provided by this invention offers enhanced robustness and adaptability. By designing a distributed robust inclusive control method for a multi-unmanned harvester system with dynamic uncertainty, the system's robustness to model uncertainty and external disturbances is significantly improved. This robustness ensures the unmanned harvester system can operate stably in complex and changing field operating environments, improving the reliability and success rate of operational tasks.

[0103] Improving the system's convergence speed and accuracy: Based on an adaptive evaluation design using reinforcement learning, the proposed method transforms the distributed robust containment control problem of a multi-UAV harvester system with dynamic uncertainty into an optimal containment control problem for its nominal plant. Distributed optimal control is achieved by solving an approximate solution to the coupled HJB equations. This approach not only ensures that the states of all following UAVs converge to the convex hull formed by the leader UAV, but also optimizes the performance function of each following UAV, thereby reducing unnecessary energy consumption and improving the system's energy efficiency.

[0104] Flexibility and Scalability: This technical solution is not limited to a specific communication topology or number of harvesters and can be flexibly applied to multi-robot systems of varying structures and scales. Furthermore, by constructing a critic neural network to approximate the HJB equation, the control algorithm is simplified and scalable, facilitating deployment and optimization in a wider range of application scenarios.

[0105] Promoting the development of multi-unmanned harvester systems: The proposal and implementation of this technical solution provides new ideas and methods for solving the distributed robust control problem of multi-unmanned harvester systems, promoting research and development in this field. It also provides useful reference and inspiration for distributed control problems in other related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 The present invention is a system flow chart of a preferred embodiment of the distributed robust control method for multiple unmanned harvesters based on reinforcement learning according to the present invention. DETAILED DESCRIPTION

[0107] In order to make the technical solution of the present invention more clear and specific to those skilled in the art, the present invention is further described in detail below with reference to embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0108] like Figure 1 As shown, the distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning provided in this embodiment includes the following steps:

[0109] Step 1: Build a communication topology using each unmanned harvester as a communication node;

[0110] Step 2: Multi-UAV harvester system modeling and problem definition;

[0111] Step 3: Construct the kinematic model of the leader and follower in the multi-UAV harvester system;

[0112] Step 4: Design a discount value function for the unmanned harvester and transform the distributed robust control into an optimal control problem for its nominal plant.

[0113] Step 5: Set up distributed robustness based on reinforcement learning and include control protocols;

[0114] Step 6: Construct a critic neural network to approximately solve the coupled Hamilton-Jacobi-Bellman equations;

[0115] Step 7: According to the proposed distributed robust control strategy, the inclusion errors of all follower unmanned harvesters are uniformly bounded.

[0116] Enhanced robustness and adaptability: By designing a distributed robust inclusion control approach for a multi-unmanned harvester system with dynamic uncertainty, the system's robustness to model uncertainty and external disturbances is significantly improved. This robustness ensures the unmanned harvester system can operate stably in complex and changing field environments, improving the reliability and success rate of operational tasks.

[0117] Improving the system's convergence speed and accuracy: Based on an adaptive evaluation design using reinforcement learning, the proposed method transforms the distributed robust containment control problem of a multi-UAV harvester system with dynamic uncertainty into an optimal containment control problem for its nominal plant. Distributed optimal control is achieved by solving an approximate solution to the coupled HJB equations. This approach not only ensures that the states of all following UAVs converge to the convex hull formed by the leader UAV, but also optimizes the performance function of each following UAV, thereby reducing unnecessary energy consumption and improving the system's energy efficiency.

[0118] Flexibility and Scalability: This technical solution is not limited to a specific communication topology or number of harvesters and can be flexibly applied to multi-robot systems of varying structures and scales. Furthermore, by constructing a critic neural network to approximate the HJB equation, the control algorithm is simplified and scalable, facilitating deployment and optimization in a wider range of application scenarios.

[0119] Promoting the development of multi-unmanned harvester systems: The proposal and implementation of this technical solution provides new ideas and methods for solving the distributed robust control problem of multi-unmanned harvester systems, promoting research and development in this field. It also provides useful reference and inspiration for distributed control problems in other related fields.

[0120] In this embodiment, step three also includes defining the inclusion error of following the unmanned harvester.

[0121] In this embodiment, step one also includes the following steps:

[0122] S11: Assume that a multi-UAV harvester system consists of M UAV harvester leaders and N UAV harvester followers;

[0123] S12: The set of followers and leaders are represented by S f and S l express, is the focus matrix of fixed payoffs from the mth UAV leader to the i-th UAV follower;

[0124] S13: If there is a connection between the mth leader and the nth follower, then m∈S l and otherwise,

[0125] S14: The undirected communication topology graph of the UAV harvester follower is represented as Λ g ={P,ξ,A}, where P = {p1,…,p N} is a set of nodes, ξ={(p i ,p j ):p i ,p j ∈P} is a set of edges, A=[a ij ] is the weighted adjacency matrix;

[0126] S15: If and only if the i-th unmanned harvester follower and the j-th unmanned harvester follower are directly connected, then (p i ,p j )∈ξ, and if (p i ,p j )∈ξ,then a ij >0; if Then a ij =0;

[0127] And for all i=1,…,N, then a ii =0;

[0128] S16: Let N i ={j:(p i ,p j )∈ξ,j≠i} is a neighbor set of the i-th follower, is the set of the ith follower and its neighbors, D=diag{v1,…,v N},in Λ g The degree matrix of , L = DA is the Laplace matrix.

[0129] In this embodiment, step 2 also includes the following steps:

[0130] S21: Construct and initialize the kinematic models of the leader and follower in a multi-UAV harvester system;

[0131] S22: Consider a multi-drone system with M drone leaders and N drone followers. The system kinematics of each drone follower is described as:

[0132]

[0133] Among them, the coordinate system XOY is the global ground inertial coordinate;

[0134] The agricultural machinery coordinate system is xoy;

[0135] The yaw angle of the agricultural machinery is φ;

[0136] The front wheel steering angle is δ;

[0137] The rear axle speed is v;

[0138] The wheelbase of the unmanned harvester is L;

[0139] The above kinematic model is regarded as a control system with control variable u = δ and state variable X = (x, y, φ), which can be expressed as:

[0140]

[0141] where X i =[x i ,y i ,u i ] T ,u i =δ i ;

[0142] The system kinematics of the mth unmanned harvester leader is given by:

[0143]

[0144] in and is a differentiable function.

[0145] In this embodiment, the finite set S = {s1, ..., s q The convex hull Co(S) of} is the smallest convex set containing all the points in S, that is:

[0146]

[0147] A distributed robust inclusion controller based on reinforcement learning is proposed to drive all following unmanned harvesters with dynamic uncertainty to converge to the convex hull spanned by the leader unmanned harvester, i.e.,

[0148] The nominal system of formula (2) can be expressed as:

[0149]

[0150] In this embodiment, the inclusion error of the following unmanned harvester is defined to measure the distance from the leading unmanned harvester group;

[0151] The inclusion error of the i-th following robot is defined as:

[0152]

[0153] Then, the kinematics including the error can be calculated as:

[0154]

[0155] In the formula

[0156] The inclusion error of the following UAV is defined as a key indicator, which measures the distance and position relationship between each following UAV and the leader UAV group;

[0157] The inclusion error reflects whether the follower UAV successfully converges and remains inside the convex hull spanned by the leader UAV.

[0158] In this embodiment, step three also includes designing a discount value function that follows the unmanned harvester and using Hamiltonian to solve the discounted optimal value function to obtain a coupled HJB equation, which is specifically:

[0159] The discount value function of the i-th following unmanned harvester is defined as:

[0160]

[0161] When 0<λ<1, the positive upper bound function Γ i (x i ) and the utility function U i (·) Designed to:

[0162]

[0163] where u (-i) ={u j |j∈N i} is the control input of the i-th neighbor following the unmanned harvester, and is a positive definite matrix;

[0164] The Hamiltonian of the i-th unmanned harvester is defined as:

[0165]

[0166] The discounted optimal value function of the i-th following unmanned harvester is:

[0167]

[0168] The HJB equation is satisfied:

[0169]

[0170] in is a set of acceptable controls. Then, the optimal inclusion control law is derived as:

[0171]

[0172] According to formula (8) and formula (9), we can get:

[0173]

[0174] In this embodiment, step 4 also includes converting the distributed robust inclusion control of the multi-unmanned harvester system into the optimal containment control of its nominal system;

[0175] Based on the obtained approximate optimal value function and approximate inclusion control law, the constraint conditions for all the inclusion errors following the unmanned harvester to be asymptotically stable or uniformly bounded are determined;

[0176] The equivalence of the problem transformation is proved by the following theorem;

[0177] set up is a continuously differentiable positive definite function, It's a K ∞ Class functions;

[0178] Consider a multi-unmanned harvester system with dynamic uncertainty, whose error kinematics is given by Equation (7), and the optimal control law is given by Equation (9). If Q ii satisfy;

[0179]

[0180] Then, it is guaranteed that the inclusion error of the i-th following unmanned harvester is asymptotically stable;

[0181] Considering the unmanned harvester system with dynamic uncertainty, the kinematics including the error given by Equation (7) and the weights of the critical neural network updated by (20), if Q ii satisfy;

[0182]

[0183] when When , the approximate inclusion control law (9) ensures that the inclusion error of each trailing robot is uniformly bounded, which means that all following UAVs enter the convex hull formed by the UAV leader.

[0184] In this embodiment, step five also includes defining the discounted optimal value function of the i-th following unmanned harvester as:

[0185]

[0186] in, is the ideal weight vector, is the activation function, h c is the number of hidden neurons, is the reconstruction error. Then, right The partial derivative of is given by:

[0187]

[0188] The approximate discounted value function is given by:

[0189]

[0190] in W ic The estimated value of , can be obtained similarly;

[0191]

[0192] According to equations (9) and (14), the optimal inclusion control law is rewritten as:

[0193]

[0194] Then, the approximate inclusion control law is given by:

[0195]

[0196] According to equations (10) and (18), the approximate Hamiltonian is:

[0197]

[0198] set up To calculate the weight estimation error, we use the gradient descent algorithm and design the weight update rule for evaluating the neural network as follows:

[0199]

[0200] In the formula α c >0 is the learning rate,

[0201] An adaptive critic design method based on reinforcement learning is combined with the approximation capability of neural networks to construct a critic neural network to learn and optimize control strategies online.

[0202] In this embodiment, a multi-unmanned harvester system including six unmanned harvesters is used in step 6 to verify the effectiveness of the developed distributed robust inclusion control scheme based on reinforcement learning;

[0203] The system consists of three leading and three following unmanned harvesters, which exchange information based on a predefined communication topology. By constructing a detailed system kinematic model and introducing dynamic uncertainty to simulate a real-world operating environment, the performance of the unmanned harvester system's control strategy was tested.

[0204] During the simulation, we focus on the dynamic changes of the included error of the unmanned harvester to verify whether it can converge to a small zero region as expected;

[0205] If, in the presence of system modeling uncertainty and external disturbances, the trajectories of all following UAVs successfully enter the convex hull formed by the leader UAV, and the inclusion error decreases rapidly and tends to be stable, it proves the efficiency and robustness of the developed control strategy in handling the collaborative operation of multiple UAV systems.

[0206] The above is only a further embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and concepts of the present invention within the scope disclosed by the present invention, which fall within the scope of protection of the present invention.

Claims

1. A distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning, characterized by: The steps include: Step 1: Build a communication topology by using each unmanned harvester as a communication node; Step 2: Multi-unmanned harvester system modeling and problem definition; Step 3: Construct the kinematic model of the leader and follower in the multi-unmanned harvester system; Step 4: Design the discount value function of the unmanned harvester and transform the distributed robust control into the optimal control problem of its nominal object; Step 5: Set up a distributed robust control protocol based on reinforcement learning; Step 6: Construct a critic neural network to approximately solve the coupled Hamilton-Jacobi-Bellman equations; Step 7: According to the proposed distributed robust control strategy, the inclusion errors of all follower unmanned harvesters are uniformly and ultimately bounded.

2. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 1 is characterized in that: Step three also includes defining the included error for following the unmanned harvester.

3. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 2 is characterized in that: Step one also includes the following steps: S11: Assume that a multi-UAV harvester system includes M UAV harvester leaders and N UAV harvester followers; S12: The set of followers and leaders are represented by S f and S l express, is the focus matrix of fixed payoffs from the mth UAV leader to the i-th UAV follower; S13: If there is a connection between the mth leader and the nth follower, then m∈S l and otherwise, S14: The undirected communication topology graph of the unmanned harvester follower is represented by Λ g = {P,ξ,A}, where P = {p1,…,p N } is a set of nodes, ξ={(p i ,p j ):p i ,p j ∈P} is a set of edges, A=[a ij ] is the weighted adjacency matrix; S15: If and only if the i-th unmanned harvester follower and the j-th unmanned harvester follower are directly connected, then (p i ,p j )∈ξ, and if (p i ,p j )∈ξ, then a ij >0; if Then a ij =0; And for all i=1,…,N, then a ii =0; S16: Let N i ={j:(p i ,p j )∈ξ,j≠i} is a neighbor set of the ith follower, is the set of the ith follower and its neighbors, D = diag{v1,…,v N },in For g The degree matrix of , L=DA is the Laplace matrix.

4. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 3 is characterized in that: Step 2 also includes the following steps: S21: Construct and initialize the kinematic model of the leader and follower in the multi-unmanned harvester system; S22: Consider a multi-drone harvester system with M drone leaders and N drone followers. The system kinematics of each drone follower is described as: Among them, the coordinate system XOY is the global ground inertial coordinate; The agricultural machinery coordinate system is xoy; The yaw angle of the agricultural machinery is φ; The front wheel steering angle is δ; The speed of the rear axle is v; The wheelbase of the unmanned harvester is L; The above kinematic model is regarded as a control system with control quantity u = δ and state quantity X = (x, y, φ), which can be expressed as: among them X i =[x i ,y i ,u i ] T ,u i =d i ; The system kinematics of the mth unmanned harvester leader is given by: in and is a differentiable function.

5. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 4 is characterized in that: Finite set S = {s1,…,s q The convex hull Co(S) of} is the smallest convex set containing all the points in S, that is: A distributed robust inclusion controller based on reinforcement learning is proposed to drive all follower harvesters with dynamic uncertainty to converge to the convex hull spanned by the leader harvester, i.e., The nominal system of formula (2) can be expressed as:

6. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 5 is characterized in that: Define the inclusion error of the follower unmanned harvester to measure the distance from the leader unmanned harvester group; Define the inclusion error of the i-th following robot as: Then, the kinematics including the error can be calculated as: In the formula The inclusion error of the following UAV harvesters is defined as a key indicator, which measures the distance and position relationship between each following UAV harvester and the group of leading UAV harvesters; The inclusion error reflects whether the follower harvester successfully converges and remains inside the convex hull spanned by the leader harvester.

7. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 6 is characterized in that: Step three also includes designing a discount value function that follows the unmanned harvester and using the Hamiltonian to solve the discounted optimal value function to obtain the coupled HJB equation, which is specifically: The discount value function of the i-th following unmanned harvester is defined as: When 0<λ<1, the positive upper bound function Γ i (x i ) and the utility function U i (·) Designed to: where u (-i) = {u j |j∈N i } is the control input of the neighbor of the i-th unmanned harvester, and is a positive definite matrix; The Hamiltonian of the i-th unmanned harvester is defined as: The discounted optimal value function of the i-th following unmanned harvester is: The HJB equation is satisfied: in is a set of acceptable controls. Then, the optimal inclusion control law is derived as: According to equations (8) and (9), we can obtain:

8. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 7 is characterized in that: Step 4 also includes converting the distributed robust containment control of the multi-unmanned harvester system into the optimal containment control of its nominal system; According to the obtained approximate optimal value function and approximate inclusion control law, the constraint conditions that the inclusion errors of all following unmanned harvesters are asymptotically stable or uniformly bounded are determined; The equivalence of the problem transformation is proved by the following theorem; set up is a continuously differentiable positive definite function, It is a K ∞ Class functions; Consider a multi-unmanned harvester system with dynamic uncertainty. Its error kinematics is given by Equation (7), and the optimal control law is given by Equation (9). If Q ii satisfy; Then, it is guaranteed that the included error of the i-th following unmanned harvester is asymptotically stable; Considering the unmanned harvester system with dynamic uncertainty, including the kinematics error given by equation (7), the weights of the critical neural network updated by (20), if Q ii satisfy; when When , the approximate containment control law (9) ensures that the containment error of each following robot is uniformly bounded, which means that all following unmanned harvesters enter the convex hull formed by the leader unmanned harvester.

9. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 8 is characterized in that: Step five also includes defining the discounted optimal value function of the i-th following unmanned harvester as: in, is the ideal weight vector, is the activation function, h c is the number of hidden neurons, is the reconstruction error. Then, right The partial derivative of is given by: The approximate discounted value function is given by: in W ic The estimated value of , similarly, can be obtained; According to equations (9) and (14), the optimal inclusion control law is rewritten as: Then, the approximate inclusion control law is given by: According to equations (10) and (18), the approximate Hamiltonian is: set up To estimate the weight error, a gradient descent algorithm is used to design and evaluate the weight update rule of the neural network: In the formula α c >0 is the learning rate, An adaptive critic design method based on reinforcement learning is combined with the approximation capability of neural networks to construct a critic neural network to learn and optimize control strategies online.

10. The distributed robust inclusion control method for multiple unmanned harvesters based on reinforcement learning according to claim 9, characterized in that: In step six, a multi-unmanned harvester system consisting of six unmanned harvesters is used to verify the effectiveness of the developed distributed robust inclusion control scheme based on reinforcement learning; The system includes three leading unmanned harvesters and three following unmanned harvesters, which exchange information based on a predefined communication topology. By building a detailed system kinematic model and introducing dynamic uncertainty to simulate the real working environment, the performance of the control strategy of the unmanned harvester system is tested. During the simulation, we focus on the dynamic changes of the included error of the unmanned harvester and verify whether it can converge to a small zero region as expected; If, in the presence of system modeling uncertainty and external disturbances, the trajectories of all following unmanned harvesters successfully enter the convex hull formed by the leader unmanned harvester, and the inclusion error decreases rapidly and tends to stabilize, it proves the efficiency and robustness of the developed control strategy in dealing with the collaborative operation of multiple unmanned harvester systems.

Images