Heterogeneous unmanned aerial vehicle layered formation control method and system based on Shapril value and coupled SAC

Abstract

The invention discloses a heterogeneous unmanned aerial vehicle layered formation control method based on a Shapley value and a coupling SAC, and the method comprises the steps: firstly determining the position and attitude information of a tracking unmanned aerial vehicle and a target unmanned aerial vehicle, and then determining a trajectory control signal through a position control module based on the position and attitude information, so as to drive the tracking unmanned aerial vehicle to track the target unmanned aerial vehicle. And determining an attitude control signal through an attitude control module so as to drive the tracking unmanned aerial vehicle to track an expected yaw trajectory, finally performing cooperative tracking control on the tracking unmanned aerial vehicle based on the trajectory control signal and the attitude control signal, calculating a tracking error variable, and judging whether the tracking error variable meets a preset condition, and if so, taking the current track control signal and the attitude control signal as an optimal control strategy, otherwise, updating the track control signal and the attitude control signal until a preset condition is met. According to the method, the problem that a heterogeneous unmanned aerial vehicle group in which part of unmanned aerial vehicles do not have the target detection capability keeps layered formation tracking of target points in a complex environment is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicles, and in particular to a heterogeneous unmanned aerial vehicle hierarchical formation control method and system based on Shapley value and coupled SAC. Background Art

[0002] With the widespread use of drones in military and civilian applications, the difficulty and complexity of the tasks they can perform are constantly increasing. For some tasks with higher difficulty and complexity, it is often necessary to rely on multiple drones to work together. Multiple drones forming a formation can expand the field of view and increase the reconnaissance and search range. Based on this, multi-drone formation flight control technology is crucial.

[0003] The existing multi-UAV formation tracking strategy relies heavily on the network topology of the overall system. Therefore, when the communication range between UAVs cannot meet the control requirements, the performance of the strategy tends to drop significantly. In addition, the existing tracking strategy has high requirements on the perception ability of network nodes. Once some nodes fail and lose their detection function, or they do not have the ability to detect targets, this will inevitably have a negative impact on the collaborative tracking performance of the entire UAV formation.

[0004] At the same time, most current drone swarm formation tracking control only considers one type of drone, which has poor adaptability when facing complex and changing mission environments, has limited fault tolerance of the entire homogeneous group, and has a single advantage in collaborative scenarios. As application scenarios become more and more complex, a single type of drone can no longer meet mission requirements.

[0005] In addition, dynamic nonlinearity is common in multi-UAV cooperative control systems. Due to the complexity of the system itself or the external environment, the mathematical model of the multi-UAV control system is often unable to accurately describe the dynamics of the actual system. The control system cannot use accurate state information, making the control strategy unstable, resulting in system performance degradation, stability destruction, and even formation failure.

[0006] Although there are some multi-UAV cooperative control methods based on traditional multi-agent reinforcement learning and artificial potential field methods, these methods have the problems of difficult convergence and chaotic formation topology. When the system environment becomes complex, multi-agent reinforcement learning algorithms and artificial potential field methods are difficult to deal with the contradictions between different factors, resulting in poor formation tracking effects, and even collisions, losses, and formation failures. Summary of the invention

[0007] In view of some or all of the problems in the prior art, the first aspect of the present invention provides a heterogeneous UAV hierarchical formation control method based on Shapley value and coupled SAC, comprising:

[0008] Confirm the position and posture of the tracking drone and the target drone, as well as their relative position information;

[0009] Based on the position and attitude information, a trajectory control signal is determined by a position control module to drive the tracking drone to track the target drone and maintain a desired formation;

[0010] Determining a posture control signal through a posture control module to drive the tracking drone to track a desired yaw trajectory;

[0011] Performing collaborative tracking control on the UAV based on the trajectory control signal and the attitude control signal, and calculating a tracking error variable; and

[0012] It is determined whether the tracking error variable meets the preset conditions. If so, the current trajectory control signal and attitude control signal are used as the optimal control strategy. Otherwise, the trajectory control signal and attitude control signal are updated until the preset conditions are met.

[0013] Furthermore, the heterogeneous UAV hierarchical formation control method further includes:

[0014] Constructing a dynamic model of a UAV, wherein the UAV is a quad-rotor UAV; and

[0015] An undirected graph is established to describe the interaction relationship of multiple UAV systems.

[0016] Furthermore, the desired formation includes a first drone group in an outer layer, a second drone group in an inner layer, and a target drone in the center, wherein the first drone group does not have a target detection function, and the second drone group has a target detection function.

[0017] Further, based on the position and posture information, determining the trajectory control signal by the position control module includes:

[0018] A multi-agent reinforcement learning method that combines artificial potential field method and Shapley value is used to determine the trajectory control signal of the UAV.

[0019] Furthermore, for each UAV, the determination of its trajectory control signal includes:

[0020] According to the position information of the homogeneous neighbor drones, heterogeneous neighbor drones and the target drone of the drone, respectively, the first, second and third action signals are determined through the Actor submodule;

[0021] Determining a fourth action signal based on the position information and the first, second, and third action signals;

[0022] Based on the first, second and third action signals, determining a first auxiliary control signal through a homogeneous subgroup interaction potential field, a target interaction potential field and a heterogeneous subgroup interaction potential field; and

[0023] Based on the first auxiliary control signal, a trajectory control signal is determined by a neural network adaptive sliding mode position control algorithm.

[0024] Furthermore, the determination of the trajectory control signal also includes:

[0025] The marginal contribution Shapley Q value function is adopted to evaluate the Actor submodule based on the fourth action signal and the position information, and perform iterative update.

[0026] Furthermore, for each UAV, the determination of its attitude control signal includes:

[0027] Determining a roll angle and a pitch angle of the virtual drone based on the trajectory control signal;

[0028] According to the roll angle and the pitch angle, a second auxiliary control signal is generated by using a PD control algorithm; and

[0029] Based on the second auxiliary control signal, a posture control signal is determined by a neural network adaptive sliding mode position control algorithm.

[0030] Based on the control method as described above, the second aspect of the present invention provides a heterogeneous UAV hierarchical formation control system based on Shapley value and coupled SAC, comprising:

[0031] A position control module, which is used to determine the trajectory control signal of the UAV according to the position and posture of the tracking UAV and the target UAV;

[0032] an attitude control module, which is used to determine an attitude control signal of the UAV according to the trajectory control signal and the attitude; and

[0033] A feedback module is used to calculate the tracking error variable to perform feedback iteration of the control signal.

[0034] Furthermore, the position control module includes:

[0035] An Actor submodule, comprising a first, a second, a third and a fourth sub-Actor, wherein the first sub-Actor is used to generate a first action signal according to the position information of a homogeneous neighboring UAV, the second sub-Actor is used to generate a second action signal according to the position information of a heterogeneous neighboring UAV, the third sub-Actor is used to generate a third action signal according to the position information of a target UAV, and the fourth sub-Actor is used to determine a fourth action signal according to the first, second, third action signals and the position information;

[0036] A critic submodule, configured to generate a marginal contribution Shapley Q value based on the fourth action signal and the position information;

[0037] an interaction potential field, comprising a homogeneous subgroup interaction potential field, a heterogeneous subgroup interaction potential field and a target interaction potential field, wherein the interaction potential field is used to generate a first auxiliary control signal according to the first, second and third action signals; and

[0038] A calculation submodule adopts a neural network adaptive sliding mode position control algorithm to determine a trajectory control signal based on the first auxiliary control signal.

[0039] Furthermore, the first, second, and third sub-actors all adopt LSTM networks, and the fourth sub-actor and the critic sub-module all adopt FC networks.

[0040] The present invention provides a heterogeneous UAV hierarchical formation control method and system based on Shapley value and coupled SAC. First, graph theory is used to represent the relationship between UAVs and UAV neighbors, an undirected graph is established to represent the communication relationship between UAVs, and an artificial potential field method is combined with a multi-agent reinforcement learning technology that integrates Shapley value of game theory. A behavior-coupled multi-agent soft actor critic hierarchical heterogeneous multi-UAV cooperative tracking control method based on Shapley value is proposed, which solves the formation maintenance and anti-collision problems under the condition that some UAV groups do not have the ability to detect targets, and the communication maintenance difficulties under the condition of different communication distances between heterogeneous subgroups, and enhances the tracking efficiency of heterogeneous multi-UAV cooperative formations, so that it can operate stably in complex and uncertain environments. Secondly, it adopts a neural network adaptive sliding mode control method, uses a radial basis neural network to fit the unknown dynamic function of the system, and is used to compensate for the auxiliary control signal, thereby designing a controller, and designing the adaptive update law of the network weight according to the Lyapunov method. Therefore, the controller design is less dependent on the accurate system dynamics model, so that the system no longer needs an accurate system dynamics model, which helps the system adapt to the dynamic changes of the environment and effectively improves the robustness of the system. The control method and system are based on the Shapley value method, using a centralized training and distributed execution implementation architecture, allocating the overall joint action value to the action value of each drone according to the marginal contribution rate of the agent member to the group, determining the average value of the marginal benefit created by each drone member in the group, solving the problem of lazy agents, promoting the development of drone group cooperative behavior and the improvement of decision-making level, improving the overall performance of the system, and enhancing the adaptability and robustness of multi-drone groups in complex dynamic environments. The improved multi-agent soft actor critic algorithm uses an actor network architecture that is first decomposed and then coupled, and uses the output action of the decomposed sub-actor network as the layered interactive potential field strength of the artificial potential field method to generate more refined and accurate auxiliary control signals, and then couples the actions of the sub-actor network to form the final action for evaluation, which helps the drone group to generate higher quality decisions, enhances the dynamic path planning ability of the artificial potential field method, and enhances the multi-intelligence reinforcement learning to deal with the contradiction between different reward functions, so that the algorithm converges. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] To further illustrate the above and other advantages and features of various embodiments of the present invention, a more specific description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof. In the accompanying drawings, for clarity, identical or corresponding parts will be represented by identical or similar reference numerals.

[0042] Figure 1A schematic flow chart of a method for controlling a heterogeneous UAV layered formation based on Shapley value and coupled SAC according to an embodiment of the present invention is shown;

[0043] Figure 2 A schematic diagram of the structure of a quad-rotor drone according to an embodiment of the present invention is shown;

[0044] Figure 3 A schematic diagram of an instantiated smooth potential function with a finite cutoff value showing one embodiment of the present invention;

[0045] Figure 4 A schematic diagram showing the structure of a heterogeneous UAV hierarchical formation control system based on Shapley value and coupled SAC according to an embodiment of the present invention; and

[0046] Figure 5a and 5b A schematic diagram showing the initial position and ideal formation of a target formation tracked by a heterogeneous UAV layered formation control method based on Shapley value and coupled SAC according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0047] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that various embodiments can be implemented without one or more specific details or with other replacement and / or additional methods, materials or components. In other cases, well-known structures, materials or operations are not shown or described in detail to avoid blurring the inventive point of the present invention. Similarly, for the purpose of explanation, specific quantities, materials and configurations are set forth to provide a comprehensive understanding of embodiments of the present invention. However, the present invention is not limited to these specific details. In addition, it should be understood that the various embodiments shown in the drawings are illustrative representations and are not necessarily drawn in correct proportions.

[0048] In this specification, reference to "one embodiment" or "the embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present invention. The phrase "in one embodiment" appearing in various places in this specification does not necessarily all refer to the same embodiment.

[0049] It should be noted that the embodiments of the present invention describe the process steps in a specific order, but this is only for the purpose of illustrating the specific embodiment, rather than limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to the adjustment of the process.

[0050] In the present invention, the drones include drones with a target detection function and drones without a target detection function. The homogeneous neighbor drones of any drone refer to drones that are adjacent to the drone and have the same expected target detection function, and the heterogeneous neighbor drones refer to drones that are adjacent to the drone and have different expected target detection functions. For example, for a drone with a target detection function, its homogeneous neighbor drones refer to drones that are adjacent to the drone and have the expected target detection function, and its heterogeneous neighbor drones refer to drones that are adjacent to the drone and do not have the expected target detection function.

[0051] In order to realize heterogeneous multi-UAV formation tracking of moving targets, the present invention uses graph theory to construct a multi-UAV formation tracking scenario model. Then, considering the internal and external environment of the UAV, the UAV is modeled based on the position subsystem and attitude subsystem.

[0052] For the position subsystem, the expected relative position information of each individual drone, its neighbor drone and the tracking target is used to construct the potential function. Using the behavior-coupled multi-agent soft actor critic hierarchical heterogeneous multi-UAV collaborative tracking control method based on Shapley value proposed by the present invention, the behavior-coupled multi-agent soft actor critic algorithm based on Shapley value is used to adjust the potential field strength of the interactive potential field, construct the formation tracking auxiliary control signal, and then the neural network adaptive sliding mode control algorithm is used to solve the error caused by system and environmental disturbances to the position control, and the position controller is designed.

[0053] For the attitude subsystem, the error variable is constructed according to the set expected yaw angle and the yaw angle of the virtual UAV, and the PD control algorithm is used to design the attitude auxiliary control signal. Then, the neural network adaptive sliding mode control algorithm is used to solve the error caused by system and environmental disturbances to the attitude control, and the attitude controller is designed.

[0054] In summary, the position control of the quadrotor drone affects the effect and dynamic response of attitude control. At the same time, attitude control is the basis for achieving precise position control. The two work closely together through coordinated feedback control to achieve flight missions.

[0055] The technical solution of the present invention is further described below in conjunction with the accompanying drawings of the embodiments.

[0056] Figure 1 A flow chart of a heterogeneous UAV layered formation control method based on Shapley value and coupled SAC according to an embodiment of the present invention is shown.

[0057] like Figure 1 As shown, a heterogeneous UAV hierarchical formation control method based on Shapley value and coupled SAC includes:

[0058] First, in step 101, the initial position and attitude information is confirmed. The position and attitude of the tracking drone and the target drone, as well as their relative position information, are confirmed. In one embodiment of the present invention, in order to realize the formation control of a multi-quadrotor heterogeneous drone group, dynamic modeling is performed and the information exchange relationship is described. In the embodiment of the present invention, the drone is a quadrotor drone, and its structure is as follows: Figure 2 The rotor UAV is a highly nonlinear underactuated system, where the control quantity is less than the system state variables, and the change of motion state is achieved through the control input generated by the four rotors.

[0059] Based on this, according to the Lagrange equation, the dynamic model of the i-th quadrotor drone is considered as follows:

[0060]

[0061] in:

[0062] i∈N={N R ,N F}={v1,v2,...,v N} is the total set of drones, where N R is a collection of drones with target detection capabilities, N F A collection of drones that do not have the ability to detect targets;

[0063] are pitch angle, roll angle and yaw angle respectively;

[0064] [x i ,y i ,z i ] is the position of the center of mass of the quadrotor drone in the inertial coordinate system;

[0065] m i is the total load mass;

[0066] [I i,x ,I i,y ,I i,z ] is the inertial rotation matrix;

[0067] g is the acceleration due to gravity;

[0068] [U i,1 ,U i,2 ,U i,3 ,U i,4 ] is the control input of the four rotors;

[0069] The external interferences on the three position channels and the three attitude channels are respectively generated by environmental uncertainties such as wind speed effects and system uncertainties; and

[0070] [τ x,i ,τ y,i ,τ z,i ] are the control components on the three position channels of the quadrotor drone, where:

[0071]

[0072] In one embodiment of the present invention, in order to describe the communication relationship between N quadrotor drones, the interaction relationship of the multi-drone system is described by establishing an undirected graph as G = (V, E, A), where each drone node is described as V = N = {v1, v2, ..., v N}, the interaction relationship between two drone nodes is described by the edge set E(G)∈{(v i ,v j )|v i ,v j ∈V(G),i≠j}, and define a ij is the i-th row and j-th column element of the cluster system spatial topology adjacency matrix A(G), a ij ≥0 indicates that the edge (v i ,v j )’s weight:

[0073]

[0074] where r c is the communication distance between the two machines, p i =[x i ,y i ,z i ] is the three-dimensional coordinate position of any UAV i.

[0075] It should be understood that for any UAV i that does not have the target detection function, there is always a connection path to a UAV with the target detection function, otherwise the UAV i cannot obtain the target UAV information and the formation fails.

[0076] Next, in step 102, a trajectory control signal is determined. Based on the position and attitude information, a trajectory control signal is determined by a position control module to drive the tracking drone to track the target drone and maintain the desired formation. In one embodiment of the present invention, the desired formation includes a first drone group in the outer layer, a second drone group in the inner layer, and a target drone in the center, wherein the first drone group does not have a target detection function, and the second drone group has a target detection function.

[0077] In one embodiment of the present invention, a potential function is constructed based on the system states and expected parameters of drone i and its neighboring drones. In order to ensure that multiple drones form a regular polygon double-layer formation to track the target g, the expected distance between individual drone i and its neighboring drones is defined as:

[0078]

[0079] in represents the expected distance between individual i of the drone group with target detection function and its neighboring individual drone j in the same group, represents the expected distance between individual i of the drone group with non-local detection function and its similar individual neighbor drone j; r R represents the expected distance between the drone group with target detection function and the target, r F Indicates the expected distance between the drone swarm and the target without the target detection function.

[0080] In one embodiment of the present invention, from the perspective of heterogeneous drone nodes, due to mission requirements or environmental interference, a certain type of drone group does not have the ability to detect the target and needs to rely on neighboring nodes to form a stable encirclement. To avoid obstruction in detecting the target, a drone group that can effectively detect the target is formed in the inner layer, and a drone group that cannot detect the target is formed in the outer layer. The distance between the two types of drone groups and the target satisfies r R <r F Double-layer encirclement tracking target.

[0081] In one embodiment of the present invention, a smooth attraction / repulsion pair function ψ is introduced α (z), used to generate control signals to enable the drone swarm to form a desired formation, is defined as follows:

[0082]

[0083] where z is the distance between the two particles, d α is the minimum point of the potential function. Figure 3 A schematic diagram of an instantiated smooth pair potential function with a finite cutoff value of an embodiment of the present invention is shown. Based on this, the interaction potential field of a homogeneous cluster is defined as and Defining the interaction potential field of heterogeneous clusters and The interaction potential field with the target is and Then instantiate the potential function:

[0084] make The instantiation potential function parameters are i,j∈N R , The instantiation potential function parameters are i,j∈N F , r R With r F is the communication distance within the same group of two types of UAVs;

[0085] make The instantiation potential function parameters are i∈N R ,j∈N F ,make The instantiation potential function parameters are i∈N F ,j∈N R ,in is the communication distance between two types of heterogeneous drone groups, Expressed as the expected spacing between the two types of drone groups; and

[0086] make The instantiation potential function parameters are (r, d) = (r det ,r R ),make To instantiate the potential function parameters are (r, d) = (r det ,r F ), where r det is the distance at which the target is normally detected by the individual drone. In one embodiment of the present invention, r / d=1.2.

[0087] The construction of the potential function is used to describe the interaction force between two agents. The potential function can be used to calculate the attraction and repulsion between the agents. By adjusting the positions of the agents, the total potential energy of the system can be minimized, so that the multi-agent system can achieve a stable formation structure and realize anti-collision function.

[0088] In one embodiment of the present invention, a multi-agent reinforcement learning method that integrates the artificial potential field method and the Shapley value is used to determine the trajectory control signal of the drone. Specifically, for each drone, first, the first, second, and third action signals are determined through the Actor submodule according to the position information of the homogeneous neighbor drones, heterogeneous neighbor drones, and target drones of the drone, and then based on the first, second, and third action signals, the first auxiliary control signal is determined through the homogeneous subgroup interaction potential field, the target interaction potential field, and the heterogeneous subgroup interaction potential field, and based on the first auxiliary control signal, the trajectory control signal is determined through the neural network adaptive sliding mode position control algorithm. In addition, based on the position information and the first, second, and third action signals, the fourth action signal is determined, and the marginal contribution Shapley Q value function is used to evaluate and iteratively update the Actor submodule based on the fourth action signal and the position information.

[0089] In one embodiment of the present invention, the position subsystem is defined as:

[0090]

[0091] in,

[0092] Based on this, the virtual drone position model is constructed as:

[0093]

[0094] where x i,s ,y i,s ,z i,s is the position signal of the virtual drone of the i-th drone, and is the auxiliary control signal to be designed.

[0095] Affected by the state constraints of position, speed, acceleration, heading angle, heading angular velocity, and heading angular acceleration in the drone model, there is a positive constant B x , B y , B z , B ψ , so that In order to avoid collisions between drones and their neighbors and targets, the safe collision avoidance distance between individual drones in the tracking drone group is set to ρ1, and the safe collision avoidance distance with the target is set to ρ2.

[0096] Based on this, a behavior-coupled multi-agent soft actor critic hierarchical heterogeneous multi-UAV cooperative tracking control method based on Shapley value is constructed for heterogeneous multi-UAV cooperative formation tracking control to efficiently achieve tracking tasks. In one embodiment of the present invention, the design of the position control module includes:

[0097] First, the marginal contribution Shapley Q value function is constructed. Let N be a collective alliance composed of multiple drones, and the individual drones in the multi-drone system form an arbitrary alliance C. The Shapley value in the cooperative game field is introduced, and the benefits are distributed according to the marginal contribution rate of the members to the alliance. The marginal contribution value of drone i is:

[0098]

[0099] where a C =[a j |j∈C] is the joint action of any UAV in the coalition C, μ C is the strategy set of alliance C.

[0100] Then the marginal contribution Shapley Q value function of UAV i is:

[0101]

[0102] Since the amount of computation required to traverse alliance C is huge, the sub-alliances are processed to obtain the approximate Shapley Q value:

[0103]

[0104] Where M represents the number of sampling times; C k ∈{C1,C2,…,C M} is the sampled sub-coalition.

[0105] The behavior-coupled multi-agent soft actor critic algorithm based on Shapley value adopts the actor-critic architecture to construct a global value function Q(s,a) for evaluating the state s = [o 1 ,…,o ∣N∣ ], execute the action set a of all agents a = [a 1 ,…,a ∣N∣ ], where o i is the incomplete observation of the i-th agent, and the individual action a i is the deterministic action function π(·|o i ). Using a deep neural network to fit the relevant functions, each agent i has a deterministic policy network and a value network Q(s,a;ω i ). Each agent i in the behavior-coupled multi-agent soft actor critic algorithm based on Shapley value uses two deep neural Shapley soft action value networks (with parameters ω1 and ω2, respectively), and takes the network that outputs the minimum action value as the target network to solve the overestimation problem. Then the value network Q(s,a;ω i ) is:

[0106]

[0107] By introducing the concept of Shapley value in game theory, the search for the optimal solution is transformed into the search for an equilibrium solution of a strategy combination that satisfies all agents. Then the strategy learning of all agents has a common objective function:

[0108]

[0109] in represents the randomness of the strategy π in state s, then the strategy learning can be described as the following optimization problem:

[0110]

[0111] In one embodiment of the present invention, different entropies are required at different training stages. In order to automatically adjust the entropy regularization term, a loss function of α is obtained:

[0112]

[0113] When the entropy of the policy is lower than When the entropy of the strategy is higher than the target value, minimizing the loss function L(α) will increase the value of α and enhance exploration. When , minimizing the loss function L(α) will reduce the value of α, making the strategy training more focused on improving the target value.

[0114] The introduction of Shapley value can solve the problem of lazy agents. At the same time, the introduction of entropy regularization term will further enhance the exploration of agent actions during training, strengthen the development of cooperative behavior and improve the decision-making level of the system.

[0115] Next, construct the state and action space. A Markov decision process usually consists of a tuple In the formation tracking scenario limited by communication, each UAV cannot perceive all the information of the environment, and each agent has a local observation o i .

[0116] In an embodiment of the present invention, the observation consists of three parts at the current time step t: Define the two-dimensional plane coordinate ζ = {x, y}, then the first part is the distance of drone i relative to target g The second part is the state information of UAV i relative to its isomorphic neighbors. where i,j∈N R / N F ; The third part is the status information of UAV i relative to its heterogeneous neighbors where i∈N F / N R , j∈N R / N F ;

[0117] In order to achieve continuous action and precise control, in one embodiment of the present invention, a continuous action space is used. For the continuous action space, the strategy of the behavior-coupled multi-agent soft actor critic algorithm based on Shapley value outputs the mean and standard deviation of the Gaussian distribution, but the sampled actions in the parameterized probability distribution are not differentiable. The reparameterization technique is used to transform the sampling process into a deterministic function. The mean μ and standard deviation σ of the action distribution output by the policy network are sampled from the unit Gaussian distribution. The noise ∈~N(0,1) is then sampled. The reparameterized action is:

[0118] a t =fθ (∈ t ;s t )=a=μ+σ⊙∈;

[0119] Next, design a reward function. The reward function can guide the agent's strategy learning, adjust the strategy through the reward of environmental feedback as an immediate evaluation of the behavior, and promote the agent to achieve the goal. In one embodiment of the present invention, the reward function achieves three goals: keep tracking the target g, do not collide with neighbors and targets, and keep the desired distance with neighbors and targets to form the desired formation. Based on this, the reward function is as follows:

[0120] Approaching the target: The reward function in this part drives UAV i to approach the dynamic target UAV and maintain the desired distance, so that the relative position error between the UAV and the target is Then the negative reward for drone i approaching the target at time step t is set as:

[0121]

[0122] Avoid collision: This part of the reward function drives UAV i to maintain a safe distance from neighbors and targets to avoid collisions. The negative reward for avoiding collisions for UAV i at time step t is set as:

[0123] r2=-r g -r n ,

[0124] where r n Negative reward function to avoid collision between drone i and its neighbors:

[0125]

[0126] r g Negative reward function to avoid collision between drone i and target:

[0127]

[0128] Formation maintenance: This part of the reward function drives UAV i to maintain the desired distance with the homogeneous and heterogeneous groups, thereby forming the desired formation. The negative reward for UAV i’s formation maintenance at time step t is set as:

[0129] r3=-r hm -r ht ,

[0130] where r hm is the negative reward function maintained between drone i and the homogeneous group formation, and the relative position error between drone i and its homogeneous neighbors is but:

[0131]

[0132] where r ht is the negative reward function maintained between UAV i and the heterogeneous fleet formation, and the relative position error between the UAV and its homogeneous neighbors is but:

[0133] as well as

[0134] Quick completion: Let ρ3 be the threshold of the reward for the formation to be designed. When the value of r3 is greater than or equal to ρ3, this part of the reward function drives the UAV system to quickly form the target formation. Design the negative reward function:

[0135]

[0136] In summary, the entire reward function of drone i at the current time step t can be derived as:

[0137] r i,t =r1+r2+r3+r4;

[0138] Next, the behavior is coupled with the soft actor critic algorithm. The drone state information s is divided into homogeneous neighbor drone position information, heterogeneous neighbor drone position information, and target drone position information. The homogeneous neighbor drone position information is used to adjust the interaction potential field strength between homogeneous drones, set as Ω∈{R,F}, the location information of heterogeneous neighboring UAVs is used to adjust the interaction potential field strength between heterogeneous UAVs, which is set as The target UAV position information is used to adjust the interaction potential field strength between the tracking UAV and the target UAV, which is set as s g .

[0139] In one embodiment of the present invention, the behavior-coupled multi-agent soft actor critic uses three different sub-actor networks to process these three types of information respectively and output a g Three actions, where Ω∈{R,F}, use the above three actions as the interaction potential field strength between the current drone and its homogeneous neighbors, heterogeneous neighbors and targets to achieve more refined and accurate drone position control, and then form a new vector with these three actions and state information s as the input of the fourth sub-actor network to generate a final action a, which is used as the input of the critic network for value evaluation and guiding the actor network.

[0140] Compared with the fully connected layer that only focuses on the current time step information, the introduction of the LSTM network helps the drone better understand the information of neighboring drones and target drones. At the same time, the LSTM network maintains a stable gradient through a special gate structure, reducing the impact of the RNN gradient disappearance and gradient explosion problems. At the same time, in order to reduce the computational complexity of the network, the sequence length of the RNN is set to 4, and only the state information of the first four time steps is processed. The first three sub-actors are set to use the LSTM network, and the fourth sub-actor uses the fully connected layer. Setting different network structures for different training purposes can reduce the complexity of the model and enable the actor network to make more accurate decisions;

[0141] Next, we design the formation tracking auxiliary control signal. According to the Shapley value-based behavior-coupled multi-agent soft actor critic hierarchical heterogeneous multi-UAV collaborative tracking control method, we design the auxiliary control signal for the UAV i∈N with the ability to detect the target. R Design auxiliary control signal:

[0142]

[0143] For the undetectable target drone i∈N F Design auxiliary control signal:

[0144]

[0145] Design height tracking auxiliary control signal: where i∈N; and

[0146] Finally, the trajectory control signal is determined by designing a neural network adaptive sliding mode control strategy. Let η = {x, y, z} and the sliding mode function is designed as:

[0147]

[0148] in c η is a positive constant to be designed.

[0149] Derivative of the sliding mode function:

[0150]

[0151] From the analysis, we can see that is an unknown nonlinear function, which includes system uncertainty and environmental interference. Using radial basis neural network to approximate the unknown nonlinear function, we can get:

[0152]

[0153] in, is the ideal weight of the neural network, is the basis function vector,

[0154] is the approximation error.

[0155] Using the current radial basis neural network prediction The predicted value is:

[0156]

[0157] Then the prediction error of the unknown function is

[0158]

[0159]

[0160] The purpose of using radial basis neural network is to make e η,i Approaching 0, the formation tracking task is completed. Then, the weight update law is designed according to the Lyapunov stability theory. The designed Lyapunov function is:

[0161]

[0162] Among them, γ>0. Then

[0163]

[0164] Set the sliding mode reaching law to:

[0165]

[0166] Among them, α η,i >0,β η,i >0, sgn(·) is the sign function.

[0167] The designed position control signal is:

[0168]

[0169] According to sin(·)+cos(·)=1, the trajectory control signal can be obtained:

[0170]

[0171] but but

[0172] Design the adaptive update law as:

[0173]

[0174] In the formula, ρη,i >0.

[0175] In summary, the position control module is used to drive the UAV to track the target and maintain the desired formation. On the one hand, the multi-agent soft actor critic algorithm that incorporates Shapley value is used to solve the problem of lazy agents in the strategy training process and realize the Nash equilibrium of cooperative games in complex dynamic environments. On the other hand, by constructing a hierarchical artificial potential field combined with reinforcement learning and then designing a neural adaptive sliding mode control algorithm to solve system uncertainty and environmental interference, a double-layer formation encirclement structure is formed with the target UAV as the center and the weaker UAVs relying on the stronger UAVs, which greatly enhances the robustness and adaptability of the entire system.

[0176] Next, in step 103, a posture control signal is determined. The posture control signal is determined by the posture control module to drive the tracking drone to track the desired yaw trajectory. Specifically, the roll angle and pitch angle of the virtual drone are determined based on the trajectory control signal, and then the second auxiliary control signal is generated by the PD control algorithm according to the roll angle and the pitch angle. Finally, the posture control signal is determined by the neural network adaptive sliding mode position control algorithm based on the second auxiliary control signal.

[0177] The position control and attitude control of a quadcopter are interdependent. Position control affects the effect and dynamic response of attitude control. At the same time, attitude control is the basis for achieving precise position control. The two need to work closely together through a coordinated feedback control system to achieve flight missions. In one embodiment of the present invention, the design of the attitude control module includes:

[0178] First, through the position control module obtained in the above steps, the roll angle and pitch angle of the virtual drone can be obtained:

[0179]

[0180] The yaw angle of the i-th virtual drone The desired yaw angle ψ is designed i,d The attitude subsystem is:

[0181]

[0182] in:

[0183] Design the desired yaw angle ψ i,d , then the yaw angle of the virtual drone is With ψ i,d The tracking error between is:

[0184]

[0185] The yaw angle model of the virtual drone is constructed as follows:

[0186]

[0187] Design attitude tracking auxiliary control signal:

[0188]

[0189] Among them, k3>0, k4>0, then:

[0190] e ψ =B(ψ s -ψ d 1 N ),

[0191] Among them, e ψ =[e ψ,1 ,…,e ψ,N ], but

[0192] According to Lyapunov stability theory, the Lyapunov function is constructed as follows:

[0193]

[0194] Among them, λ1 is the largest eigenvalue of B, a is a positive constant, and λ1 is the largest characteristic root of B. Select appropriate parameter values ​​k3, k4, λ1, and a so that

[0195] Next, a neural network adaptive sliding mode control strategy is designed to design the attitude controller. The sliding mode function is designed as:

[0196]

[0197] in c ξ is a positive constant to be designed.

[0198] Derivative of the sliding mode function:

[0199]

[0200] Among them, κ={2,3,4}.

[0201] From the analysis, we can see that is an unknown nonlinear function, which includes system uncertainty and environmental interference. Using radial basis neural network to approximate the unknown nonlinear function, we can get:

[0202]

[0203] in, is the ideal weight of the neural network, is the basis function vector,

[0204] is the approximation error.

[0205] Using the current radial basis neural network prediction The predicted value is:

[0206]

[0207] Then the prediction error of the unknown function is

[0208]

[0209] The purpose of using radial basis neural network is to make e ξ,i Approaching 0, the formation tracking task is completed. By designing the Lyapunov function, the attitude control module U is designed. i,κ for

[0210]

[0211] Among them, α ξ,i >0,β ξ,i >0.

[0212] According to Lyapunov stability theory, we have:

[0213]

[0214] but The design weight update law is as follows

[0215]

[0216] In the formula, ρ ξ,i >0.

[0217] In summary, by designing an attitude control module to drive the drone to track the target and maintain the desired yaw angle, on the one hand, PD control is used to achieve a rapid reduction in attitude error, making the system response faster; on the other hand, a neural adaptive sliding mode control algorithm is designed to solve system uncertainty and environmental interference, so that the system can form a more precise maneuver and achieve the encirclement target;

[0218] Next, in step 104, the tracking error variable is calculated. The UAV is cooperatively tracked and controlled based on the trajectory control signal and the attitude control signal, and the tracking error variable is calculated. as well as

[0219] Finally, in step 105, the iterative update is performed. Is it satisfied? where c p >0 is a custom smaller constant. If the above conditions are met, the control strategy of the current tracking drone i is the optimal control strategy. Otherwise, repeat the above steps to update the control strategy until the above conditions are met.

[0220] Figure 4 FIG. 1 is a schematic diagram showing a structure of a heterogeneous UAV hierarchical formation control system based on Shapley value and coupled SAC according to an embodiment of the present invention. Figure 4 As shown, a heterogeneous UAV hierarchical formation control system based on Shapley value and coupled SAC includes a position control module, an attitude control module and a feedback module. The position control module is used to determine the trajectory control signal of the UAV according to the position and attitude of the tracking UAV and the target UAV, the attitude control module is used to determine the attitude control signal of the UAV according to the trajectory control signal and attitude, and the feedback module is used to calculate the tracking error variable to perform feedback iteration of the control signal.

[0221] As shown in the figure, in one embodiment of the present invention, the position control module includes:

[0222] Actor submodule, comprising first, second, third and fourth sub-actors, wherein the first sub-actor is used to generate a first action signal according to the location information of a homogeneous neighboring drone, the second sub-actor is used to generate a second action signal according to the location information of a heterogeneous neighboring drone, the third sub-actor is used to generate a third action signal according to the location information of a target drone, and the fourth sub-actor is used to determine a fourth action signal according to the first, second, third action signals and the location information. In one embodiment of the present invention, the first, second, and third sub-actors respectively include LSTM networks, and the fourth sub-actor and the Critic submodule include FC networks;

[0223] A Critic submodule, configured to generate a marginal contribution Shapley Q value based on the fourth action signal and the position information;

[0224] an interaction potential field, comprising a homogeneous subgroup interaction potential field, a heterogeneous subgroup interaction potential field and a target interaction potential field, wherein the interaction potential field is used to generate a first auxiliary control signal according to the first, second and third action signals; and

[0225] A calculation submodule adopts a neural network adaptive sliding mode position control algorithm to determine a trajectory control signal based on the first auxiliary control signal.

[0226] Based on the control system, according to the Shapley value-based behavior-coupled multi-agent soft actor critic hierarchical heterogeneous multi-UAV collaborative tracking control method described in the previous steps, first initialize the tracking UAV position and velocity and the tracking target position p i,t and p g,t , initialize the relative position information and instantiate the potential function; next, initialize the actor network and divide the overall state information into three types of position information, including homogeneous neighbor drone position information, heterogeneous neighbor drone position information and target drone position information, as the input of the three sub-actor networks constructed with LSTM networks, obtain three actions as the potential field strength of the hierarchical interactive potential field, and finally use a new sub-actor network to couple the three actions to generate the final action; then construct the individual action value function Individual dual soft action value function j∈{1,2}, scores the final action, and updates the individual action value function through the global Shapley Q-value loss function back gradient propagation Promote the training of the optimal strategy; repeat the above steps to obtain the formation tracking auxiliary control signal Then, the position control module U is designed through the neural network adaptive sliding mode control algorithm. i,1 At the same time, the desired yaw angle is set, and the yaw angle attitude tracking auxiliary control signal is designed according to the PD algorithm. Then the virtual drone roll angle calculated by the designed horizontal position controller is and pitch angle And obtained by the virtual drone yaw angle model The attitude control module U is designed by using the neural network adaptive sliding mode control algorithm. i,κ , k = {2,3,4}. Based on this, the tracking error variable is calculated And determine whether it satisfies c p >0 is a custom smaller constant. If the above conditions are met, the control strategy of the current tracking drone i is the optimal control strategy. Otherwise, repeat the above steps to update the control strategy until the above conditions are met.

[0227] Figure 5a and 5bThe initial position and ideal formation diagram of the target formation tracking using the heterogeneous UAV formation control method based on Shapley value and coupled SAC according to one embodiment of the present invention are shown. In this task, the total set of tracking UAVs is represented as N = {N R ,N F}, where N R is the set of drones that can effectively detect the target, N F A collection of drones that cannot detect targets; at least 3 quad-rotor drones with detection capabilities, 3 quad-rotor drones without target detection capabilities, and 1 target drone; the communication distance between the two types of drones The communication relationship between tracking UAVs is represented by an undirected graph.

[0228] If we use 5 quad-rotor drones with detection capabilities and 5 quad-rotor drones without target detection capabilities, the communication distance between the two types of drones is They need to form a double-layer regular pentagonal formation with the target drone as the center, drones without the ability to detect targets in the outer layer, and drones with the ability to detect targets in the inner layer to track the target drone. Five quadrotor drones with detection capabilities and five quadrotor drones without the ability to detect targets start from any initial position and form a formation to track a moving target with an unknown motion trajectory. The initial drone position state is as follows: Figure 5a As shown, the heights of the drones can be different. After adopting the collaborative tracking control method of the present invention, the effect of the ideal multi-drone collaborative hunting game confrontation is as follows: Figure 5b shown.

[0229] The heterogeneous UAV layered formation control method and system of the present invention solves the problems of formation maintenance and collision avoidance under the condition that some UAV groups do not have the ability to detect targets, and the difficulty of maintaining communication under the condition that the communication distances between heterogeneous subgroups are different.

[0230] According to the Lagrange equation, the dynamic model of the i-th quadrotor drone is considered, and the virtual drone position model is constructed for the position subsystem. Through the proposed behavior-coupled multi-agent soft actorcritic hierarchical heterogeneous multi-UAV collaborative tracking control method based on Shapley value, auxiliary control signals are designed for two types of drone groups, among which the auxiliary control signals are designed for the drone i∈N with the ability to detect the target. R Design auxiliary control signal:

[0231]

[0232] For the undetectable target drone i∈N F Design auxiliary control signal:

[0233]

[0234] Output of the behavior-coupled multi-agent soft actor critic algorithm based on Shapley value a g , used to adjust the potential field strength of three types of interactive potential fields, and to enhance the artificial potential field method for formation maintenance and collision avoidance control of multiple quad-rotor drones, thereby solving the problem of communication maintenance difficulties and enabling drones to make more intelligent decisions.

[0235] At the same time, the heterogeneous UAV hierarchical formation control method and system also solves the problem of dynamic nonlinearity in multi-UAV systems. The design of the UAV i controller is divided into a position subsystem and an attitude subsystem:

[0236]

[0237] For the unknown nonlinear function and where η={x,y,z}, Through the neural network adaptive sliding mode control method, the sliding mode function is designed as:

[0238]

[0239] By using radial basis neural network to approximate unknown nonlinear functions,

[0240]

[0241] in and is the predicted value of the ideal weight of the neural network, and is the basis function vector.

[0242] According to Lyapunov stability theory, the adaptive update law is designed as:

[0243]

[0244] By introducing the neural network adaptive sliding mode control algorithm, the multi-UAV system has strong robustness in the face of system uncertainty and external interference, and can quickly adjust the control law according to system changes, solving the problem of dynamic nonlinearity in the multi-UAV system, thereby improving the tracking accuracy of the control system.

[0245] The heterogeneous UAV hierarchical formation control method and system improve the training stability and convergence speed of multi-agent reinforcement learning in complex environments. The multi-agent soft actor critic algorithm is a powerful machine learning method, but it faces the problem of difficult convergence of strategy training in the case of high environmental complexity. The present invention adopts a Shapley value-based method and uses a centralized training distributed execution implementation architecture to distribute the overall joint action value to the action value of each UAV according to the marginal contribution rate of the agent member to the group, and determines the average value of the marginal benefit created by each UAV member in the group. The introduced Shapley Q value function is:

[0246]

[0247] Since the amount of computation required to traverse alliance C is huge, the sub-alliances are processed to obtain the approximate Shapley Q value:

[0248]

[0249] By introducing the Shapley Q value method, the lazy agent problem is solved.

[0250] Then, the actor network architecture of first decomposing and then coupling is used, the output action of the decomposed sub-actor network is used as the strength of the layered interactive potential field, and the action of the sub-actor network is coupled to form the final action for evaluation. This method helps the drone swarm to make higher-quality decisions, enhances the dynamic path planning ability of the artificial potential field method, sets corresponding sub-actor networks for different training goals, and enhances multi-intelligence reinforcement learning to deal with the contradictions between different reward functions.

[0251] By incorporating Shapley Q value into the multi-agent soft actor critic algorithm to solve the credit allocation problem and the lazy agent problem, and using the decomposition and then coupling actor network architecture to solve the problem that reinforcement learning is difficult to converge in complex environments, the network architecture of the above methods is combined to improve the training stability and convergence speed of multi-agent reinforcement learning in complex environments.

[0252] Although various embodiments of the present invention are described above, it should be understood that they are presented as examples only and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should only be defined according to the attached claims and their equivalents.

Claims

1. A heterogeneous UAV layered formation control method based on Shapley value and coupled SAC, characterized in that: Includes steps: Confirm the location and attitude information of the tracking drone and the target drone; Based on the position and attitude information, a trajectory control signal is determined by a position control module to drive the tracking drone to track the target drone and maintain a desired formation; Determining a posture control signal through a posture control module to drive the tracking drone to track a desired yaw trajectory; Performing collaborative tracking control on the tracking UAV based on the trajectory control signal and the attitude control signal, and calculating a tracking error variable; as well as It is determined whether the tracking error variable meets the preset conditions. If so, the current trajectory control signal and attitude control signal are used as the optimal control strategy. Otherwise, the trajectory control signal and attitude control signal are updated until the preset conditions are met.

2. The heterogeneous UAV layered formation control method according to claim 1, characterized in that: Also includes the steps: Constructing a dynamic model of a UAV, wherein the UAV is a quad-rotor UAV; and An undirected graph is established to describe the interaction relationship of multiple UAV systems.

3. The heterogeneous UAV layered formation control method according to claim 1, characterized in that: The desired formation includes a first drone group in an outer layer, a second drone group in an inner layer, and a target drone in the center, wherein the first drone group does not have a target detection function, and the second drone group has a target detection function.

4. The heterogeneous UAV layered formation control method according to claim 1, characterized in that: Based on the position and posture information, determining the trajectory control signal by the position control module includes the following steps: A multi-agent reinforcement learning method that combines artificial potential field method and Shapley value is used to determine the trajectory control signal of the UAV.

5. The heterogeneous UAV layered formation control method according to claim 1, characterized in that: For each UAV, the determination of its trajectory control signal includes the following steps: According to the position information of the homogeneous neighbor drones, heterogeneous neighbor drones and the target drone of the drone, respectively, the first, second and third action signals are determined through the Actor submodule; Determining a fourth action signal based on the position information and the first, second, and third action signals; Based on the first, second and third action signals, determining a first auxiliary control signal through a homogeneous subgroup interaction potential field, a target interaction potential field and a heterogeneous subgroup interaction potential field; as well as Based on the first auxiliary control signal, a trajectory control signal is determined by a neural network adaptive sliding mode position control algorithm.

6. The heterogeneous UAV layered formation control method according to claim 5, characterized in that: The determination of the trajectory control signal also includes the steps of: The marginal contribution Shapley Q value function is adopted to evaluate the Actor submodule based on the fourth action signal and the position information, and perform iterative update.

7. The heterogeneous UAV layered formation control method according to claim 1, characterized in that: For each drone, the determination of its attitude control signal includes the following steps: Determining a roll angle and a pitch angle of the virtual drone based on the trajectory control signal; According to the roll angle and the pitch angle, a second auxiliary control signal is generated by using a PD control algorithm; and Based on the second auxiliary control signal, a posture control signal is determined by a neural network adaptive sliding mode position control algorithm.

8. A heterogeneous UAV hierarchical formation control system based on Shapley value and coupled SAC, characterized in that: include: A position control module, which is configured to determine a trajectory control signal of the UAV according to the positions and postures of the tracking UAV and the target UAV, so as to drive the tracking UAV to track the target UAV and maintain a desired formation; an attitude control module, which is configured to determine an attitude control signal of the UAV according to the trajectory control signal and the attitude, so as to drive the tracking UAV to track a desired yaw trajectory; as well as A feedback module is configured to calculate a tracking error variable to perform feedback iteration of the control signal.

9. The heterogeneous UAV hierarchical formation control system according to claim 8, characterized in that: The position control module comprises: An actor submodule, comprising a first, a second, a third and a fourth sub-actor, wherein the first sub-actor is configured to generate a first action signal according to the position information of a homogeneous neighboring drone, the second sub-actor is configured to generate a second action signal according to the position information of a heterogeneous neighboring drone, the third sub-actor is configured to generate a third action signal according to the position information of a target drone, and the fourth sub-actor is configured to determine a fourth action signal according to the first, second, third action signals and the position information; a Critic submodule, configured to generate a marginal contribution Shapley Q value based on the fourth action signal and the position information; an interaction potential field, comprising a homogeneous subgroup interaction potential field, a heterogeneous subgroup interaction potential field and a target interaction potential field, wherein the interaction potential field is configured to generate a first auxiliary control signal according to the first, second and third action signals; and The calculation submodule is configured to adopt a neural network adaptive sliding mode position control algorithm to determine a trajectory control signal based on the first auxiliary control signal.

10. The heterogeneous UAV hierarchical formation control system according to claim 9, characterized in that: The first, second, and third sub-actors all adopt LSTM networks, and the fourth sub-actor and the Criti c sub-module all adopt FC networks.

Images