Layered behavior decision-making method of unmanned aerial vehicle cluster for ground capture

Abstract

The invention belongs to the technical field of unmanned aerial vehicle clusters, and particularly relates to an unmanned aerial vehicle cluster hierarchical behavior decision-making method oriented to ground hunting, which comprises the following steps of: 1, determining various constraint conditions and income targets in a ground hunting task so as to establish a random game model oriented to the ground hunting task; the second step of considering the prediction problem of the motion trail of the surrounding target and the optimization problem of the multi-agent decision algorithm, and establishing an unmanned aerial vehicle cluster behavior decision network based on the improved MASAC algorithm, and the third step of online decision making. According to the method, the random game model oriented to the ground surrounding task can be established, then the position information of the surrounding target at the next moment can be predicted, the predicted trajectory data is input into the bottom layer MASAC decision network, and finally the optimal unmanned aerial vehicle cluster behavior decision strategy oriented to the ground surrounding task is obtained through offline training. And the ground surrounding of the target is completed by adopting an offline trained unmanned aerial vehicle cluster behavior decision-making method.

Description

Technical Field

[0001] The present invention belongs to the technical field of drone clusters, and in particular relates to a drone cluster hierarchical behavior decision method for ground capture. Background Art

[0002] With the development of artificial intelligence, intelligent methods represented by deep reinforcement learning have played an increasingly important role in solving multi-agent decision-making problems. UAV formations are expected to play an extremely important role in science and technology with informationization, unmanned and intelligent development as the key development points, and have broad application prospects because of their stronger precise positioning capabilities, networked cooperation capabilities, high flexibility and adaptability.

[0003] Taking advantage of the strong fitting ability and fast online speed of neural networks, deep reinforcement learning algorithms are used to make decisions on the behavior of drone swarms, thereby achieving ground roundups, which is an important development direction for future drone applications. In the process of making decisions on the behavior of drone swarms for ground roundups, the size and topological structure of the drone swarm are dynamically changing.

[0004] With the increasing number of drones and capture targets and the changing complexity of the battlefield environment, the difficulty of drone swarms in completing behavioral decisions is gradually increasing. How to improve the intelligence level of large-scale drone swarm behavior decision-making in a dynamic game environment will be the next key technical development direction. For drone swarm behavior decision-making tasks for ground capture missions, the current research has the following difficulties:

[0005] 1. Existing studies all assume that UAVs fly in free space or (and) at different altitudes, ignoring the obstacle avoidance problem of UAVs in complex three-dimensional urban environments with multiple obstacles and the collision avoidance problem between our UAVs;

[0006] 2. Existing studies have not considered factors such as the uncertainty of the capture target. In the process of cluster collaborative capture in a complex dynamic environment, if the movement trajectory of the capture target can be predicted in a timely and accurate manner, it can provide strong support for our decision-making;

[0007] 3. Existing multi-agent decision-making algorithms, such as the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, have problems such as hyper-parameter sensitivity, high computational complexity, and easy to fall into local optimality;

[0008] Therefore, considering the obstacle avoidance and collision avoidance problems in the ground encirclement and capture process, the prediction problem of the motion trajectory of the capture target, and the optimization problem of the multi-agent decision-making algorithm, the UAV cluster behavior decision-making algorithm based on the deep reinforcement learning algorithm is studied to realize the trajectory prediction and encirclement of the ground target by the UAV swarm, which is of vital importance to the intelligent coordination of the behavior of the UAV swarm in complex scenarios. Summary of the invention

[0009] The purpose of the present invention is to provide a hierarchical behavior decision-making method for drone swarms for ground capture, which can determine various elements such as the state set of the drone swarm behavior decision-making process, establish a random game model for ground capture tasks, and then predict the position information of the capture target at the next moment, and input the predicted trajectory data into the underlying MASAC decision network. Finally, the optimal drone swarm behavior decision strategy for ground capture tasks is obtained through offline training, and the drone swarm behavior decision method trained offline is used to complete the ground capture of the target.

[0010] The technical solution adopted by the present invention is as follows:

[0011] The hierarchical behavior decision-making method of drone swarm for ground capture includes the following steps:

[0012] Step 1: Clarify the various constraints and profit targets in the ground capture mission, and determine the elements of the state set, action set, immediate return value and discount factor in the drone swarm behavior decision-making process, so as to establish a stochastic game model for the ground capture mission;

[0013] Step 2: Consider the prediction problem of the target's motion trajectory and the optimization problem of the multi-agent decision-making algorithm, and establish a drone swarm behavior decision network based on the improved MASAC algorithm to determine the network architecture and parameter update method of the group prediction network, decision network and evaluation network;

[0014] Step 3: Online decision making, using the trained network to make real-time decisions;

[0015] After completing the learning process through the first and second steps, the decision network, evaluation network, and group trajectory prediction network parameters of each drone are also determined. Then, according to the target assigned to each of our drones, the decision network determines the acceleration values ​​that the drone needs to take in three directions based on the state information and the neural network parameters fitted during the offline training process;

[0016] At this point, since parameter training is no longer performed, the evaluation network does not participate in the selection of maneuver decision results. The decision network outputs the control amount of our UAV for the ground capture mission, and repeats this process until the capture mission is completed for all our UAVs.

[0017] The technical effects achieved by the present invention are:

[0018] The characteristics and beneficial effects of the present invention are:

[0019] (1) The present invention aims at two typical capture targets, ground armed personnel and armored vehicles. First, the obstacle avoidance and collision avoidance problems in the task process are considered, and multiple constraints are determined. Taking the task completion rate and round time step as indicators, the state set and other elements of the UAV cluster behavior decision-making process are determined, and a random game model for ground capture tasks is established;

[0020] Secondly, considering the prediction problem of the movement trajectory of the captured target and the optimization problem of the multi-agent decision-making algorithm, a hierarchical MASAC algorithm framework is proposed. The long short-term memory prediction network is used as the top network. The historical trajectory and environmental data of the captured target are used to predict the location information of the captured target at the next moment, and the predicted trajectory data is input into the bottom MASAC decision network.

[0021] Finally, the optimal UAV swarm behavior decision-making strategy for ground capture tasks is obtained through offline training, and the offline trained UAV swarm behavior decision-making method is used to complete the ground capture of the target.

[0022] (2) The present invention innovatively considers the high mobility of the target during the capture process, designs a target trajectory prediction network for ground capture tasks, studies the network back propagation algorithm, realizes accurate prediction of the target's future trajectory in complex environments, and improves the ability to predict the target's future state. The prediction algorithm avoids the use of a large amount of prior knowledge of domain experts and enhances the accuracy of the prediction.

[0023] (3) The present invention innovatively proposes a UAV swarm behavior decision-making algorithm based on group prediction. Aiming at the random game model for ground capture tasks, the present invention designs a decision network based on the MASAC algorithm, takes maximizing future accumulated reward value and maximizing entropy as training objectives, and builds a UAV swarm behavior decision-making algorithm framework. It realizes UAV swarm decision-making for ground capture tasks in complex urban building environments, and improves the intelligence level of UAVs in dynamic game environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is the overall technical solution of the present invention;

[0025] Figure 2 It is a ground roundup scene diagram in the present invention;

[0026] Figure 3 It is a structural diagram of the improved MASAC algorithm in the present invention;

[0027] Figure 4 It is a task benefit curve diagram in the present invention;

[0028] Figure 5 It is a curve diagram of the number of steps per round when the task is completed in the present invention;

[0029] Figure 6 It is a task completion rate curve diagram in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose and advantages of the present invention more clearly understood, the present invention is specifically described below in conjunction with embodiments. It should be understood that the following text is only used to describe one or several specific embodiments of the present invention, and does not strictly limit the scope of protection of the specific claims of the present invention.

[0031] The present invention aims to propose a UAV swarm behavior decision-making method based on an improved MASAC algorithm. Specifically, the present invention targets two typical capture targets, ground armed personnel and armored vehicles. First, the obstacle avoidance and collision avoidance problems in the mission process are considered, and multiple constraints are determined. The task completion rate and round time step are used as indicators to determine the state set and other elements of the UAV swarm behavior decision-making process, and establish a random game model for ground capture tasks; secondly, the prediction problem of the capture target's motion trajectory and the optimization problem of the multi-agent decision algorithm are considered, and a hierarchical MASAC algorithm framework is proposed, with the Long Short-Term Memory (LSTM) prediction network as the top-level network, and the historical trajectory and environmental data of the capture target are used to predict the location information of the capture target at the next moment, and the predicted trajectory data is input into the bottom-level MASAC decision network; finally, the optimal UAV swarm behavior decision-making strategy for ground capture tasks is obtained through offline training, and the offline trained UAV swarm behavior decision-making method is used to complete the ground capture of the target. The overall technical scheme is as follows: Figure 1 As shown;

[0032] like Figure 1-6 As shown in the figure, the hierarchical behavior decision method of drone cluster for ground capture includes the following steps:

[0033] S1: Clarify the various constraints and profit targets in the ground capture mission, and determine the elements such as the state set, action set, immediate return value and discount factor in the decision-making process of the drone swarm behavior, so as to establish a random game model for the ground capture mission;

[0034] First, according to the characteristics of the UAV kinematic model, the four constraints of the UAV are mainly considered: speed constraint, acceleration constraint, obstacle avoidance and collision avoidance constraint, and motion boundary constraint. Among them, the speed constraint and acceleration constraint of the UAV do not exceed the maximum maneuverability of the UAV. The obstacle avoidance and collision avoidance constraint ensures the safe and effective operation of the UAV in complex environments. The motion boundary constraint limits the range of the UAV's movement; secondly, considering the ground capture task, while ensuring that all our UAVs complete the capture task, we must also consider the number of steps required to complete the task and shorten the task completion time. Therefore, the maximum task completion rate and the minimum round time step are used as the decision-making benefit goals; thirdly, in order to realize the collaborative training of our UAVs, it is necessary to define the state set, action set, reward function and discount factor of the UAV cluster behavior decision process. The state set describes all the information that affects the UAV's behavior decision, the action set contains all the operations that the UAV can perform, and the reward function evaluates the effect of the UAV taking a specific action to guide its learning process. The discount factor determines the current value of the future reward and affects the UAV's trade-off between immediate and long-term benefits. Based on the above factors, a random game model is established for the ground capture task. The specific ground capture scene is shown in the figure below. Figure 2 As shown, our ten drones respectively capture five targets, including two soldiers and three tanks. When all five targets are successfully captured, the ground capture mission is considered completed;

[0035] Specifically, the following steps are included:

[0036] S11: Determine the constraints. According to the characteristics of the UAV kinematic model, consider the four constraints of speed constraint, acceleration constraint, obstacle avoidance constraint and motion boundary constraint;

[0037] The calculation formulas for velocity constraints, acceleration constraints, obstacle avoidance constraints, and motion boundary constraints are as follows:

[0038] Speed ​​constraint: During the roundup process, each of our drones, target soldiers, and tanks cannot exceed their maximum speed. The function of the speed constraint is:

[0039] ||v i ||≤v maxi ,||v j1 ||≤v maxj1 ,||v j2 ||≤v maxj2 (1)

[0040] Among them, v i 、v j1 and v j2 are the speeds of our drone, target soldier, and target tank, respectively, where i∈[1,I n ],j1∈[1,J n1],j2∈[1,J n2 ];

[0041] I n , J n1 and J n2 They are the number of our drones, the number of target soldiers, and the number of target tanks;

[0042] v maxi 、v maxj1 and v maxj2 They represent the maximum speeds of our drone, target soldier, and target tank respectively;

[0043] Acceleration constraint: The acceleration of each of our drones, target soldiers, and target tanks cannot exceed their maximum acceleration. The function of the acceleration constraint is:

[0044] ||a i ||≤a maxi ,||a j1 ||≤a maxj1 ,||a j2 ||≤a maxj2 (2)

[0045] Among them, a i 、a j1 and a j2 They represent the acceleration of our UAV, target soldier, and target tank respectively;

[0046] a maxi 、a maxj1 and a maxj2 They represent the maximum acceleration of our UAV, target soldier, and target tank respectively;

[0047] In order for our drone to capture the target, assuming that the maximum acceleration of our drone is greater than the maximum acceleration of all targets (if the speed and acceleration of our drone are both less than the target to be captured, it will be difficult for our drone to complete the task), and the maximum acceleration of the target tank is greater than the maximum acceleration of the target soldier, then:

[0048] a maxi >a maxj2 >a maxj1 , i∈[1,I n ],j1∈[1,J n1 ],j2∈[1,J n2 ] (3)

[0049] Obstacle avoidance and collision avoidance constraints: For safety reasons, our drones and environmental obstacles cannot collide with each other, and the relative distance between the two needs to be kept within a safe range. The function of the obstacle avoidance and collision avoidance constraints is:

[0050]

[0051] Among them, Δp ii′ =||p i -p i′ || represents the distance between the ith friendly UAV and the ith′ friendly UAV;

[0052] Δp ij =||p i -p j || represents the distance between the i-th UAV and the j-th environmental obstacle;

[0053] Motion boundary constraint: Our drone cannot exceed the given mission area during flight. The function of its motion boundary constraint is:

[0054]

[0055] Among them, p i,j,k Indicates the three-dimensional position information of our drone, p i Indicates the position of the drone in the x-axis direction, p j Indicates the position of the drone in the y-axis direction, p k Indicates the position of the drone in the z-axis direction, i bound represents the boundary constraint of the drone in the x-axis direction, j bound represents the boundary constraint of the drone in the y-axis direction, k bound Represents the boundary constraints of the drone in the z-axis direction.

[0056] S12: Clarify the decision-making benefit target. Consider the ground roundup mission. While ensuring that all our drones complete the roundup mission, we must also consider the number of steps required to complete the mission and shorten the mission completion time. Therefore, the maximum mission completion rate and the minimum round time step are used as the decision-making benefit targets.

[0057] When the drone performs a ground capture mission, assume that there are N task Our drone is at t task If all targets are captured simultaneously within time steps, the capture mission is considered successful, and the expression of the performance index is:

[0058] J task =T task (6)

[0059] Among them, T task It is the time step required for the successful capture of multiple drones. The optimization goal is to make J task Minimum, that is, multiple drones capture the target in the shortest time;

[0060] In the definition of Ntest In the test rounds, if there are M test In this experiment, our drone completes the capture mission, so the function formula for defining the success rate is:

[0061]

[0062] Among them, η represents the success rate of multi-UAV roundup tasks, and the training goal is to maximize η, that is, to maximize the success rate of multi-UAV ground roundups.

[0063] S13: Define the state set, action set, terminal reward function and discount factor of the drone swarm behavior decision-making process, and establish a random game model for ground capture tasks based on the above elements.

[0064] According to the constraints and decision-making benefit targets determined in equations (1) to (7), a random game model for ground roundup tasks is established, and the calculation formulas for the state set, action set, immediate return benefit value, and discount factor are established as follows:

[0065] State set S:

[0066] The position and speed information of all our drones are taken as the state set S of the drones. UAV , the position information and speed information of all captured targets are taken as the target state set S tar , the relative position information between the obstacle and our UAV is taken as the environmental state set S O , then the calculation function of the global state set S of the ground roundup task is:

[0067] S={S UAV ,S tar ,S O}={p UAV1 ,...,p UAVN ,v UAV1 ,...,v UAVN ,p tar1 ,...,p tarM ,v tar1 ,...,v tarM ,O 11 ,...,O NL}(8)

[0068] Among them, N represents the total number of our drones, M represents the total number of encircled targets, L represents the total number of obstacles, and p UAV1 ,...,p UAVN represents the three-dimensional position information of our drone, v UAV1 ,...,v UAVN Represents the three-dimensional speed information of our drone, p tar1 ,...,p tarMRepresents the three-dimensional position information of the capture target, v tar1 ,...,v tarM Represents the three-dimensional velocity information of the captured target, O 11 ,...,O NL Represents the relative position information of each friendly UAV to each environmental obstacle;

[0069] Action set a:

[0070] Taking the three-dimensional acceleration information of our drone as action a, the calculation function of action set a is:

[0071] a={a x ,a y ,a z} (9)

[0072] Among them, a x Indicates the acceleration of the drone in the x-axis direction, a y Indicates the acceleration of the drone in the y-axis direction, a z Indicates the acceleration of the drone in the z-axis direction;

[0073] Immediate return value r:

[0074] The immediate return benefit is the feedback value received from the environment when the drone transfers from the current state to the next state after performing a certain action. In order to ensure the capture capability and safety of the drone, the study considers the relative position of our drone and the capture target, as well as the distance from the target point to establish a reward and punishment mechanism, and determines the single-step decision immediate return benefit value r, which consists of two parts: process reward and terminal reward;

[0075] Among them, the process reward r pro The reward r is calculated based on the distance from the target. t and the distance reward r from the obstacle o It consists of two parts, as follows:

[0076]

[0077] Among them, d t Indicates the distance between the drone and the target, d w Indicates the minimum collision distance of the drone, d o Indicates the distance between the drone and the nearest obstacle;

[0078] ρ t is a critical value set. When the distance between the drone and the target is greater than the critical value, this collision penalty is added. When the distance between the drone and the target is less than the critical value, its main task is to capture the ground target and ignore this collision penalty.

[0079] k1, k2, k3 are constant coefficients;

[0080] Terminal Rewards end When the requirements for the end of the round are met, the immediate return value of the current state will receive a larger terminal reward or penalty depending on whether the round is successful, as follows:

[0081]

[0082] Among them, k4 is a constant coefficient;

[0083] Discount factor γ:

[0084] The discount factor γ indicates the importance of future earnings relative to current earnings, which is: 0<γ<1;

[0085] Among them, when γ = 0, it means that only current benefits are considered without considering future benefits. When γ = 1, it means that future benefits and current benefits are equally important.

[0086] S2: Consider the prediction problem of the target's motion trajectory and the optimization problem of the multi-agent decision-making algorithm, and establish a drone swarm behavior decision network based on the improved MASAC algorithm to determine the network architecture and parameter update method of the group prediction network, decision network and evaluation network;

[0087] First, considering the uncertainty of the trajectory of the target in the actual ground capture mission, in order to enable our drone to complete the ground capture mission more efficiently, it is necessary to design a prediction network to predict the trajectory of the target. Therefore, the LSTM network is introduced as the top-level prediction network for the entire drone cluster behavior decision-making; secondly, in order to solve the problems of hyper-parameter sensitivity, high computational complexity, and easy to fall into local optimality in the existing multi-agent decision-making algorithm, the traditional single-agent SAC decision-making algorithm is extended to the multi-agent decision-making field, and the MASAC autonomous decision-making algorithm is used as the underlying decision-making architecture to design the decision network and evaluation network of the drone respectively; finally, it is necessary to determine the network architecture and parameter update method of the group prediction network, decision network and evaluation network respectively. The network architecture defines the input and output and data processing method of the network. The parameter update method needs to determine the loss function of the network and adjust the internal parameters of the network according to the feedback of the loss function.

[0088] The algorithm structure diagram of the improved MASAC is as follows Figure 3As shown in the figure, the swarm prediction network uses the historical trajectory and environmental data of the target to predict the location information of the target at the next moment, and inputs the predicted trajectory data into the underlying MASAC decision network. The decision network combines the target prediction trajectory with our drone state data to give the action to be taken by the drone at the next moment. Each action will be evaluated by the corresponding evaluation network, output a value estimate, and use the maximum entropy learning method to add entropy terms in the optimization process of the decision network to guide the learning process of the drone.

[0089] Specifically, the following steps are included:

[0090] S21: Design of LSTM prediction network, decision network and evaluation network architecture;

[0091] In the ground capture mission, the LSTM network is used to predict the future trajectory of all targets. LSTM is a special type of recurrent neural network (RNN), which is particularly suitable for processing and predicting long-term dependency problems in time series data. In the ground capture mission, the LSTM network can be used to predict the state of the capture target, thereby providing support for the behavior decision of the drone cluster. The main architecture of the LSTM prediction network includes the input layer, LSTM layer and output layer;

[0092] When designing the LSTM prediction network architecture, the LSTM prediction network architecture can be designed first as follows:

[0093] In the ground capture task, the state input of the LSTM network is the state information S of each capture target history Q beats. pre ,as follows:

[0094] S pre ={S tar_(t-Q) ,...,S tar_i ,...,S tar_t},i∈[tQ,t] (12)

[0095] Among them, S tari is the target state set for the i-th shot, S tar_t is the target state set at the current moment;

[0096] The LSTM layer controls the inflow, storage, and outflow of information through a special gating mechanism, thereby effectively learning long-term dependencies. In the designed LSTM prediction network, several LSTM layers are included to improve the network learning ability and trajectory prediction ability. First, the input gate determines which new information is stored in the cell state, and then the forget gate determines which information is discarded by weighting the cell state at the previous moment and the current input. Finally, the output gate determines which information is output in the next layer or at the next moment.

[0097] The output of the prediction network includes the position and speed information of each capture target at the next moment, as follows:

[0098] S out ={p out1 ,...,p outM ,v out1 ,...,v outM} (13)

[0099] Among them, p out1 ,...,p outM represents the predicted position of M capture targets at the next moment, v out1 ,...,v outM Indicates the predicted speed of M encircled targets at the next moment;

[0100] Then the decision network architecture is designed, where the input of the decision network is the drone state set S UAV , the target prediction value S out and environmental obstacle information S O The designed function is:

[0101] s t ={S UAV ,S out ,S O} (14)

[0102] Among them, s t is the current state of the drone, then the output of the drone’s decision network is expressed as t Select action a at the current moment according to probability π, then:

[0103] a=π(s t ) (15)

[0104] Finally, the evaluation network architecture is designed. The input of the evaluation network is the current state and action of our drone, the state and benefit value of the next moment after taking the action. The designed function is:

[0105] S critic ={s t ,a t ,s t+1 ,r t+1} (16)

[0106] Among them, s t is the current status information of our drone, a t is the action information taken by our drone at the current moment, s t+1is the state information at the next moment after taking action, r t+1 The profit information obtained at the next moment.

[0107] The output of the evaluation network is the value Q of taking the action in the current state.

[0108] In addition, the evaluation network provides real-time evaluation and feedback of the action value function and incorporates the entropy term into the reward signal, thereby guiding the decision network to perform maximum entropy learning. This method makes full use of the feedback of the evaluation network, not only pursuing the optimal strategy, but also significantly enhancing the ability to explore the environment, which is of great significance for improving the exploration and robustness of our drones in complex and changeable urban building environments;

[0109] S22: Design of parameter updating method.

[0110] When designing the parameter update method, the LSTM prediction network parameters are updated first;

[0111] In order to ensure the real-time prediction of target trajectory in ground roundup scenarios, it is necessary to implement target trajectory prediction based on the target trajectory prediction network framework and adopt the method of offline training and online application. Therefore, considering the temporal nature of state information, the time-based back propagation algorithm is studied, and the parameters such as the number of sample iterations and learning rate in the network training process are determined based on experience and offline experiments, and the weights of the trajectory prediction network are updated; and in the actual online prediction process, the trained target trajectory prediction network is used to achieve real-time online prediction of the target trajectory.

[0112] Therefore, after the sample data is constructed, the training samples and prediction samples are normalized. Then the training samples are input into the LSTM network model to train the network, and the network parameters are updated according to the loss size. Finally, the test samples are used to test the network performance and obtain the prediction results.

[0113] In the design process of the back propagation algorithm of the target trajectory prediction network under the ground capture task, the loss function plays a role in guiding the learning of the neural network. The loss function design needs to comprehensively consider the optimization goal, so the design is as follows:

[0114]

[0115] Where n is the number of batch samples in each round of training, i∈[1,n] is the i-th sample in the batch, and P i Represents the predicted value of the network output, O i Represents the true value of the target trajectory. The optimization goal of the prediction network is to reduce the error between the network's predicted output of the target trajectory and the true value, that is, to make MSE approach 0;

[0116] Next, the decision network parameters are updated;

[0117] In ground capture missions, drones need to make quick decisions in uncertain environments to capture targets. The SAC algorithm encourages drones to explore unknown or uncertain actions by maximizing policy entropy, which can more effectively adapt to complex urban building environments.

[0118] Therefore, the goal of the decision network is designed to maximize the expected discounted reward plus the policy entropy. The gradient update formula of the decision network is:

[0119]

[0120] The entropy part serves as the learning objective of the drone, which improves the stability of training and makes the decision more robust.

[0121] Considering the correlation between the empirical data used to train the network, in order to improve the convergence ability of the network, a target decision network and a target evaluation network are designed to output the target Q value. The weights of the target decision network and the target evaluation network are updated by soft update, which are:

[0122]

[0123] Among them, θ′ is the weight of the target decision network, Evaluate network weights for the target;

[0124] 0<τ<1 is the soft update coefficient;

[0125] Finally, the evaluation network parameters are updated;

[0126] In the learning framework of centralized training and distributed execution, the evaluation network approximates the cumulative benefits and policy entropy of our drone based on the global state, action, and future trajectory of the captured target. However, in the approximation process, there will always be over-estimation problems. Therefore, a dual network structure is used to estimate the Q value, and the smaller Q value of the two Q values ​​is selected as the estimated value. The loss function of the evaluation network updates the network parameters by back-propagating by minimizing the time difference error, as follows:

[0127]

[0128] in, represents the parameters of the evaluation network of our UAV i;

[0129] And y(r,s′,d)) represents the approximate target value, and the calculation formula is:

[0130]

[0131] in, represents the strategy target network and evaluation target network of our drone i, and their parameters are d represents whether the round is over.

[0132] S3: Online decision making, using the trained network to make real-time decisions.

[0133] After completing the offline learning process through S1 and S2, the decision network, evaluation network, and group trajectory prediction network parameters of each drone are also determined. Then, according to the target assigned to each of our drones, the decision network determines the acceleration values ​​that the drone needs to take in three directions based on the state information and the neural network parameters fitted during the offline training process;

[0134] At this point, the state data obtained by the interaction between the drone and the environment and the predicted trajectory of the encircled target are input into the trained decision network. The decision network can quickly calculate the control amount of the drone's decision to realize the drone's cluster behavior decision;

[0135] At this point, since no more parameter training is performed, the evaluation network does not participate in the selection of maneuver decision results, and the decision network outputs the control amount of our drone for the ground capture mission. This process is repeated until the capture mission is completed for all our drones.

[0136] The SAC (Soft Actor-Critic) algorithm is a single-agent deep reinforcement learning algorithm proposed by Tuomas Haarnoja of Deep Mind in the United States at the ICML conference in 2018. This method is an algorithm based on the maximum entropy theory. It introduces the concept of maximum entropy on the basis of maximizing future cumulative rewards, and tries to maximize the balance between the reward value and the entropy value (i.e., the randomness of the strategy), and can solve the reinforcement learning problems of discrete action space and continuous action space. To this end, for the problem of drone cluster behavior decision-making, this patent, based on this method, extends the decision-making algorithm to the field of multi-agent decision-making, comprehensively considers the obstacle avoidance and collision avoidance problems in the ground round-up process, the prediction problem of the movement trajectory of the round-up target, the optimization problem of the multi-agent decision-making algorithm, etc., takes the task completion rate and round time step as indicators, establishes a random game model for ground round-up tasks, and proposes a MASAC hierarchical decision network framework in combination with the prediction network, and finally realizes the ground round-up of multiple targets by our drone group.

[0137] Result analysis:

[0138] In order to verify the effectiveness of the drone swarm behavior decision algorithm proposed in the present invention, a ground capture scenario in which 10 drones of our side capture 2 types of 5 targets is constructed in a 3D map of size 275*10*200. The capture targets consist of 2 soldiers and 3 tanks, of which each soldier requires 1 drone to capture and each tank requires 2 drones to capture. After all 5 targets are captured successfully, the entire ground capture mission is considered completed. In each initialization scenario, the initial position and preset trajectory of the capture target are randomly initialized to ensure that the capture target has sufficient randomness.

[0139] The improved MASAC algorithm adopted by the present invention improves the convergence speed and stability of the ground roundup task. The simulation results obtained are shown in the figure below: Figure 4-Figure 6 As shown, the task benefit curves, step counts per round when the task is completed, and task completion rate curves of the improved MASAC UAV swarm behavior decision algorithm proposed by the present invention and the traditional MASAC algorithm are compared. The horizontal axes in the figures are the number of training rounds, the blue lines are the index curves of the traditional MASAC algorithm, and the yellow lines are the index curves of the improved MASAC algorithm. They are described separately: Figure 4 This is a comparison of the task benefit curve. The vertical axis is the benefit value of the ground roundup task. It can be seen from the figure that the improved MASAC algorithm can converge quickly in terms of task benefit. Figure 5 The figure shows the comparison of the number of steps per round when the task is completed. The vertical axis in the figure is the number of round steps at the end of the ground round-up task. It can be seen from the figure that the improved MASAC algorithm can converge faster in terms of the number of round steps; Figure 6 This is a comparison of the task completion rate curve. The vertical axis in the figure is the task success rate. It can be seen from the figure that the task success rate using the improved MASAC algorithm is more stable after reaching the maximum value.

[0140] The above is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention shall be implemented according to the conventional means in the art unless otherwise specified and limited.

Claims

1. A hierarchical behavior decision-making method for drone swarms for ground capture, characterized by: The following steps are involved: S1: Clarify the various constraints and profit targets in the ground capture mission, and determine the elements of the state set, action set, immediate return value and discount factor in the UAV swarm behavior decision-making process, so as to establish a random game model for the ground capture mission; S2: Consider the prediction problem of the target's motion trajectory and the optimization problem of the multi-agent decision-making algorithm, and establish a drone swarm behavior decision network based on the improved MASAC algorithm to determine the network architecture and parameter update method of the group prediction network, decision network and evaluation network; S3: Online decision making, using the trained network to make real-time decisions; After completing the learning process through S1 and S2, the decision network, evaluation network, and group trajectory prediction network parameters of each drone are also determined. Then, according to the target assigned to each of our drones, the decision network determines the acceleration values ​​that the drone needs to take in three directions based on the state information and the neural network parameters fitted during the offline training process; At this point, since parameter training is no longer performed, the evaluation network does not participate in the selection of maneuver decision results. The decision network outputs the control amount of our UAV for the ground capture mission, and repeats this process until the capture mission is completed for all our UAVs.

2. The UAV cluster hierarchical behavior decision method for ground capture according to claim 1 is characterized by: The S1 comprises the following steps: S11: Determine the constraints. According to the characteristics of the UAV kinematic model, consider the four constraints of speed constraint, acceleration constraint, obstacle avoidance constraint and motion boundary constraint; S12: Clarify the decision-making benefit target. Consider the ground roundup mission. While ensuring that all our drones complete the roundup mission, we must also consider the number of steps required to complete the mission and shorten the mission completion time. Therefore, the maximum mission completion rate and the minimum round time step are used as the decision-making benefit targets. S13: Define the state set, action set, terminal reward function and discount factor of the drone swarm behavior decision-making process, and establish a random game model for ground capture tasks based on the above elements.

3. The UAV cluster hierarchical behavior decision method for ground capture according to claim 2 is characterized by: The calculation formulas of the speed constraint, acceleration constraint, obstacle avoidance constraint and motion boundary constraint in S11 are as follows: Speed ​​constraint: During the roundup process, each of our drones, target soldiers, and tanks cannot exceed their maximum speed. The function of the speed constraint is: ||v i ||≤v maxi ,||v j1 ||≤v maxj1 ,||v j2 ||≤v maxj2 (1) Among them, v i 、v j1 and v j2 are the speeds of our drone, target soldier, and target tank, respectively, where i∈[1,I n ],j1∈[1,J n1 ],j2∈[1,J n2 ]; I n , J n1 and J n2 They are the number of our drones, the number of target soldiers, and the number of target tanks; v maxi 、v maxj1 and v maxj2 They represent the maximum speeds of our drone, target soldier, and target tank respectively; Acceleration constraint: The acceleration of each of our drones, target soldiers, and target tanks cannot exceed their maximum acceleration. The function of the acceleration constraint is: ||a i ||≤a maxi ,||a j1 ||≤a maxj1 ,||a j2 ||≤a maxj2 (2) Among them, a i 、a j1 and a j2 They represent the acceleration of our UAV, target soldier, and target tank respectively; a maxi 、a maxj1 and a maxj2 They represent the maximum acceleration of our UAV, target soldier, and target tank respectively; In order for our drone to capture the target, assuming that the maximum acceleration of our drone is greater than the maximum acceleration of all targets, and the maximum acceleration of the target tank is greater than the maximum acceleration of the target soldier, then: a maxi >a maxj2 >a maxj1 , i∈[1,I n ],j1∈[1,J n1 ],j2∈[1,J n2 ] (3) Obstacle avoidance and collision avoidance constraints: Our drones and environmental obstacles cannot collide with each other, and the relative distance between the two needs to be kept within a safe range. The function of the obstacle avoidance and collision avoidance constraints is: Δp ii′ >d out,I ,i′∈[1,N I ],i′≠i (4) Δp ij >d out,J ,i∈[1,N I ],j∈[1,N J ] Among them, Δp ii′ =||p i -p i′ || represents the distance between the ith friendly UAV and the ith′ friendly UAV; Δp ij =||p i -p j || represents the distance between the i-th UAV and the j-th environmental obstacle; d out,I Indicates the minimum distance allowed between our drones; d out,J Indicates the minimum distance allowed between our drone and environmental obstacles. Motion boundary constraint: Our drone cannot exceed the given mission area during flight. The function of its motion boundary constraint is: Among them, p i,j,k Indicates the three-dimensional position information of our drone, p i Indicates the position of the drone in the x-axis direction, p j Indicates the position of the drone in the y-axis direction, p k Indicates the position of the drone in the z-axis direction, i bound represents the boundary constraint of the drone in the x-axis direction, j bound represents the boundary constraint of the drone in the y-axis direction, k bound Represents the boundary constraints of the drone in the z-axis direction.

4. The method for making decisions on the hierarchical behavior of drone swarms for ground capture according to claim 3 is characterized by: In S12, when the UAV performs a ground capture mission, it is assumed that there are N task Our drone is at t task If all targets are captured simultaneously within time steps, the capture mission is considered successful, and the expression of the performance index is: J task =T task (6) Among them, T task represents the time step required for the success of the multi-UAV roundup mission; In the definition of N test In the test rounds, if there are M test In this experiment, our drone completes the capture mission, so the function formula for defining the success rate is: Among them, η represents the success rate of multi-UAV roundup mission.

5. The method for making decisions on the hierarchical behavior of drone swarms for ground capture according to claim 4 is characterized in that: In S13, according to the constraints and decision-making benefit targets determined in formulas (1) to (7), a random game model for ground roundup tasks is established, and a calculation formula for the state set, action set, immediate return benefit value and discount factor is established as follows: State set S: The position and speed information of all our drones are taken as the state set S of the drones. UAV , the position information and speed information of all captured targets are taken as the target state set S tar , the relative position information between the obstacle and our UAV is taken as the environmental state set S O , then the calculation function of the global state set S of the ground roundup task is: S={S UAV ,S tar ,S O }={p UAV1 ,...,p UAVN ,v UAV1 ,...,v UAVN ,p tar1 ,...,p tarM ,v tar1 ,...,v tarM ,O 11 ,...,O NL }(8) Among them, N represents the total number of our drones, M represents the total number of encircled targets, L represents the total number of obstacles, and p UAV1 ,...,p UAVN represents the three-dimensional position information of our drone, v UAV1 ,...,v UAVN Represents the three-dimensional speed information of our drone, p tar1 ,...,p tarM Represents the three-dimensional position information of the capture target, v tar1 ,...,v tarM Represents the three-dimensional velocity information of the captured target, O 11 ,...,O NL Represents the relative position information of each friendly UAV to each environmental obstacle; Action set a: Taking the three-dimensional acceleration information of our drone as action a, the calculation function of action set a is: a={a x ,a y ,a z } (9) Among them, a x Indicates the acceleration of the drone in the x-axis direction, a y Indicates the acceleration of the drone in the y-axis direction, a z Indicates the acceleration of the drone in the z-axis direction; Immediate return value r: In order to ensure the capture capability and safety of the drone, a reward and punishment mechanism is established by considering the relative position of our drone and the capture target, as well as the distance from the target point, and determining the immediate return benefit value r of the single-step decision. The immediate return benefit value consists of two parts: process reward and terminal reward. Among them, the process reward r pro The reward r is calculated based on the distance from the target. t and the distance reward r from the obstacle o It consists of two parts, as follows: Among them, d t Indicates the distance between the drone and the target, d w Indicates the minimum collision distance of the drone, d o Indicates the distance between the drone and the nearest obstacle; ρ t is a critical value set. When the distance between the drone and the target is greater than the critical value, this collision penalty is added. When the distance between the drone and the target is less than the critical value, its main task is to capture the ground target and ignore this collision penalty. k1, k2, k3 are constant coefficients; Terminal Rewards end When the requirements for the end of the round are met, the immediate return value of the current state will receive a larger terminal reward or penalty depending on whether the round is successful, as follows: Among them, k4 is a constant coefficient; Discount factor γ: The discount factor γ indicates the importance of future earnings relative to current earnings, which is: 0<γ<1; Among them, when γ = 0, it means that only current benefits are considered without considering future benefits. When γ = 1, it means that future benefits and current benefits are equally important.

6. The UAV swarm hierarchical behavior decision method for ground capture according to claim 1 is characterized by: The S2 includes the following steps: S21: Design of LSTM prediction network, decision network and evaluation network architecture; S22: Design of parameter updating method.

7. The method for making decisions on the hierarchical behavior of drone swarms for ground capture according to claim 6 is characterized by: In S21, when designing the LSTM prediction network architecture, the LSTM prediction network architecture may be designed first as follows: In the ground capture task, the state input of the LSTM network is the state information S of each capture target history Q beats. pre ,as follows: S pre ={S tar_(t-Q) ,...,S tar_i ,...,S tar_t },i∈[t-Q,t] (12) Among them, S tari is the target state set for the i-th shot, S tar_t is the target state set at the current moment; The output of the prediction network includes the position and speed information of each capture target at the next moment, as follows: S out ={p out1 ,...,p outM ,v out1 ,...,v outM } (13) Among them, p out1 ,...,p outM represents the predicted position of M capture targets at the next moment, v out1 ,...,v outM Indicates the predicted speed of M encircled targets at the next moment; Then the decision network architecture is designed, where the input of the decision network is the drone state set S UAV , the target prediction value S out and environmental obstacle information S O The designed function is: s t ={S UAV ,S out ,S O } (14) Among them, s t is the current state of the drone, then the output of the drone’s decision network is expressed as t Select action a at the current moment according to probability π, then: a=π(s t ) (15) Finally, the evaluation network architecture is designed. The input of the evaluation network is the current state and action of our drone, the state and benefit value of the next moment after taking the action. The designed function is: S critic ={s t ,a t ,s t+1 ,r t+1 } (16) Among them, s t is the current status information of our drone, a t is the action information taken by our drone at the current moment, s t+1 is the state information at the next moment after taking action, r t+1 The profit information obtained at the next moment.

8. The UAV swarm hierarchical behavior decision method for ground capture according to claim 7 is characterized by: When designing the parameter update method in S22, the LSTM prediction network parameters are first updated, and the update function is: Where n is the number of batch samples in each round of training, i∈[1,n] is the i-th sample in the batch, and P i Represents the predicted value of the network output, O i Represents the true value of the target trajectory; Next, the decision network parameters are updated, and the goal of the decision network is designed to maximize the expected discounted reward plus the policy entropy. The gradient update formula of the decision network is: The weights of the target decision network and the target evaluation network are updated by soft updating, which are: Among them, θ′ is the weight of the target decision network, Evaluate network weights for the target; 0<τ<1 is the soft update coefficient; Finally, the evaluation network parameters are updated, and the dual network structure is used to estimate the Q value. The smaller Q value of the two Q values ​​is selected as the estimated value. The loss function of the evaluation network updates the network parameters by back propagating by minimizing the time difference error, as follows: in, represents the parameters of the evaluation network of our UAV i; And y(r,s′,d)) represents the approximate target value, and the calculation formula is: in, It represents the strategy target network and the evaluation target network of our drone i, and their parameters are θ i ′, d represents whether the round is over.

Images