Efficient cooperative task planning method for heterogeneous unmanned aerial vehicle cluster

Abstract

The invention discloses a heterogeneous unmanned aerial vehicle cluster efficient cooperative task planning method, which belongs to the field of unmanned aerial vehicle cluster task planning, and comprises the following steps: S1, inputting the total number of heterogeneous cluster unmanned aerial vehicles, the number of opposite ground targets, role division interval time steps, constraint strength between a global strategy and a role strategy, and the number of predefined roles; s2, realizing role division of the unmanned aerial vehicles, namely intelligent agents, by characterizing each action in the action space and characterizing the state of each unmanned aerial vehicle; s3, dynamically selecting the roles of the unmanned aerial vehicle according to the state observation of the unmanned aerial vehicle, and updating every T time steps; s4, generating and storing trajectory data by role strategy sampling, and performing initialization training to obtain a decision strategy of global collaborative planning; and S5, continuously updating and generating trajectory data according to a role strategy, training and guiding a global strategy, finally obtaining the global strategy, and performing model reasoning.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle cluster task planning, and specifically relates to an efficient collaborative task planning method for a heterogeneous unmanned aerial vehicle cluster. Background Art

[0002] Drones have a series of outstanding features such as environmental information perception, autonomous decision-making, and the ability to carry functional payloads. However, single drones often have limited survivability and single functions, so the application of drone swarms has received more and more attention. The efficient coordination of drone swarms inevitably requires cluster intelligence and global decision-making. Using swarm intelligence to achieve drone swarm task coordination further improves the performance ceiling of drones in various scenarios.

[0003] However, with the increase in the number of drones in drone swarms and the emergence of heterogeneous models, swarm collaborative decision-making faces a series of prominent problems such as huge decision space, redundant environmental information, highly coupled tasks, and high computational complexity. The collaborative decision-making method of drone swarms needs to have higher information processing capabilities, stronger adaptability to dynamic environments, and compatibility with larger cluster sizes. As drone swarms face collaborative decision-making in larger-scale scenario tasks, it is necessary to further improve the training efficiency of the cluster collaborative decision-making method and ensure its effectiveness.

[0004] In applications, multi-agent reinforcement learning algorithms generally face the problems of poor scalability, excessive observation dimensions as the number of agents increases, and sparse global rewards. We establish a mechanism for role policy to guide global policy, combine the framework of role division and role policy to promote global policy training with the traditional multi-agent reinforcement learning actor-critic framework, and initialize and accelerate global policy training by maintaining and updating role policies. This allows the decision space of different individuals in the multi-agent system to be limited by role selection, and assists global policy training under sparse reward conditions with relatively dense role rewards. This method is applied to heterogeneous drone clusters to maximize the performance of each drone in the heterogeneous cluster and achieve optimal collaborative task planning.

[0005] Previous methods for cooperative task planning of UAV clusters were mainly based on optimization methods and heuristic methods. Specifically, for the problem of cooperative task planning of multiple UAVs, the main models currently studied include improved particle swarm optimization, multi-dimensional multi-selection backpack, multi-traveling salesman, mixed integer linear programming, and dynamic network flow optimization. The above methods describe more about the time constraints in the task planning process, but do not fully consider the dynamic characteristics of the UAVs. In addition, the above methods are difficult to consider the heterogeneity of the cooperative task planning cluster. Mixed integer linear programming and dynamic network flow optimization models are only applicable to task planning problems with a small scope, a small number of targets, and a single environmental threat source. In addition, there is a method that abstracts the multi-machine cooperative problem into multiple independent multi-traveling salesman problems, optimizes the shortest route length, and then uses the simulated annealing algorithm to solve it. Although this method reduces the complexity of the problem, it ignores the essence of requiring collaboration between individuals in cluster task planning. Summary of the invention

[0006] In view of this, the object of the present invention is to provide an efficient collaborative task planning method for a heterogeneous UAV cluster.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] An efficient collaborative task planning method for a heterogeneous UAV cluster includes the following steps:

[0009] S1: Input the total number of heterogeneous swarm drones N1, the number of enemy ground targets N2, the time step of role division interval T, the constraint strength ξ between the global strategy and the role strategy, and the number of predefined roles K;

[0010] S2: By characterizing each action in the action space and the state of each drone, the roles of drones, i.e., intelligent agents, are divided;

[0011] S3: Based on the state observation of the drone, the drone role is dynamically selected and updated every T time steps;

[0012] S4: generating and storing trajectory data by sampling the role strategy, and initializing training with the trajectory data to obtain a decision strategy for global collaborative planning;

[0013] S5: According to the role strategy, the trajectory data is continuously updated and trained to guide the global strategy. Finally, the global strategy is obtained and the model is inferred to achieve the generation and update of decision-making plans for heterogeneous cluster collaborative task planning.

[0014] Furthermore, in step S2, the entire action space is divided into several subspaces to determine different roles. Based on the role-driven value decomposition method RODE, different actions are represented as action representations, which are supervised by the forward prediction model, and a process representing the current agent observation is added as the input of the role strategy during learning and training. The model of the representation process is trained through the following loss function:

[0015]

[0016] Where θ o and θ a Represents the parameters of observation and action, p o and p r By e Representations serve as predictors of observations and actions; λ e represents the proportionality coefficient, a -i represents the set of actions of other agents except its own actions, o i Represents own observation, Embedded representations representing the actions of each agent; after obtaining the action embeddings, these embedded representations are clustered to generate different action subspaces, and then different roles are obtained among all agents.

[0017] Further, in step S3, by calculating the hidden vector and role embedding representation The dot product between them assigns the agents to different roles. Through the hidden layer It is obtained by linear layer calculation. Calculated as:

[0018]

[0019] The role selection is updated by the following loss function, where the subscript and superscript “S” represent “selection”:

[0020]

[0021] in θ S ,ξ S Represent the parameters of multilayer perceptron and GRU, fully connected linear network with fixed-length embedded representation, and hypernetwork respectively; γ represents the reward discount factor, They represent the global action value function when the agent selects its role, and ρ represents the set of agent roles.

[0022] Furthermore, in step S4, each agent maintains a global strategy network, and according to the result of the overall role division, maintains several different role strategy networks respectively; the global strategy By parameters Indicates that the training goal is to maximize the global cumulative reward Role Strategy The network consists of parameters Indicates that the training goal is to maximize the role reward

[0023] Establish and The correlation measure between them is used to evaluate and improve the training efficiency, which is defined as follows:

[0024]

[0025] in represents the trajectory of agent i;

[0026] Based on the correlation metric, for the role strategy There is a cooperation-oriented objective function:

[0027]

[0028] Among them, ξ is a control The hyperparameter of the update range; the constraint strength is inversely proportional to the size of ξ: when the value of ξ is large, it means that the constraint on the correlation between the two strategies is relatively weak. On the contrary, when ξ accepts a smaller value, it means that the constraint between the two strategies is stronger.

[0029] For each agent's role-strategy The original learning objective function based on the CTDE framework is:

[0030]

[0031] Using the cutting and constraint operations of the PPO algorithm, Limited to [1-ε,1+ε], ε represents the policy clipping coefficient.

[0032] Furthermore, in step S5, according to the role strategy and the global strategy, the role strategy is used to generate data trajectories to initialize the global strategy, and the mechanism of continuously guiding the global strategy training optimization is as follows:

[0033]

[0034] Where ζ represents the constraint strength, μ represents the reduction factor, represents the global advantage function.

[0035] The beneficial effects of the present invention are as follows: the present invention innovatively proposes an actor-critic multi-agent reinforcement learning algorithm based on role division to promote global strategy in UAV collaborative task planning. When solving the problem of collaborative planning of heterogeneous UAV clusters, by establishing role division in the multi-agent system of heterogeneous clusters, it is possible to dynamically and flexibly set the functions of each UAV in the overall task, thereby reducing the decision-making space and improving the efficiency of training and decision-making. At the same time, the present invention further establishes a real simulation environment for collaborative task planning of heterogeneous UAV clusters that takes into account the kinematic and dynamic characteristics of UAVs, and verifies the effect of collaborative task planning of UAV clusters under the proposed method in the environment, proving that the proposed method can overcome the problem of collaborative task planning of large-scale heterogeneous UAV clusters, and realize the maximization of the effectiveness of each UAV in the heterogeneous cluster and the generation of the most optimized collaborative task planning scheme. The method proposed in the present invention has advantages in both training efficiency and model effect, exceeding various baseline algorithms.

[0036] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art may be taught from the practice of the present invention. The objectives and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solution and beneficial effects of the present invention clearer, the present invention provides the following drawings for illustration:

[0038] Figure 1 The collaborative task planning diagram of a heterogeneous UAV swarm, where (a) is a schematic diagram of the swarm collaborative task planning scenario, and (b) is a visualization diagram of the swarm collaborative task planning simulation environment;

[0039] Figure 2 Diagram of a multi-agent reinforcement learning framework where role policies guide global policies;

[0040] Figure 3 (a)-(d) in the figure are test result diagrams in four verification simulation environments. DETAILED DESCRIPTION

[0041] In the present invention, the scenario of collaborative task planning of heterogeneous UAV cluster is as follows: Figure 1 As shown, Figure 1 As shown in the real collaborative mission planning simulation environment in (b), there are three types of drones with different functions, namely, drones carrying reconnaissance payloads, strike payloads, and jamming payloads respectively. The three types of drones together constitute a heterogeneous drone cluster. Figure 1 Correspondence of ground targets of different shapes in (b) Figure 1Different types of ground targets in (a). The overall collaborative mission goal of our heterogeneous drone cluster is to eliminate all the enemy's ground targets.

[0042] Aiming at the goal of the above-mentioned heterogeneous drone cluster collaborative task planning, the actual scenario design of the two-party confrontation process is as follows: We use a heterogeneous drone cluster carrying reconnaissance payloads and strike payloads to perform target search, target strike and damage assessment tasks. The tasks and task objectives performed by a specific drone are generated by the drone cluster collaborative decision-making algorithm proposed by the present invention. When the situation changes during the execution of the task, for example: our drone detects the opponent's target in an area where the opponent's target may exist, our drone completes the strike on the opponent's target, our drone completes the damage assessment of the opponent's target that has been hit, our drone is destroyed by the opponent's ground air defense system, etc., the drone cluster responds quickly and intelligently generates a new collaborative decision-making action plan. The task is completed when our drone cluster destroys all the opponent's targets; when all our unmanned aerial vehicles in the combat area are destroyed, the task fails; in addition, when the collaborative planning time of our drone cluster exceeds the specified maximum duration, the task is forced to fail.

[0043] Based on this, the present invention provides an efficient collaborative task planning method for heterogeneous drone clusters, and an actor-critic multi-agent reinforcement learning algorithm based on role division to promote global strategy. Aiming at the collaborative task scenario of heterogeneous drone clusters, a multi-agent reinforcement learning algorithm with role strategy guiding global strategy is proposed, and a new dynamic game model of drone cluster confrontation is established. The heterogeneous drone cluster is divided into different roles according to the requirements of the scene task, which reduces the decision space of each drone and improves the efficiency of training.

[0044] A multi-agent system refers to a group of self-controlled and interactive agents that form a fully cooperative, fully competitive or hybrid overall system through sensor perception, actuator execution and other operations in the same environment. The rewards obtained by each agent in the system for issuing actions will be affected by the actions of other agents. The reinforcement learning method is applied to the multi-agent system. By using the characteristics of environmental information to generate gradients and learn the best strategy, it can improve the collaborative ability of agents and the stability of decision-making models. UAV swarm task planning is essentially a group intelligence technology. UAV swarms need to consider the coordination and consistency between multiple drones and the constraints of jointly executing tasks. Through information interaction and resource sharing, efficient collaboration is achieved to achieve optimal or approximate optimal global task indicators. The present invention mainly solves how to apply the new method of multi-agent reinforcement learning to the collaborative task planning of drone swarms with heterogeneous characteristics, so as to enhance the efficiency of cluster collaborative task planning and the optimality of strategy. Specifically, by modeling the role concept into the multi-agent reinforcement learning algorithm of the Actor-Critic architecture, a mechanism such as role strategy enhancement global strategy framework is established to improve the training efficiency and model performance of the heterogeneous drone swarm collaborative task planning algorithm.

[0045] Based on the actor-critic framework in multi-agent reinforcement learning, a multi-agent cooperative task framework enhanced by role strategy is established; the overall algorithm consists of three parts: role division; intra-role and inter-role cooperation; and role strategy enhanced global strategy learning. The role strategy guides the global strategy in an innovative way to achieve the optimization and training of the global strategy under the guidance of role strategy and sparse rewards. The overall architecture of the designed algorithm is as follows: Figure 2 shown.

[0046] Considering that using individual exploration as a sampling method for initializing global policy training is inefficient for large-scale multi-agent cooperative task scenarios, this scheme decides to adopt a more effective method at the role level. Therefore, first establish a role division mechanism and complete the role assignment for each subject (according to the time interval of each role re-division). Then, in the initial stage of model training, instead of pursuing the maximization of the global reward, the local scene of each agent is used as the starting point for training. After dividing the agent into multiple roles, initialize and maintain the policy network corresponding to each role for cooperative behavior exploration and trajectory generation. Then design the reward function R: S×A×S→R K+1 , using the corresponding role rewards of K roles, we explore the cooperative behavior between agents at the role level and seek to jointly achieve cooperation within and between roles. In addition, the reward of each role involves the cooperative behavior between different roles, which encourages collaboration between roles to a certain extent.

[0047] In order to achieve the role strategy to promote global strategy training, we must first perform accurate role division. We choose to divide the entire action space into several subspaces to determine different roles, and refer to the role-driven value decomposition method, namely the RODE algorithm. In order to establish a stable division mechanism, different actions are represented as action representations, which are supervised by the forward prediction model, unlike the original forward model in RODE, and a process that represents the current agent observation is added as the input of the role strategy during learning and training. Then, the model of the representation process can be trained by the following loss function:

[0048]

[0049] In the above formula, θ o and θ a Represents the parameters of observation and action, p o and p r By e Representations serve as predictors of observations and actions. e represents the proportionality coefficient, a -i represents the set of actions of other agents except its own actions, o i Represents own observation, The embedded representations represent the actions of each agent. After obtaining the action embeddings, these embedded representations are clustered to generate different action subspaces, and then different roles are obtained among all agents. The role selection module calculates the hidden vector and role embedding representation The dot product between them assigns the agents to different roles. Through the hidden layer It is obtained by linear layer calculation, and It can be calculated as:

[0050]

[0051] And the following loss function is designed to update the role selection module, where the subscript and superscript “S” represent “selection”:

[0052]

[0053] in θ S ,ξ S Represent the parameters of the multilayer perceptron and GRU, the fully connected linear network with a fixed-length embedded representation, and the hypernetwork. γ represents the reward discount factor, They represent the global action value function when the agent selects its role. In addition, ρ represents the set of agent roles.

[0054] The whole method needs to be trained in two stages, because the initial stage of sampling is crucial to improving the training efficiency and the effect of the final model. Therefore, there are two series of policy networks that need to be trained and maintained. Each agent needs to maintain a global policy network, and according to the results of the overall role division, maintain several different role policy networks. Global policy By parameters Indicates that the training goal is to maximize the global cumulative reward The role strategy The network consists of parameters Indicates that the training goal is to maximize the role reward Since the ultimate training goal is to obtain the optimal global strategy, we should establish and The correlation between them is used to evaluate and improve the training efficiency. So we define it as follows: represents the trajectory of agent i).

[0055]

[0056] On the one hand, based on the above related metrics, for role strategies There is a cooperation-oriented objective function:

[0057]

[0058] Among them, ξ is a control The update range hyperparameter. The constraint strength is inversely proportional to the size of ξ: when the value of ξ is large, it indicates that the constraint on the correlation between the two policies is relatively weak. On the contrary, when ξ accepts a smaller value, it indicates that the constraint between the two policies is stronger. On the other hand, for each agent's role-policy The original learning objective function based on the CTDE framework is:

[0059]

[0060] Referring to the cutting and constraint operations of the PPO algorithm, Limited to [1-ε,1+ε]. ε represents the policy clipping coefficient.

[0061] Then, according to the role strategy and the global strategy, the role strategy is used to generate data trajectories to initialize the global strategy, and the mechanism of continuously guiding the global strategy training optimization is as follows, where ζ represents the constraint strength and μ represents the reduction factor. represents the global advantage function.

[0062]

[0063] like Figure 3As shown in (a)-(d) in the figure, it was finally tested in a verification simulation environment, proving that the method proposed in the present invention has advantages in both training efficiency and model effect, surpassing various baseline algorithms.

[0064] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. An efficient collaborative task planning method for heterogeneous UAV clusters, characterized by: The following steps are involved: S1: Input the total number of heterogeneous swarm drones N1, the number of enemy ground targets N2, the time step of role division interval T, the constraint strength ξ between the global strategy and the role strategy, and the number of predefined roles K; S2: By characterizing each action in the action space and the state of each drone, the roles of drones, i.e., intelligent agents, are divided; S3: Based on the state observation of the drone, the drone role is dynamically selected and updated every T time steps; S4: generating and storing trajectory data by sampling the role strategy, and initializing training with the trajectory data to obtain a decision strategy for global collaborative planning; S5: According to the role strategy, the trajectory data is continuously updated and trained to guide the global strategy. Finally, the global strategy is obtained and the model is inferred to achieve the generation and update of decision-making plans for heterogeneous cluster collaborative task planning.

2. The method for efficient collaborative task planning of heterogeneous UAV clusters according to claim 1 is characterized by: In step S2, the entire action space is divided into several subspaces to determine different roles. Based on the role-driven value decomposition method RODE, different actions are represented as action representations, which are supervised by the forward prediction model. A process representing the current agent observation is added as the input of the role strategy during learning and training. The model of the representation process is trained through the following loss function: Where θ o and θ a Represents the parameters of observation and action, p o and p r By e Representations serve as predictors of observations and actions; λ e represents the proportionality coefficient, a -i represents the set of actions of other agents except its own actions, o i Represents own observation, Embedded representations representing the actions of each agent; After obtaining the action embeddings, these embedded representations are clustered to generate different action subspaces, and then different roles are obtained among all agents.

3. The efficient collaborative task planning method for heterogeneous UAV clusters according to claim 1 is characterized by: In step S3, by calculating the hidden vector and role embedding representation The dot product between them assigns the agents to different roles. Through the hidden layer It is obtained by linear layer calculation. Calculated as: The role selection is updated by the following loss function, where the subscript and superscript "S" represent "selection": in θ S ,ξ S Represent the parameters of multilayer perceptron and GRU, fully connected linear network with fixed-length embedded representation, and hypernetwork respectively; γ represents the reward discount factor, They represent the global action value function when the agent selects its role, and ρ represents the set of agent roles.

4. The efficient collaborative task planning method for heterogeneous UAV clusters according to claim 1 is characterized by: In step S4, each agent maintains a global strategy network, and according to the result of the overall role division, maintains several different role strategy networks respectively; the global strategy By parameters Indicates that the training goal is to maximize the global cumulative reward Role Strategy The network consists of parameters Indicates that the training goal is to maximize the role reward Establish and The correlation measure between them is used to evaluate and improve the training efficiency, which is defined as follows: in represents the trajectory of agent i; Based on the correlation metric, for the role strategy There is a cooperation-oriented objective function: Among them, ξ is a control The hyperparameter of the update range; the constraint strength is inversely proportional to the size of ξ: when the value of ξ is large, it means that the constraint on the correlation between the two strategies is relatively weak. On the contrary, when ξ accepts a smaller value, it means that the constraint between the two strategies is stronger. For each agent's role-strategy The original learning objective function based on the CTDE framework is: Using the cutting and constraint operations of the PPO algorithm, Limited to [1-ε,1+ε], ε represents the policy clipping coefficient.

5. The method for efficient collaborative task planning of heterogeneous UAV clusters according to claim 1 is characterized by: In step S5, according to the role strategy and the global strategy, the role strategy is used to generate data trajectories to initialize the global strategy, and the mechanism of continuously guiding the global strategy training optimization is as follows: Where ζ represents the constraint strength, μ represents the reduction factor, represents the global advantage function.

Images