A heterogeneous unmanned aerial vehicle cluster efficient cooperative task planning method

Abstract

The application discloses a kind of heterogeneous unmanned aerial vehicle cluster efficient cooperative task planning method, belong to unmanned aerial vehicle cluster task planning field, including the following steps: S1: input heterogeneous cluster unmanned aerial vehicle total number, opposite ground target number, role division interval time step, constraint strength between global strategy and role strategy, predefined role number;S2: by the representation of each action in action space and the representation of each unmanned aerial vehicle state, realize the role division of unmanned aerial vehicle, i.e. intelligent agent;S3: according to the state observation of unmanned aerial vehicle, dynamically carry out unmanned aerial vehicle role selection, update every T time step;S4: trajectory data is generated and stored by role strategy sampling, and the decision strategy of global cooperative planning is obtained by initialization training;S5: according to role strategy, constantly update trajectory data and train guide global strategy, finally obtain global strategy and carry out model inference.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicle (UAV) cluster mission planning, and in particular relates to an efficient collaborative mission planning method for heterogeneous UAV clusters. Background Art

[0002] Drones (UAVs) possess a range of outstanding capabilities, including environmental awareness, autonomous decision-making, and the ability to carry functional payloads. However, individual drones often have limited survivability and limited functionality. Consequently, the application of UAV swarms is gaining increasing attention. Efficient coordination of UAV swarms inevitably requires intelligent swarming and holistic decision-making. Leveraging swarm intelligence to achieve coordinated UAV swarming tasks further raises the performance ceiling of UAVs in various scenarios.

[0003] However, with the increasing number of drones in a swarm and the emergence of heterogeneous models, swarm collaborative decision-making faces a series of prominent challenges, including a large decision space, redundant environmental information, highly coupled tasks, and high computational complexity. This requires swarm collaborative decision-making methods with higher information processing capabilities, stronger adaptability to dynamic environments, and compatibility with larger swarms. As swarms face the challenge of collaborative decision-making in larger scenarios and tasks, it is necessary to further improve the training efficiency of swarm collaborative decision-making methods while ensuring their effectiveness.

[0004] Multi-agent reinforcement learning algorithms generally face challenges in application, such as poor scalability, excessive observation dimensionality as the number of agents increases, and sparse global rewards. We establish a mechanism whereby role policies guide the global policy. We combine the framework of role partitioning and role policies promoting global policy training with the traditional actor-critic framework for multi-agent reinforcement learning. By maintaining and updating role policies, we initialize and accelerate global policy training. This allows for the definition of the decision space for different individuals in a multi-agent system through role selection, and facilitates global policy training under sparse reward conditions through denser role rewards. We apply this method to a heterogeneous drone swarm, maximizing the efficiency of each drone within the swarm and achieving optimal collaborative mission planning.

[0005] Previous approaches to collaborative mission planning for UAV swarms have primarily relied on optimization and heuristic methods. Specifically, for multi-UAV collaborative mission planning, the main models currently under investigation include improved particle swarm optimization, multi-dimensional multi-choice knapsack, multi-traveling salesman, mixed integer linear programming, and dynamic network flow optimization. These approaches primarily address time constraints in the mission planning process, but fail to fully consider the dynamic characteristics of the UAVs. Furthermore, these approaches struggle to account for the heterogeneity of collaborative mission planning swarms. Mixed integer linear programming and dynamic network flow optimization models are only applicable to mission planning problems with a small scope, a small number of targets, and a single source of environmental threats. Alternatively, approaches abstract the multi-UAV collaborative problem into multiple independent multi-traveling salesman problems, optimizing for the shortest path length, and then solving them using a simulated annealing algorithm. While this approach reduces the complexity of the problem, it ignores the essential requirement for collaboration between individuals in swarm mission planning. Summary of the Invention

[0006] In view of this, the object of the present invention is to provide an efficient collaborative task planning method for heterogeneous UAV clusters.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] An efficient collaborative task planning method for a heterogeneous UAV cluster includes the following steps:

[0009] S1: Input the total number of heterogeneous swarm drones N1, the number of enemy ground targets N2, the role division interval time step T, the constraint strength ξ between the global strategy and the role strategy, and the number of predefined roles K;

[0010] S2: By characterizing each action in the action space and each drone's state, the roles of drones (i.e., intelligent agents) are divided;

[0011] S3: Based on the state observation of the UAV, the UAV role is dynamically selected and updated every T time steps;

[0012] S4: Generate and store trajectory data by sampling the role strategy, and initialize the training with the trajectory data to obtain a decision strategy for global collaborative planning;

[0013] S5: Based on the role strategy, trajectory data is continuously updated and trained to guide the global strategy. Finally, the global strategy is obtained and model reasoning is performed to achieve the generation and update of decision-making solutions for heterogeneous cluster collaborative task planning.

[0014] Furthermore, in step S2, the entire action space is divided into several subspaces to determine different roles. Based on the role-driven value decomposition method RODE, different actions are represented as action representations, which are supervised by the forward prediction model. A process representing the current agent observation is added as the input of the role strategy during learning and training. The model of the representation process is trained using the following loss function:

[0015]

[0016] Where θ o and θ a Represents the parameters of observation and action, p o and p r By e Representations serve as predictors of observations and actions; λ e represents the proportionality coefficient, a -i represents the set of other agent actions except its own actions, o i Represents self-observation, Embedded representations representing the actions of each agent; after obtaining the action embeddings, these embedded representations are clustered to generate different action subspaces, and then different roles are obtained among all agents.

[0017] Further, in step S3, by calculating the hidden vector and role embedding representation The dot product between them assigns the agents to different roles, where Through the hidden layer It is obtained by linear layer calculation. Calculated as:

[0018]

[0019] The character selection is updated by the following loss function, where the subscript and superscript “S” represent “selection”:

[0020]

[0021] in θ S ,ξ S Represent the parameters of the multilayer perceptron and GRU, the fully connected linear network with fixed-length embedded representation, and the hypernetwork respectively; γ represents the reward discount factor, They represent the global action value function when the agent selects the role, and ρ represents the set of agent roles.

[0022] Furthermore, in step S4, each agent maintains a global strategy network, and according to the result of the overall role division, maintains several different role strategy networks respectively; global strategy By parameters Indicates that the training goal is to maximize the global cumulative reward Role Strategy The network is parameterized Indicates that the training goal is to maximize the role reward

[0023] Establish and The correlation measure between them is used to evaluate and improve training efficiency, which is defined as follows:

[0024]

[0025] in represents the trajectory of agent i;

[0026] Based on the correlation metric, for the role strategy There is a cooperation-oriented objective function:

[0027]

[0028] Where ξ is a control The update range hyperparameter; the constraint strength is inversely proportional to the size of ξ: when the value of ξ is large, it means that the constraint on the correlation between the two policies is relatively weak. On the contrary, when ξ accepts a smaller value, it means that the constraint between the two policies is stronger.

[0029] For each agent's role-strategy Its original learning objective function based on the CTDE framework is:

[0030]

[0031] Using the cutting and constraint operations of the PPO algorithm, Limited to [1-ε,1+ε], ε represents the policy clipping coefficient.

[0032] Furthermore, in step S5, according to the role strategy and the global strategy, the role strategy is used to generate data trajectories to initialize the global strategy, and the mechanism of continuously guiding the global strategy training optimization is as follows:

[0033]

[0034] Where ζ represents the constraint strength, μ represents the reduction factor, represents the global advantage function.

[0035] The beneficial effects of the present invention are as follows: the present invention innovatively proposes an actor-critic multi-agent reinforcement learning algorithm based on role division to promote global strategy in UAV collaborative task planning. When solving the problem of collaborative planning of heterogeneous UAV clusters, by establishing role division in the multi-agent system of heterogeneous clusters, it is possible to dynamically and flexibly set the functions of each UAV in the overall task, thereby reducing the decision-making space and improving the efficiency of training and decision-making. At the same time, the present invention further establishes a real simulation environment for collaborative task planning of heterogeneous UAV clusters that takes into account the kinematic and dynamic characteristics of UAVs, and verifies the effect of collaborative task planning of UAV clusters under the proposed method in the environment, proving that the proposed method can overcome the problem of collaborative task planning of large-scale heterogeneous UAV clusters, and realize the maximization of the effectiveness of each UAV in the heterogeneous cluster and the generation of the most optimized collaborative task planning scheme. The method proposed by the present invention has advantages in both training efficiency and model effect, surpassing various baseline algorithms.

[0036] Other advantages, objectives and features of the present invention will be described in the following description and will be apparent to those skilled in the art to some extent, or those skilled in the art can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention provides the following drawings for illustration:

[0038] Figure 1 This is a diagram of the collaborative task planning of a heterogeneous UAV swarm, where (a) is a schematic diagram of the swarm collaborative task planning scenario, and (b) is a visualization diagram of the swarm collaborative task planning simulation environment.

[0039] Figure 2 Diagram of a multi-agent reinforcement learning framework where role policies guide global policies;

[0040] Figure 3 (a)-(d) in the figure are test result diagrams under four verification simulation environments. DETAILED DESCRIPTION

[0041] In this invention, the scenario of collaborative task planning of heterogeneous UAV cluster is as follows: Figure 1 As shown, Figure 1 As shown in the real collaborative mission planning simulation environment in (b), there are three types of drones with different functions, namely drones carrying reconnaissance payloads, strike payloads, and jamming payloads respectively. The three types of drones together constitute a heterogeneous drone cluster. Figure 1 (b) Correspondence of ground targets of different shapes Figure 1Different types of ground targets in (a). The overall collaborative mission goal of our heterogeneous drone swarm is to eliminate all enemy ground targets.

[0042] To achieve the aforementioned goal of collaborative mission planning for heterogeneous drone swarms, the actual scenario design for the two-way confrontation process is as follows: Our side uses a heterogeneous drone swarm carrying reconnaissance and strike payloads to perform target search, target strike, and damage assessment tasks. The specific tasks and mission objectives performed by each drone are generated by the drone swarm collaborative decision-making algorithm proposed in this invention. When the situation changes during mission execution—for example, if our drone detects an enemy target in an area where it may be present, our drone completes a strike on an enemy target, our drone completes a damage assessment of an already struck enemy target, or our drone is destroyed by the enemy's ground-based air defense system—the drone swarm quickly responds and intelligently generates a new collaborative decision-making action plan. The mission is completed when our drone swarm destroys all enemy targets; the mission fails when all our drones in the combat area are destroyed. Furthermore, if the collaborative planning time for our drone swarm exceeds a specified maximum duration, the mission is forced to fail.

[0043] Based on this, the present invention provides an efficient collaborative task planning method for heterogeneous drone swarms, and an actor-critic multi-agent reinforcement learning algorithm based on role-based division, where role strategies promote global strategies. For collaborative task scenarios involving heterogeneous drone swarms, this paper proposes a multi-agent reinforcement learning algorithm that uses role strategies to guide global strategies. A new dynamic game model for drone swarms is established, which divides the heterogeneous drone swarm into different roles based on the requirements of the scenario and tasks, reducing the decision-making space for each drone and improving training efficiency.

[0044] Multi-agent system refers to a group of self-controlled and interactive agents, which form a complete cooperation type, complete competition type or mixed type of overall system through sensor perception, actuator execution and other operations in the same environment. The reward obtained by each agent in the system will be affected by the actions of other agents. Reinforcement learning method is applied to multi-agent system, which can improve the cooperation ability of agents and the stability of decision model by using environmental information characteristics to generate gradient and learn the best strategy. The task planning of UAV cluster is essentially a kind of swarm intelligence technology. The UAV cluster needs to consider the coordination consistency among multiple UAVs and the constraint relationship of jointly executing tasks, and realizes efficient cooperation through information interaction and resource sharing, so as to achieve the global task index optimization or approximate optimization. The present application mainly solves how to apply the new method of multi-agent reinforcement learning to the heterogeneous UAV cluster cooperative task planning, so as to enhance the efficiency and optimality of the cluster cooperative task planning. Specifically, by modeling the role concept into the multi-agent reinforcement learning algorithm of Actor-Critic architecture, the role policy is established to enhance the global policy framework and other mechanisms, so as to improve the training efficiency and model performance of the heterogeneous UAV cluster cooperative task planning algorithm.

[0045] Based on the actor-critic framework in multi-agent reinforcement learning, a role policy enhanced multi-agent cooperative task framework is established. The algorithm as a whole contains three parts, role division, intra- and inter-role cooperation, and global policy learning of role policy enhancement. The way of guiding global policy by role policy innovatively realizes the optimization and training of global policy under the guidance of role policy and sparse reward. The overall architecture of the designed algorithm is shown in Figure 2 .

[0046] Considering that using individual exploration as the sampling method of global policy training initialization is less efficient for multi-agent cooperative task scenarios with large scale, the present application decides to use a more efficient method at the role level. Therefore, first, the role division mechanism is established, and the role allocation of each agent is completed (according to the time interval of each role redivision). Then, in the initial stage of model training, instead of pursuing the maximization of global reward, the local scene of each agent is taken as the starting point of training. After dividing the agents into multiple roles, the corresponding policy network of each role is initialized and maintained for cooperative behavior exploration and trajectory generation. Then, the reward function R:SxAxS→R K+1 is designed, which uses the role rewards of K roles to explore the cooperative behavior among agents at the role level and seek to realize the cooperation within and between roles. In addition, the reward of each role involves the cooperative behavior between different roles, which to some extent encourages the cooperation between roles.

[0047] To achieve the promotion of role policy to global policy training, the first step is to make accurate role division. The way of dividing the whole action space into several subspaces is chosen to determine different roles, and the role-driven value decomposition method, namely RODE algorithm, is referred to. In order to establish a stable division mechanism, unlike the original forward model in RODE, different actions are represented as action representations, supervised by a forward prediction model, and a representation process of the current agent observation is added as the input of the role policy during learning and training. Then, the model of the representation process can be trained by the following loss function:

[0048]

[0049] In the above formula, θ o and θ a represent the parameters of observation and action, p o and p r are represented by ξ e characterized as a predictor of observation and action. λ e represents the proportion coefficient, a -i represents the set of other agent actions except the self-action, o i represents the self-observation, represents the embedding representation of each agent action. After obtaining the action embedding, the embedding representations will be clustered to generate different action subspaces, and then different roles will be obtained among all agents. The role selection module assigns the agents to different roles by calculating the dot product between the hidden vector and the role embedding representation , where is obtained by linear layer calculation through the hidden layer , and can be calculated as:

[0050]

[0051] And the following loss function is designed to update the role selection module, where the subscript and superscript "S" represent "selection":

[0052]

[0053] where θ S ,ξ S represent the parameters of multi-layer perceptron and GRU, fully connected linear network with mapping fixed length embedding representation, and super network respectively. γ represents the reward discount factor, represent the global action value function when the agent selects the role. In addition, ρ represents the set of agent roles.

[0054] The entire method needs to be trained in two stages, because the initial sampling stage is crucial to improving training efficiency and the effectiveness of the final model. Therefore, there are two series of policy networks that need to be trained and maintained. Each agent needs to maintain a global policy network and, based on the overall role division results, maintain several different role policy networks. Global Policy By parameters Indicates that the training goal is to maximize the global cumulative reward The role strategy The network is parameterized Indicates that the training goal is to maximize the role reward Since the ultimate training goal is to obtain the optimal global strategy, we should establish and The correlation between them is used to evaluate and improve the training efficiency. So we define it as follows: represents the trajectory of agent i).

[0055]

[0056] On the one hand, based on the above related metrics, for the role strategy There is a cooperation-oriented objective function:

[0057]

[0058] Where ξ is a control The update range hyperparameter. The constraint strength is inversely proportional to the size of ξ: when the value of ξ is large, it means that the constraint on the correlation between the two policies is relatively weak. On the contrary, when ξ accepts a smaller value, it means that the constraint between the two policies is stronger. On the other hand, for each agent's role-policy Its original learning objective function based on the CTDE framework is:

[0059]

[0060] Referring to the cutting and constraint operations of the PPO algorithm, Limited to [1-ε,1+ε]. ε represents the policy clipping coefficient.

[0061] Then, according to the role strategy and the global strategy, the role strategy is used to generate data trajectories to initialize the global strategy, and the mechanism of continuously guiding the global strategy training optimization is as follows, where ζ represents the constraint strength and μ represents the reduction factor. represents the global advantage function.

[0062]

[0063] like Figure 3As shown in (a)-(d) in the figure, it was finally tested in a verification simulation environment, which proved that the method proposed in the present invention has advantages in both training efficiency and model effect, surpassing various baseline algorithms.

[0064] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. An efficient collaborative task planning method for heterogeneous UAV swarms, characterized by: The following steps are involved: S1: Input the total number of heterogeneous swarm drones N1, the number of enemy ground targets N2, the role division interval time step T, the constraint strength ξ between the global strategy and the role strategy, and the number of predefined roles K; S2: By characterizing each action in the action space and each drone's state, the roles of drones (i.e., intelligent agents) are divided; S3: Based on the state observation of the UAV, the UAV role is dynamically selected and updated every T time steps; S4: Generate and store trajectory data by sampling the role strategy, and initialize the training with the trajectory data to obtain a decision strategy for global collaborative planning; S5: Based on the role strategy, trajectory data is continuously updated and trained to guide the global strategy. Finally, the global strategy is obtained and model reasoning is performed to achieve the generation and update of decision-making solutions for heterogeneous cluster collaborative task planning.

2. The efficient collaborative task planning method for heterogeneous UAV clusters according to claim 1 is characterized by: In step S2, the entire action space is divided into several subspaces to determine different roles. Based on the role-driven value decomposition method RODE, different actions are represented as action representations, which are supervised by the forward prediction model. A process representing the current agent observation is added as the input of the role strategy during learning and training. The model of the representation process is trained using the following loss function: Where θ o and θ a Represents the parameters of observation and action, p o and p r By e Representations serve as predictors of observations and actions; λ e represents the proportionality coefficient, a -i represents the set of other agent actions except its own actions, o i Represents self-observation, Embedded representations representing the actions of each agent; After obtaining action embeddings, these embedded representations are clustered to generate different action subspaces, and then different roles are obtained among all agents.

3. The method for efficient collaborative task planning for heterogeneous UAV clusters according to claim 1 is characterized by: In step S3, by calculating the hidden vector and role embedding representation The dot product between them assigns the agents to different roles, where Through the hidden layer It is obtained by linear layer calculation. Calculated as: The character selection is updated by the following loss function, where the subscript and superscript "S" represent "selection": in θ S ,ξ S Represent the parameters of the multilayer perceptron and GRU, the fully connected linear network with fixed-length embedded representation, and the hypernetwork respectively; γ represents the reward discount factor, They represent the global action value function when the agent selects the role, and ρ represents the set of agent roles.

4. The method for efficient collaborative task planning for heterogeneous UAV clusters according to claim 1 is characterized by: In step S4, each agent maintains a global strategy network, and according to the result of the overall role division, maintains several different role strategy networks respectively; global strategy By parameters Indicates that the training goal is to maximize the global cumulative reward Role Strategy The network is parameterized Indicates that the training goal is to maximize the role reward Establish and The correlation measure between them is used to evaluate and improve training efficiency, which is defined as follows: in represents the trajectory of agent i; Based on the correlation metric, for the role strategy There is a cooperation-oriented objective function: Where ξ is a control The update range hyperparameter; the constraint strength is inversely proportional to the size of ξ: when the value of ξ is large, it means that the constraint on the correlation between the two policies is relatively weak. On the contrary, when ξ accepts a smaller value, it means that the constraint between the two policies is stronger. For each agent's role-strategy Its original learning objective function based on the CTDE framework is: Using the cutting and constraint operations of the PPO algorithm, Limited to [1-ε,1+ε], ε represents the policy clipping coefficient.

5. The method for efficient collaborative task planning for heterogeneous UAV clusters according to claim 1 is characterized by: In step S5, according to the role strategy and the global strategy, the role strategy is used to generate data trajectories to initialize the global strategy, and the mechanism of continuously guiding the global strategy training optimization is as follows: Where ζ represents the constraint strength, μ represents the reduction factor, represents the global advantage function.

Images