An unmanned aerial vehicle cooperative task planning method based on MA3C algorithm

Abstract

The present application relates to a kind of unmanned aerial vehicle cooperative task planning method based on MA3C algorithm, belong to unmanned aerial vehicle task planning field.The method includes the following steps: S1: establishing unmanned aerial vehicle model;S2: establishing task model;S3: modeling task allocation variable;S4: modeling task utility function;S5: modeling unmanned aerial vehicle cooperative task planning constraint condition;S6: determine unmanned aerial vehicle candidate task selection strategy;S7: modeling system state, action and reward;S8: modeling and training MA3C network;S9: based on MA3C algorithm determines unmanned aerial vehicle cooperative task allocation and trajectory planning strategy.The present application establishes unmanned aerial vehicle model and task model, and models task allocation variable and unmanned aerial vehicle trajectory planning variable, determines the optimal task allocation and trajectory planning strategy, to realize unmanned aerial vehicle cooperative task average utility maximization and the efficient application of unmanned aerial vehicle cooperative task planning.

Description

Technical Field

[0001] This invention belongs to the field of UAV mission planning and relates to a UAV collaborative mission planning method based on the MA3C algorithm. Background Technology

[0002] The accurate and reliable execution of UAV missions depends on a reasonable and efficient mission planning strategy. It is necessary to comprehensively analyze mission elements based on various constraints such as the environmental information perceived by the UAV, mission requirements, and onboard mission payload, optimize the scheduling and deployment of various resources, determine the UAV mission allocation and trajectory planning strategy, and ensure that the UAV completes the mission in the best way.

[0003] With the dynamic changes in the mission environment and the increasing complexity and diversity of mission requirements, the mission execution capabilities of a single UAV are severely limited by factors such as its functional type, onboard payload, flight capability, and battery power. Multi-UAV collaborative technology, through close cooperation between individual UAVs to collaboratively complete mission scheduling, can significantly and effectively improve the mission execution capabilities of UAVs and enhance system safety and reliability, and has become a trend in the development of UAV applications.

[0004] Current research on UAV collaborative mission planning includes: for underwater target search and tracking mission planning, modeling it as a problem of maximizing the search space and minimizing terminal error, and determining the mission planning strategy by jointly solving the optimization problem; for example, considering multi-aircraft mission execution scenarios in disaster relief, modeling the UAV path planning problem as a variant of the vehicle path problem to minimize the path length; however, existing research rarely considers the task allocation and the relationships between sub-tasks in multi-task scenarios, making it difficult to apply existing mechanisms efficiently. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a UAV collaborative task planning method based on the MA3C algorithm, which solves the technical problem that the low correlation between task allocation and sub-tasks in multi-task scenarios of UAVs makes it difficult to apply existing mechanisms efficiently.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A UAV cooperative mission planning method based on the MA3C algorithm includes the following steps:

[0008] S1: Establish a drone model, including a mission resource model and a drone mission capability model;

[0009] S2: Establish task models, including independent task models and combined task models, and use a directed acyclic graph (DAG) to describe the relationships between subtasks in the combined task model;

[0010] S3: Variables assigned to the modeling task;

[0011] S4: Modeling task utility function;

[0012] S5: Model the constraints for UAV collaborative mission planning, including task constraint modeling and UAV constraint modeling;

[0013] The task constraint modeling includes: task allocation constraint modeling, task execution constraint modeling, and task completion time constraint modeling;

[0014] The UAV constraint modeling includes: UAV performance constraint modeling and UAV safety constraint modeling;

[0015] The modeling of UAV performance constraints includes: UAV energy constraint modeling and UAV mission resource constraint modeling;

[0016] The modeling of UAV safety constraints includes: UAV flight distance constraint modeling, UAV safety constraint modeling, and UAV no-fly zone constraint modeling.

[0017] S6: Determine the drone candidate task selection strategy based on task resource availability, task timeliness, and drone energy consumption limitations;

[0018] S7: Modeling the system's state, actions, and rewards in time slots;

[0019] S8: Model and train the MA3C algorithm;

[0020] S9: Determine the UAV collaborative task allocation and trajectory planning strategy based on the MA3C algorithm.

[0021] Furthermore, in S1, establishing a drone model specifically includes: N drones, defining U... n Let U be the nth UAV, 1≤n≤N; divide the system time into time slots of equal length, define T as the total number of time slots, and τ as the length of each time slot; the UAV's flight altitude is fixed at h, and define time slot t as U. n The two-dimensional coordinates are

[0022] The drone departs from its initial position to perform a mission and returns to its destination before running out of energy. The drone's maximum speed is v. max ;make U n In the flight direction of time slot t, v n,t ∈[0,v max ] represents U n Flight speed in time slot t; U n The projection at the two-dimensional position in time slot t+1 is: make Represents the initial time U n The energy required for a drone to perform a mission; let Y represent the quantity of the various resource types needed to perform the mission. Represents the initial time U n The amount of the y-th type of resource carried, 1≤y≤Y.

[0023] Furthermore, in S2, establishing a task model specifically includes: the system contains M tasks, and defining Θ. m For the m-th task, 1≤m≤M, it is represented by the quadruple. It means that s m =[s m,1 ,...,s m,Y ] T Indicates full execution Θ m Required task resources, s m,y Indicates execution Θ m The required amount of the y-th type of resource; R m For the drone to fully execute Θ m Available benefits; Indicates the deadline for task execution; task types include independent tasks and combined tasks;

[0024] Independent tasks cannot be broken down into subtasks; the corresponding benefits can only be obtained when the entire task is fully executed.

[0025] A composite task consists of multiple independently executable subtasks, K m Represents Θ m The number of subtasks, Θ m,i Represents Θ m The i-th subtask, 1≤i≤K m ;

[0026] If Θ m If it is an independent task, then K m =1,Θ m =Θ m,i ;

[0027] If Θ m For combined tasks, then K m >1; Subtask Θ m,i Model as a pair It means that s m,i =[s m,i,1 ,...,s m,i,y ,...s m,i,Y ] T Represents Θ m,i Required task resources, s m,i,y Indicates execution Θ m,iThe required amount of the y-th type of resource, R m,i To fully execute Θ m,i The potential gains; Θ m The two-dimensional coordinates are represented as q m =[x m ,y m ] T ;

[0028] A directed acyclic graph (DAG) is used to describe the relationships between the subtasks of the combined task; let G... m =(V m E m W m ) represents the combined task Θ m A DAG, in which V m ={Θ m,i} represents a set of nodes; E m ={E m,i,j |1≤i≠j≤K m} represents the set of relationships between subtasks, E m,i,j Represents the connection and combination task Θ m Subtasks Θ m,i With Θ m,j The edge; W m ={ω m,i,j} represents the set of weights for the relationships between subtasks, where ω m,i,j ∈{0,1} represents the combined task Θ m Subtasks Θ m,i With Θ m,j The relationship between them, ω m,i,j =1 indicates subtask Θ m,i With subtasks Θ m,j There is a direct dependency between them, i.e., subtasks Θ m,j It needs to be done in subtask Θ m,i Execute after execution is complete; otherwise, ω m,i,j =0.

[0029] Furthermore, in S3, the modeling task allocation variable model specifically includes: Let α n,m,i,t ∈{0,1} represents the drone task assignment variable;

[0030] If α n,m,i,t =1, then it means U n Execute subtask Θ in time slot t m,i Otherwise, α n,m,i,t = 0, 1≤n≤N, 1≤m≤M, 1≤i≤K m ,0≤t≤T.

[0031] Furthermore, in S4, modeling the task utility function model specifically includes:

[0032] The task utility function for time slot t is expressed as:

[0033]

[0034] Where ε1 and ε2 are the weighting coefficients for mission benefits and drone energy consumption, respectively. Indicates the rewards for completing the task. This represents the energy consumed by the UAV in time slot t;

[0035] Will Modeling is performed, represented as:

[0036]

[0037] Where, β m,i,t ={0,1} represents the task execution identifier. β m,i,t =1 indicates that the drone executes Θ in time slot t. m,i Otherwise, β m,i,t =0;1 {x} This indicates an indicator function; if the logical expression {x} is true, then 1... {x} =1, otherwise 1 {x} =0;

[0038] Will Modeling is performed, represented as:

[0039]

[0040]

[0041] Among them, E n,t U represents the t-th time slot n Energy consumed U represents the t-th time slot n Consumed flight energy U represents the t-th time slot n Energy consumed in performing the task;

[0042] Will Modeling is performed, represented as:

[0043]

[0044] Where P0 and P0′ are constants, U tip v0 is the tip velocity of the rotor blades, v0 is the average rotor induced velocity when the UAV is hovering, and ξ is the tip velocity of the rotor blades. d and ξ r These represent the fuselage drag ratio and rotor reliability, respectively.air and S r These are air density and rotor disk area, respectively;

[0045] Will Modeling is performed, represented as:

[0046]

[0047] in, U n The basic energy consumed when using the y-th resource.

[0048] Furthermore, in S5, the establishment of a UAV collaborative task planning constraint model specifically includes:

[0049] The task allocation constraints are modeled and represented as follows:

[0050]

[0051]

[0052]

[0053] in, Represents the drone collaboration variables for subtasks;

[0054] like but otherwise,

[0055] The constraints for task execution are modeled and represented as follows:

[0056]

[0057] in, Subtask Θ m,i Start time;

[0058] The task completion time constraint is modeled as follows:

[0059]

[0060] The energy constraint conditions for the UAV are modeled and expressed as follows:

[0061]

[0062]

[0063] in, E represents the remaining available energy in the t-th time slot Un. th This indicates the threshold value for the remaining energy of the drone;

[0064] The resource constraints of the drone are modeled and represented as follows:

[0065]

[0066]

[0067]

[0068] in, U represents the t-th time slot n The remaining amount of the y-th resource, s m,i,y,t Represents the t-th time slot subtask Θ m,i The amount of the yth type of resource still needed. U n The amount of resource type y consumed in a time slot;

[0069] The flight distance constraint of the UAV is modeled and represented as follows:

[0070]

[0071] in, U-shaped drone n Maximum flight speed;

[0072] The safety constraints of drones are modeled and represented as follows:

[0073]

[0074] in, This indicates the minimum safe distance between drones;

[0075] Modeling the no-fly zone constraints for drones, as follows:

[0076]

[0077] Here, Γ is the set of all points in the region containing the threat.

[0078] Furthermore, in step S6, a candidate task selection strategy for the UAV is determined based on the availability of task resources, the timeliness of the task, and the energy consumption limitations of the UAV, specifically as follows:

[0079] Let Φ n U-shaped drone n The candidate task set is initialized to

[0080] The conditions for mission resource availability are: Based on the resources carried by the UAV and the resource requirements of the mission, determine whether the UAV meets the resource availability conditions. If the UAV...n Execute only subtasks starting from the initial position. m,i ,like Then U n Subtasks Θ can be satisfied m,i Resource availability conditions;

[0081] The timeliness condition for the mission is: based on the UAV mission execution time and mission deadline requirements, determine whether the UAV meets the mission timeliness condition. If the UAV... n Execute only subtasks starting from the initial position. m,i Calculate the required time, denoted as . like Then U n Subtasks Θ can be satisfied m,i Timeliness requirements;

[0082] The conditions for limiting the energy consumption of drones are as follows: Based on the available energy of the drone and the energy required to perform the mission, determine whether the drone meets the mission energy consumption limit. If the drone U n Execute only subtasks starting from the initial position. m,i And return to the destination to calculate U. n The corresponding energy consumption is denoted as like Then U n Satisfying subtasks Θ m,i Energy consumption constraints;

[0083] If the drone U n Simultaneously satisfy subtasksΘ m,i Given the time and energy consumption constraints, then the subtask Θ m,i For U drones n Candidate tasks, update U n The candidate task set is Φ n =Φ n ∪{Θ m,i}

[0084] Furthermore, in S7, the state space model of the system in time slot t is represented as:

[0085]

[0086] in, For drone location collection, This is the collection of remaining available energy for the drone. J represents the collection of drone resources. t ={β m,i,t} represents a set of task execution identifiers;

[0087] Modeling the joint action space of the system in time slot t, we can express it as follows:

[0088] a t ={a 1,t ,...,a n,t ,...,a N,t} (twenty one)

[0089] in, Indicates time slot U n The action space includes flight speed, flight direction, and task assignment variables;

[0090] The excitation model for the system in time slot t is expressed as:

[0091] r t =U t (twenty two).

[0092] Furthermore, in S8, modeling and training the MA3C algorithm specifically includes:

[0093] Treat N drones as N intelligent agents, and construct an actor network and a critic network for each agent, using parameters θ and θ'. v Initialize the global actor network and the global critic network using parameters θ′ and θ. v Initialize the actor network and critic network, synchronizing parameters θ′ = θ, θ v ′=θ v Each intelligent agent actor network adjusts its actions based on the current state of the environment. t Network parameters θ′ and strategy π(a) t |s t ;θ) Select action a t , obtain state s t Value V(s) t ;θ v ) and instant rewards r t and transition to the next state s t+1 ;

[0094] Updating the actor network using the loss function is expressed as:

[0095] f π (θ)=logπ(a t |s t ;θ)A(s t ,a t )+cH(π(s t ,θ)) (23)

[0096]

[0097] Where A(s) t a t H(π(s)) is the advantage function used to evaluate the advantage of the current state-action pair.t ,θ)) is the policy cross-entropy, and c is the entropy coefficient, which is used to control the regularization strength of the entropy;

[0098] The commentator network is updated using the loss function, expressed as:

[0099] f v (θ)=A(s t ,a t ) 2 (25)

[0100] The parameters of the actor network and the critic network are updated based on the accumulated gradient, as follows:

[0101]

[0102]

[0103] Update the global actor network parameters and the global critic network parameters, as shown below:

[0104]

[0105]

[0106] Where η is the learning rate, ε is a very small positive number, and the estimated slope is g = αg + (1-α)Δθ. 2 α is a momentum; estimate the slope.

[0107] Furthermore, in S9, the strategy for determining the UAV cooperative task allocation and trajectory planning based on the MA3C algorithm specifically includes:

[0108] Under the conditions of satisfying task association, task allocation, and UAV performance constraints, the UAV cooperative task allocation and trajectory planning strategy is optimized and determined with the goal of maximizing the system's average excitation.

[0109]

[0110] in, and These are the optimal task allocation strategy and the trajectory planning strategy.

[0111] The beneficial effects of this invention are as follows:

[0112] This invention targets scenarios involving multiple heterogeneous drones and multiple tasks. The task types include independent tasks and combined tasks. Independent tasks cannot be broken down into subtasks, while combined tasks consist of multiple independently executable subtasks. These subtasks have certain relationships with each other. The average utility function is modeled and maximized as the optimization objective to achieve joint optimization of task allocation and drone trajectory planning, thereby realizing the efficient application of drone collaborative task planning.

[0113] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0114] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0115] Figure 1 Schematic diagram of a scenario for planning drone collaborative tasks;

[0116] Figure 2 This is a flowchart illustrating the method described in this invention. Detailed Implementation

[0117] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0118] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0119] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0120] Please see Figures 1-2 This is a collaborative task planning method for unmanned aerial vehicles (UAVs) based on the MA3C algorithm.

[0121] Assuming the scenario involves multiple heterogeneous drones and multiple tasks, the task types include independent tasks and combined tasks. Independent tasks cannot be broken down into subtasks, while combined tasks consist of multiple independently executable subtasks with certain interrelationships. The optimization objective is to maximize the average utility function to achieve joint optimization of task allocation and drone trajectory planning.

[0122] Figure 1 The diagram illustrates a scenario for planning collaborative tasks using drones. As shown, the network contains multiple heterogeneous drones and multiple tasks. The task types include independent tasks and combined tasks. Independent tasks cannot be broken down into sub-tasks, while combined tasks consist of multiple independently executable sub-tasks. These sub-tasks have certain relationships with each other, and drones need to fly above the tasks to perform them. The average utility function is maximized by optimizing task allocation and drone trajectory planning.

[0123] Figure 2 The figure shows a flowchart of the method described in this invention. The method specifically includes the following steps:

[0124] Step 1: Create a drone model

[0125] Building a drone model specifically includes: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] n Let N represent the nth drone, where 1 ≤ n ≤ N, and N is the number of drones.

[0126] The system time is divided into time slots of equal length, let T represent the total number of time slots, and let the length of each time slot be τ; the UAV's flight altitude is fixed at h, let t be the time slot U. n The two-dimensional coordinates are The drone departs from its initial position to perform its mission and returns to its destination before running out of energy; the drone's maximum speed is v. max ;make Un In the flight direction of time slot t, v n,t ∈[0,v max ] represents U n Flight speed in time slot t; U n The projection at the two-dimensional position in time slot t+1 is: make Represents the initial time U n The energy required for a drone to perform a mission; let Y represent the quantity of the various resource types needed to perform the mission. Represents the initial time U n The amount of the y-th type of resource carried, 1≤y≤Y.

[0127] Step 2: Establish a task model

[0128] Establishing a task model specifically includes: the system contains M tasks, let Θ m Let m be the m-th task, 1 ≤ m ≤ M, represented by a quadruple. It means that s m =[s m,1 ,...,s m,Y ] T Indicates full execution Θ m Required task resources, s m,y Indicates execution Θ m The required amount of the y-th type of resource, R m For the drone to fully execute Θ m Available benefits This indicates the deadline for task execution; task types include independent tasks and combined tasks. Independent tasks cannot be broken down into subtasks, and the corresponding benefits are only obtained when the entire task is fully executed; combined tasks consist of multiple independently executable subtasks, K. m Represents Θ m The number of subtasks, Θ m,i Represents Θ m The i-th subtask, 1≤i≤K m If Θ m If it is an independent task, then K m =1,Θ m =Θ m,i If Θ m For combined tasks, then K m >1; Subtask Θ m,i Model as a pair It means that s m,i =[s m,i,1 ,...,s m,i,y ,...s m,i,Y ] T Represents Θ m,iRequired task resources, s m,i,y Indicates execution Θ m,i The required amount of the y-th type of resource, R m,i To fully execute Θ m,i The potential gains; Θ m The two-dimensional coordinates are represented as q m =[x m ,y m ] T ;

[0129] A directed acyclic graph (DAG) is used to describe the relationships between the subtasks of the combined task; let G... m =(V m E m W m ) represents the combined task Θ m A DAG, in which V m ={Θ m,i} represents a set of nodes; E m ={E m,i,j |1≤i≠j≤K m} represents the set of relationships between subtasks, E m,i,j Represents the connection and combination task Θ m Subtasks Θ m,i With Θ m,j The edge; W m ={ω m,i,j} represents the set of weights for the relationships between subtasks, where ω m,i,j ∈{0,1} represents the combined task Θ m Subtasks Θ m,i With Θ m,j The relationship between them, ω m,i,j =1 indicates subtask Θ m,i With subtasks Θ m,j There is a direct dependency between them, i.e., subtasks Θ m,j It needs to be done in subtask Θ m,i Execute after execution is complete; otherwise, ω m,i,j =0.

[0130] Step 3: Assign variables to the modeling task

[0131] The modeling task assignment variables specifically include: Let α n,m,i,t ∈{0,1} represents the drone task assignment variable, if α n,m,i,t =1, indicating U n Execute subtask Θ in time slot t m,i Otherwise, α n,m,i,t = 0, 1≤n≤N, 1≤m≤M, 1≤i≤K m ,0≤t≤T.

[0132] Step 4: Model the task utility function

[0133] The modeling task utility function specifically includes: the time slot t task utility function. Where ε1 and ε2 are the weighting coefficients for mission benefits and drone energy consumption, respectively. Representing the reward for task execution, modeled as Where, β m,i,t ={0,1} represents the task execution identifier. β m,i,t =1 indicates that the drone executes Θ in time slot t. m,i Otherwise, β m,i,t =0;1 {x} This indicates an indicator function; if the logical expression {x} is true, then 1... {x} =1, otherwise 1 {x} =0; This represents the energy consumed by the UAV in time slot t, according to the formula... calculate Where E n,t U represents the t-th time slot n The energy consumed, according to the formula Calculate E n,t ,in, U represents the t-th time slot n The formula for calculating the consumed flight energy is:

[0134] Where P0 and P0′ are constants, U tip v0 is the tip velocity of the rotor blades, v0 is the average rotor induced velocity when the UAV is hovering, and ξ is the tip velocity of the rotor blades. d and ξ r These represent the fuselage drag ratio and rotor reliability, respectively. air and S r These are air density and rotor disk area, respectively; U represents the t-th time slot n The energy consumed in performing a task is modeled as in U n The basic energy consumed when using the y-th resource.

[0135] Step 5: Model the constraints for UAV collaborative task planning

[0136] The specific constraints for modeling UAV collaborative task planning include: task allocation constraints modeled as follows: in Denotes the drone cooperative variables of the subtask, if but otherwise, Task execution constraints are modeled as in Subtask Θ m,i Start time; task completion time constraints are modeled as follows:

[0137] Unmanned aerial vehicle (UAV) constraints include UAV performance constraints and UAV safety constraints; UAV performance constraints include UAV energy constraints and UAV mission resource constraints. The UAV energy constraint conditions are modeled as follows: in, U represents the t-th time slot n The remaining usable energy, according to the formula calculate E th This represents the remaining energy threshold of the drone; the drone resource constraint model is as follows: U represents the t-th time slot n The remaining amount of the y-th resource is modeled as follows: U n The amount of resource type y consumed in a time slot; s m,i,y,t Represents the t-th time slot subtask Θ m,i The amount of the y-th type of resource still needed is modeled as follows: Modeling the flight distance constraint of the UAV as in, U-shaped drone n Maximum flight speed; UAV safety constraints modeled as in, This represents the minimum safe distance between drones, and the drone no-fly zone constraint model is as follows: Where Γ is the set of all points in the region containing the threat.

[0138] Step Six: Determine the UAV candidate mission selection strategy

[0139] The specific strategies for determining candidate drone missions include:

[0140] a) Initialization: Let Φ n U-shaped drone n The candidate task set is initialized to

[0141] b) Determine candidate tasks based on the availability of task resources, the timeliness of the task, and the energy consumption limitations of the UAV;

[0142] c) Resource Availability Conditions: Determine whether the UAV meets the resource availability conditions based on the resources carried by the UAV and the resource requirements of the mission. Specifically, if the UAV U... n Execute only subtasks starting from the initial position. m,i ,like Then Un Subtasks Θ can be satisfied m,i Resource availability conditions;

[0143] d) Mission timeliness requirements: Determine whether the UAV meets the mission timeliness requirements based on the UAV mission execution time and deadline requirements. Specifically, if the UAV... n Execute only subtasks starting from the initial position. m,i Calculate the required time, denoted as . like Then U n Subtasks Θ can be satisfied m,i Timeliness requirements;

[0144] e) Drone Energy Consumption Limitations: Based on the drone's available energy and the energy required to perform the mission, determine whether the drone meets the mission energy consumption limits. Specifically, if the drone U... n Execute only subtasks starting from the initial position. m,i And return to the destination to calculate U. n The corresponding energy consumption is denoted as like Then U n Satisfying subtasks Θ m,i Energy consumption constraints;

[0145] f) If the drone U n Simultaneously satisfy subtasksΘ m,i Given the time and energy consumption constraints, then the subtask Θ m,i For U drones n Candidate tasks, update U n The candidate task set is Φ n =Φ n ∪{Θ m,i};

[0146] Step 7: Model the system state, actions, and rewards

[0147] The modeling system's state, actions, and rewards specifically include: the modeling system's state space in time slot t. in For drone location collection, This is the collection of remaining available energy for the drone. J represents the collection of drone resources. t ={β m,i,t} represents the set of task execution identifiers; modeling the joint action space a of the time slot t system. t ={a 1,t ,...,a n,t ,...,a N,t},in Indicates time slot U nThe action space includes flight speed, flight direction, and task assignment variables. The reward function r of the modeling system in time slot t is... t =U t .

[0148] Step 8: Model and train the MA3C algorithm

[0149] Modeling and training the MA3C algorithm specifically involves: treating N drones as N agents, and constructing an actor network and a critic network for each agent, using parameters θ and θ'. v Initialize the global actor network and the global critic network using parameters θ′ and θ′. v Initialize the actor network and critic network, synchronizing parameters θ′ = θ, θ′ v =θ v Each intelligent agent actor network adjusts its actions based on the current state of the environment. t Network parameters θ′ and strategy π(a) t |s t ;θ) Select action a t , obtain state s t Value V(s) t ;θ v ) and instant rewards r t and transition to the next state s t+1 ;

[0150] Using the loss function f π (θ)=logπ(a t |s t ;θ)A(s t ,a t )+cH(π(s t The actor network is updated using the θ) function, where the advantage function is defined as follows: Used to evaluate the advantage of the current state-action pair, H(π(s) t ,θ)) represents the policy cross-entropy, and c is the entropy coefficient used to control the regularization strength of the entropy; using the loss function f v (θ)=A(s t ,a t ) 2 Update the critic network; based on the accumulated gradients, update the parameters of the actor network and the critic network, specifically as follows: Update global parameters, specifically: Where η is the learning rate, ε is a very small positive number, and the estimated slope is g = αg + (1-α)Δθ. 2 α is a momentum. Wherein, the estimated slope g′=αg′+(1-α)Δθ v 2 ;

[0151] Step 9: Determine UAV task allocation and trajectory planning strategies based on the MA3C algorithm.

[0152] The specific steps for determining UAV task allocation and trajectory planning strategies based on the MA3C algorithm include: optimizing and determining the UAV task allocation and trajectory planning strategies with the objective of maximizing the system's average reward, while satisfying the constraints of task association, task allocation, and UAV performance.

[0153]

[0154] in, and These are the optimal task allocation strategy and the trajectory planning strategy, respectively.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A UAV cooperative mission planning method based on the MA3C algorithm, characterized in that: The method includes the following steps: S1: Establish a drone model, including a mission resource model and a drone mission capability model; S2: Establish task models, including independent task models and combined task models, and use a directed acyclic graph (DAG) to describe the relationships between subtasks in the combined task model; S3: Variables assigned to the modeling task; S4: Modeling task utility function; S5: Model the constraints for UAV collaborative mission planning, including mission constraint modeling and UAV constraint modeling; The task constraint modeling includes: task allocation constraint modeling, task execution constraint modeling, and task completion time constraint modeling; The UAV constraint modeling includes: UAV performance constraint modeling and UAV safety constraint modeling; the UAV performance constraint modeling includes: UAV energy constraint modeling and UAV mission resource constraint modeling. The modeling of UAV safety constraints includes: UAV flight distance constraint modeling, UAV safety constraint modeling, and UAV no-fly zone constraint modeling. S6: Determine the drone candidate task selection strategy based on task resource availability, task timeliness, and drone energy consumption limitations; S7: Modeling the system's state, actions, and rewards in time slots; S8: Model and train the MA3C algorithm, specifically including: Treat N drones as N intelligent agents, and construct an actor network and a critic network for each agent, using parameters... and Initialize the global actor network and the global critic network using parameters. and Initialize the actor network and critic network, and synchronize parameters. , Each intelligent agent actor network adjusts its actions based on the current state of the environment. Network parameters and strategies Select Action , get the state value and instant rewards and transition to the next state. ; Updating the actor network using the loss function is represented as: （23） （24） in, The advantage function is used to evaluate the advantage of the current state-action pair. For policy cross-entropy, This is the entropy coefficient, used to control the strength of entropy regularization. , Discount factor; The commentator network is updated using the loss function, expressed as: （25） The parameters of the actor network and the critic network are updated based on the accumulated gradient, as follows: （26） （27） Update the global actor network parameters and the global critic network parameters, as shown below: （28） （29） in, It's the learning rate. The estimated slope of the actor network is a very small positive number. , For a momentum; the estimated slope of the commentator network ; S9: Determine the UAV collaborative task allocation and trajectory planning strategy based on the MA3C algorithm.

2. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 1, characterized in that: In step S1, establishing the drone model specifically includes: N A drone, defined U n For the first n A drone, Divide the system time into time slots of equal length and define... T is Total number of time slots For each time slot length; the drone's flight altitude is fixed at... h ,definition t Time slot U n The two-dimensional coordinates are ; The drone departs from its initial position to perform the mission and returns to its destination before running out of energy. The drone's maximum speed is... ;make U n In the time slot t Flight direction U n In the time slot t Flight speed; U n In the time slot t The two-dimensional position projection of +1 is: .

3. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 2, characterized in that: In step S2, establishing a task model specifically includes: the system containing... M One task, defined For the first m One task, , by quadruple It means that among them Indicates full execution Required task resources Indicate execution The required first The amount of resources of each type of resource; For drones to fully execute Available benefits; Indicates the deadline for task execution; task types include independent tasks and combined tasks; Independent tasks cannot be broken down into subtasks; the corresponding benefits can only be obtained when the entire task is fully executed. A composite task consists of multiple independently executable subtasks. express The number of subtasks express The i Sub-tasks ; like For an independent task, then , ; like To combine tasks, then Subtask Model as a pair It means that, among them express Required task resources Indicate execution The required first y The amount of resources of each type of resource. To fully execute The potential gains; The two-dimensional coordinates are represented as ; A directed acyclic graph (DAG) is used to describe the relationships between the subtasks of the combined task; let... Representing combined tasks The DAG, in which, Represents a set of nodes; This represents the set of relationships between subtasks. Indicates connection composition task subtasks and The edge; This represents the set of weights representing the relationships between subtasks, where... Representing combined tasks subtasks and The relationship between them Subtasks sub-tasks There is a direct dependency between them, i.e., subtasks. Need to be done in subtasks Execute after execution is complete, otherwise, .

4. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 3, characterized in that: In S3, the modeling task assigns variable models, specifically including: Let Represents the variable for assigning drone tasks; like Then it means U n Execute the subtask in time slot t. ,otherwise, , .

5. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 4, characterized in that: In S4, the modeling task utility function model specifically includes: Time slot t The task utility function is expressed as: （1） in, and These are the weighting coefficients for mission benefits and drone energy consumption, respectively. Indicates the rewards for completing the task. Indicates time slot t The energy consumed by drones; Will Modeling is performed, represented as: （2） in, Indicates the task execution identifier. , Indicates that the drone is in the time slot t implement ,otherwise, ; Indicates an indicator function, if the logical expression If true, then ,otherwise ; Will Modeling is performed, represented as: （3） （4） in, Indicates the first t Each time slot U n Energy consumed Indicates the first t Each time slot U n Consumed flight energy Indicates the first t Each time slot U n Energy consumed in performing the task; Will Modeling is performed, represented as: （5） in and It is a constant. Let be the tip velocity of the rotor blade. The average rotor-induced speed when the drone is hovering. and These are the fuselage drag ratio and rotor reliability, respectively. and These are air density and rotor disk area, respectively; Will Modeling is performed, represented as: （6） in, U n Execute using the y The basic energy consumed by a resource.

6. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 5, characterized in that: In S5, the model for establishing constraints for UAV collaborative task planning is established, specifically including: The task allocation constraints are modeled and represented as follows: （7） （8） （9） in, Represents the drone collaboration variables for subtasks; like ,but ,otherwise, , ; The constraints for task execution are modeled and represented as follows: （10） in, Subtasks Start time; The task completion time constraint is modeled as follows: （11） The energy constraint conditions for the UAV are modeled and expressed as follows: ， （12） （13） in, This represents the remaining available energy in the t-th time slot Un. This indicates the threshold value for the remaining energy of the drone; The resource constraints of the drone are modeled and represented as follows: （14） （15） （16） in, Indicates the first t Each time slot U n The remaining number y The amount of such resources Indicates the first t Subtasks in time slots The first one still needed y The amount of such resources U n The first time slot consumed y Seed resource quantity; The flight distance constraint of the UAV is modeled and represented as follows: （17） in, U-shaped drone n Maximum flight speed; The safety constraints of drones are modeled and represented as follows: ， （18） in, This indicates the minimum safe distance between drones; Modeling the no-fly zone constraints for drones, as follows: （19） in, It is the set of all points in the area containing the threat.

7. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 6, characterized in that: In step S6, a candidate task selection strategy for the UAV is determined based on the availability of task resources, the timeliness of the task, and the energy consumption limitations of the UAV. Specifically: make Indicates drone The candidate task set is initialized to ; The conditions for mission resource availability are: Based on the resources carried by the UAV and the resource requirements of the mission, determine whether the UAV meets the resource availability conditions. If the UAV... n Execute only subtasks starting from the initial position. ,like , then U n Subtasks can be satisfied Resource availability conditions; The timeliness condition for the task is: based on the drone task execution time and task deadline requirements, determine whether the drone meets the timeliness condition. If the drone U n Execute only subtasks starting from the initial position. Calculate the required time, denoted as . ,like , then U n Subtasks can be satisfied Timeliness requirements; The conditions for limiting the energy consumption of drones are as follows: Based on the available energy of the drone and the energy required to perform the mission, determine whether the drone meets the mission energy consumption limit. If the drone U n Execute only subtasks starting from the initial position. And return to the destination to calculate U n The corresponding energy consumption is denoted as ,like , then U n Satisfy subtasks Energy consumption constraints; If the drone U n Simultaneously satisfy subtasks Given the time and energy consumption constraints, the subtasks For U drones n Candidate tasks, update U n The candidate task set is .

8. The UAV cooperative mission planning method based on the MA3C algorithm according to claim 7, characterized in that: In S7, the system is in time slots t The state-space model is represented as: （20） in, For drone location collection, This is the collection of remaining available energy for the drone. This represents a collection of drone resources. A collection representing task execution identifiers; For the system in time slots t The joint action space model is represented as: （21） in, express t Time slot U n The action space includes flight speed, flight direction, and task assignment variables; The excitation model for the system in time slot t is expressed as: （22）。 9. A UAV cooperative mission planning method based on the MA3C algorithm according to claim 8, characterized in that: In step S9, the strategy for determining UAV cooperative task allocation and trajectory planning based on the MA3C algorithm specifically includes: Under the conditions of satisfying task association, task allocation, and UAV performance constraints, the UAV cooperative task allocation and trajectory planning strategy is optimized and determined with the goal of maximizing the system's average excitation. （30） in, , and These are the optimal task allocation strategy and the trajectory planning strategy.

Images