Distributed dynamic cooperation method and system for unmanned aerial vehicle group

Abstract

The invention discloses an unmanned aerial vehicle group distributed dynamic cooperation method and system, and relates to the technical field of unmanned aerial vehicles. The method comprises the following steps: firstly, establishing an unmanned aerial vehicle group Markov game model, and defining an unmanned aerial vehicle state, an environment state, a task state and a comprehensive state matrix of each unmanned aerial vehicle; then, collecting real unmanned aerial vehicle task data, counting resource demands, initializing an unmanned aerial vehicle group environment, and constructing a simulation environment; unmanned aerial vehicle actions and probabilities are generated through an action strategy neural network, task priorities and actions are generated in combination with a task and environment state matrix, combined actions are formed, and a reward value is calculated. The value evaluation neural network evaluates the current action value, and stores the data to generate a trajectory. And constructing a loss function of the action strategy and the value evaluation network for parameter updating. After network parameters are updated, strategy similarity check is carried out, and a training strategy is adjusted if necessary. The efficient collaborative operation of the unmanned aerial vehicle group is realized, and the task execution efficiency and the resource utilization rate are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned aerial vehicles, and in particular relates to a distributed dynamic collaboration method and system for a group of unmanned aerial vehicles. Background Art

[0002] With the rapid development of drone technology, drones are increasingly used in various fields, including military reconnaissance, disaster monitoring, smart city management, etc. In these application scenarios, multiple drones are often required to work together to complete complex tasks. Therefore, how to achieve effective collaboration between drones has become a hot topic and difficulty in current research.

[0003] The challenges faced by drone collaboration are as follows:

[0004] Environmental complexity: When performing tasks, drones often face complex and changing environments, including obstacles, weather conditions, communication status, etc. These factors pose great challenges to the flight and coordination of drones.

[0005] Mission diversity: Different missions have different requirements for drones, including the location, type, priority, and deadline of the mission. How to achieve effective switching and coordination between multiple missions is an urgent problem to be solved.

[0006] Limited resources: UAVs have limited resources such as battery power and task queue length. How to achieve effective coordination and resource allocation between UAVs under limited resources is another important challenge.

[0007] At present, although some methods and technologies for UAV collaboration have been proposed, they often have some limitations. For example, some methods are only applicable to specific environments and tasks, lacking versatility and flexibility; some methods are insufficient in computational complexity and real-time performance, making it difficult to meet the needs of practical applications.

[0008] The present invention proposes a distributed dynamic collaboration method for UAV swarms based on a Markov game model, aiming to solve the above problems and improve the collaboration efficiency and task completion quality of UAV swarms by introducing reinforcement learning and distributed control technology. Summary of the invention

[0009] In order to overcome the shortcomings and deficiencies of the above-mentioned prior art, the first purpose of the present invention is to provide a distributed dynamic collaboration method for drone swarms; the second purpose of the present invention is to provide a distributed dynamic collaboration system for drone swarms.

[0010] The first object of the present invention adopts the following technical solution:

[0011] A distributed dynamic collaboration method for drone swarms, the process is as follows:

[0012] Step 1: Establish a Markov game model for distributed dynamic coordination of drone swarms, including defining drone status, environment status, task status, and the environment status of each drone;

[0013] The drone status includes the drone's location, speed, battery power, task queue length, and waiting time of the task at the head of the task queue; the environmental status includes the location of obstacles in the environment, weather conditions, and communication status between drones; the task status includes the location, type, priority, and deadline of the task; the environmental status of each drone is a comprehensive state matrix formed by combining task and environmental information for each drone, which is used to describe the complete state of each drone in the current environment.

[0014] Step S11, define drone state: drone_state = {position, velocity, battery_level, task_queue_length, task_queue_head_delay};

[0015] Among them, drone_state indicates the state of the drone; position indicates the position of the drone; velocity indicates the speed of the drone; battery_level indicates the battery level of the drone; task_queue_length indicates the length of the drone task queue; task_queue_head_delay indicates the waiting time of the head task in the task queue;

[0016] Step S12, define the environment state: env_state = {obstacle_positions, weather_conditions, communication_status};

[0017] Among them, env_state represents the state of the environment; obstacle_positions represents the position of obstacles in the environment; weather_conditions represents the current weather conditions; communication_status represents the communication status between drones;

[0018] Step S13, define the task state: task_state = {task_location, task_type, task_priority, task_deadline}; wherein task_location indicates the location of the task; task_type indicates the type of task; task_priority indicates the priority of the task; task_deadline indicates the deadline of the task;

[0019] Step S14: define the environmental state of each drone, including the task and environment state matrix Mtask_and_env and the drone information matrix Mdrone:

[0020] Mtask_and_env=[task_location, task_type, task_priority, task_deadline, obs tacle_positions, weather_conditions, communication_status];

[0021] Mdrone=[position,velocity,battery_level,task_queue_length,task_queue_head_delay].

[0022] Step 2: Collect real drone mission data, count the resources required for the largest mission in the mission file, and initialize the drone swarm environment; extract mission information according to the mission file, build a sample set, and replay the mission in a simulation environment to build a drone swarm simulation environment;

[0023] Step 3: Generate the drone’s actions and their probabilities through the action strategy neural network NNactor;

[0024] Step 4: Use the value evaluation neural network NNcritic to evaluate the value of taking the set action in the current state;

[0025] Step 5: Construct the loss function of the action strategy and value evaluation network for parameter updating.

[0026] Preferably, step 2 is as follows:

[0027] Step S21, collect real drone task files, count the amount of resources required for the largest task in the task files, and initialize the drone swarm environment. The drone swarm environment consists of multiple drones, and the state of each drone is represented by a six-tuple {drone position, drone speed, drone battery power, task queue length, waiting time of the head task in the task queue, drone id};

[0028] Among them, the drone position includes longitude, latitude and altitude; the drone speed includes horizontal speed and vertical speed; the drone battery power indicates the current remaining battery power; the task queue length indicates the number of tasks in the drone task queue; the waiting time of the task queue head task indicates the waiting time of the first task in the task queue; the drone id indicates the unique identifier of the drone;

[0029] Step S22: extract each task information in order according to the task file, perform data cleaning according to the threshold, retain the tasks with resource competition, and build a sample set; randomly select samples from the sample set and put them into the task queue for playback to build a drone swarm simulation environment.

[0030] Preferably, step three is as follows:

[0031] S31, obtaining the cluster state vector S of the current drone swarm simulation environment;

[0032] S = [q, N];

[0033] Among them, q represents the number of tasks in the current task queue, and N represents the number of drones in the current drone swarm;

[0034] S32, how to convert the cluster state vector S to obtain the task and environment state matrix Mtask_and_env and the drone information matrix Mdrone corresponding to each drone;

[0035] S33, generating a drone priority discount factor;

[0036] The hidden state of each drone is used to record the historical task execution information of the drone, and the hidden state and Mtask_and_env of each drone are input into the action strategy neural network NNactor to generate the corresponding drone priority discount factor;

[0037] S34, generating task priorities and actions;

[0038] Generate task priority based on Mtask_and_env, generate action adrone of a single drone and log value log_p(adrone) of probability of selecting current action adrone; how to generate priority, action and probability

[0039] S35, generating joint actions and performing resource allocation;

[0040] Combine the individual actions of all drones to generate a joint action ajoint; adjust resource allocation according to the task priority of each drone to ensure that high-priority tasks can be completed first; apply the joint action ajoint to the current drone swarm environment, update the status of each drone, and execute the corresponding task;

[0041] And follow ajoint to allocate resources how to allocate resources to make the action effective;

[0042] S36. Calculate the reward value R of each drone according to the formula i , the calculation formula is as follows:

[0043] R i =α·ω p,i +β·ω b,i ;

[0044] Among them, α and β are hyperparameters; ω p is the set of task waiting time ratios of all UAVs; ω p,i is the task waiting time ratio of UAV i; ω b is the set of mission completion efficiency ratios of all drones; ω b,i is the mission completion efficiency ratio of UAV i.

[0045] Preferably, step 4 is as follows:

[0046] S41. Calculate the action value of each drone: input the drone information matrix Mdrone into the evaluation neural network NNcritic to calculate the value vdrone of the current action that each drone will obtain; S42. Store the relevant data in MAPPO_buffer. When the current sample completes the scheduling, a trajectory is generated; the relevant data refers to all information related to the action, state change and reward performed by the current drone;

[0047] make

[0048] ti={s1,s2,...,sN,a1,a2,...aN,log_p1,log_p2,...,log_pN,v1,v2,...vN};

[0049] And store it in MAPPO_buffer;

[0050] Among them, sN represents the state vector; aN represents the action vector; log_pN represents the logarithmic probability; vN represents the value vector; N is the number of drones; when the current sample completes the scheduling, a trajectory {t1, t2, ..., tT} is generated.

[0051] Preferably, step five also includes updating network parameters, which is as follows:

[0052] When the number of trajectories in MAPPO_buffer reaches the threshold, the network parameters are updated:

[0053] The fitted action policy function and value evaluation function of NNactor and NNcritic are represented as πθ and Vω respectively, where θ and ω are the parameters of the action policy network NNactor and the value evaluation network NNcritic respectively;

[0054] The advantage function is estimated using the GAE formula, which is a weighted average of multiple value estimates, as shown in the formula:

[0055]

[0056] Among them, A t is the advantage function, defined as the reward at the current moment t minus the baseline value; t represents the current moment; t+1 represents the next moment; l represents the time interval; γ is the discount factor; λ represents the proportion of the value estimate of each step in the future, V(s t ) represents the value evaluation of the state at time t; T represents the trajectory length; reward-to-go calculation is performed, as shown in the formula, which is used to update the parameters of the value evaluation neural network NNcritic:

[0057]

[0058] Among them, G t is the return value; that is, the sum of all rewards from the current moment to the end of the trajectory; T represents the trajectory length; define the loss functions Lπ and LV of the action strategy network and the value evaluation network, as shown in the formula:

[0059]

[0060] Among them, L π (θ) is the loss function of the action strategy network; L V (ω) is the loss function of the value evaluation network; is the expectation operator;

[0061] Through the loss function of the action policy network and the value evaluation network, the parameters of the action policy network and the value evaluation network are updated using gradient back propagation, and the parameters become θ and ω.

[0062] Preferably, step five also includes a strategy similarity check, which is as follows:

[0063] After completing a network parameter update, the strategy similarity is calculated, as shown in the formula, that is, KL divergence to determine whether trajectory generation needs to be re-performed:

[0064]

[0065] Among them, πθ old is the parameter of the old version; πθ(a|s) represents the probability of selecting action a according to the strategy π determined by the parameter θ when the state s is given;

[0066] When the KL divergence is less than the preset threshold, the training is performed by introducing the scaling of the probability, that is, the loss function of the action strategy neural network becomes as shown in the formula:

[0067]

[0068] Among them, L π (θ) is the loss function; D is the sample distribution in the experience replay buffer; ∈ is a decimal constant, which is a very small positive number used to prevent division by zero errors;

[0069] S53, repeat steps 3 to 5 until a predetermined epoch value is reached.

[0070] The second object of the present invention adopts the following technical solution:

[0071] A distributed dynamic collaboration system for drone swarms is used to implement a distributed dynamic collaboration method for drone swarms. The system includes a model building module, an environment initialization module, an action strategy generation module, a value evaluation module, and a network parameter update module.

[0072] Model building module: used to define the drone state, environment state, and mission state; generate a comprehensive state matrix for each drone;

[0073] Environment initialization module: used to collect and count mission data, initialize the drone swarm environment, build mission sample sets and simulate the environment;

[0074] Action strategy generation module: used to obtain the cluster state vector, convert it to obtain the drone and task environment state matrix, generate the drone priority discount factor, generate task priority and action, merge actions to generate joint actions, and calculate the drone reward value;

[0075] Value evaluation module: used to calculate the value of drone actions, store data and generate trajectories;

[0076] Network parameter update module: used to update action strategies and value evaluation network parameters, estimate the advantage function using the GAE formula, calculate reward-to-go, define and calculate the loss function, perform strategy similarity checks, and adjust training strategies.

[0077] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0078] 1. By defining the drone state, environment state and task state, the model of the present invention can comprehensively consider various factors when the drone performs the task, so as to formulate a more reasonable coordination strategy; the drone's actions and their probability generation enable the drone to make optimal or near-optimal decisions based on the current state and environment, thereby improving the overall operation efficiency.

[0079] 2. By evaluating the value of taking set actions in the current state, the present invention enables the drone to select the action that is most conducive to task execution, thereby shortening the task completion time; task playback and simulation in a simulation environment help to discover potential problems and bottlenecks in advance and optimize task allocation and scheduling.

[0080] 3. The present invention can handle uncertainty and dynamically changing environments through the Markov game model, so that the drone swarm can respond flexibly when faced with emergencies or environmental changes; by constructing the loss function of the action strategy and value evaluation network and updating the parameters, the model can continuously learn and optimize, thereby improving the ability to predict and respond to future situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0082] Figure 1 A flow chart of a distributed dynamic collaboration method for a drone swarm according to the present invention is shown;

[0083] Figure 2 A module diagram of a distributed dynamic collaborative system of a drone swarm according to the present invention is shown;

[0084] Figure 3 A flow chart of generating drone actions and their probabilities according to the present invention is shown. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] In addition, the described features, structures or characteristics may be combined in one or more example embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the example embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, steps, etc. may be adopted. In other cases, well-known structures, methods, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0087] Embodiment 1:

[0088] See also Figure 1 As shown, a distributed dynamic collaboration method for unmanned aerial vehicles in this embodiment has the following process:

[0089] Step 1: Establish a Markov game model for distributed dynamic coordination of drone swarms, including defining the drone state, environment state, task state, and the environment state of each drone.

[0090] Drone status: including but not limited to the drone's location, speed, battery level, task queue length, and waiting time for the task at the head of the task queue.

[0091] Environmental status: including but not limited to the location of obstacles in the environment, weather conditions, and communication status between drones.

[0092] Task status: including but not limited to the task's location, type, priority, and deadline.

[0093] Environmental status of each UAV: ​​Combining mission and environmental information, a comprehensive state matrix of each UAV is formed to describe the complete state of each UAV in the current environment.

[0094] The specific steps are as follows:

[0095] Step S11, define drone state: drone_state = {position, velocity, battery_level, task_queue_length, task_queue_head_delay};

[0096] Among them, drone_state indicates the state of the drone; position indicates the position of the drone; velocity indicates the speed of the drone; battery_level indicates the battery power of the drone; task_queue_length indicates the length of the drone task queue; task_queue_head_delay indicates the waiting time of the head task in the task queue.

[0097] Step S12, define the environment state: env_state = {obstacle_positions, weather_conditions, communication_status};

[0098] Among them, env_state represents the state of the environment; obstacle_positions represents the position of obstacles in the environment; weather_conditions represents the current weather conditions; communication_status represents the communication status between drones;

[0099] Step S13, define the task state: task_state = {task_location, task_type, task_priority, task_deadline}; wherein task_location indicates the location of the task; task_type indicates the type of task; task_priority indicates the priority of the task; task_deadline indicates the deadline of the task;

[0100] Step S14: define the environmental state of each drone, including the task and environment state matrix Mtask_and_env and the drone information matrix Mdrone:

[0101] Mtask_and_env=[task_location, task_type, task_priority, task_deadline, obs tacle_positions, weather_conditions, communication_status];

[0102] Mdrone=[position,velocity,battery_level,task_queue_length,task_queue_he ad_delay].

[0103] Step 2: Collect real drone mission data, count the resources required for the largest task in the mission file, and initialize the drone swarm environment; according to the mission file, extract mission information, build a sample set, and replay the mission in a simulation environment to build a drone swarm simulation environment.

[0104] The specific steps are as follows:

[0105] Step S21, collecting real drone task files, counting the amount of resources required for the largest task in the task file, and initializing the drone swarm environment, wherein the drone swarm environment is composed of multiple drones, and the state of each drone is represented by a six-tuple {drone position, drone speed, drone battery power, task queue length, waiting time of the head task in the task queue, drone id}; wherein the drone position includes longitude, latitude and altitude; the drone speed includes horizontal speed and vertical speed; the drone battery power represents the current remaining battery power; the task queue length represents the number of tasks in the drone task queue; the waiting time of the head task in the task queue represents the waiting time of the first task in the task queue; the drone id represents the unique identifier of the drone;

[0106] Step S22: extract each task information in order according to the task file, perform data cleaning according to the threshold, retain the tasks with resource competition, and build a sample set; randomly select samples from the sample set and put them into the task queue for playback to build a drone swarm simulation environment.

[0107] Step 3: Generate the drone’s actions and their probabilities through the action strategy neural network NNactor.

[0108] See also Figure 3 As shown, the specific steps are as follows:

[0109] S31. Obtain the cluster state vector S of the current drone swarm simulation environment.

[0110] S = [q, N];

[0111] Among them, q represents the number of tasks in the current task queue, and N represents the number of drones in the current drone swarm.

[0112] S32. By converting the cluster state vector S, the task and environment state matrix Mtask_and_env and the drone information matrix Mdrone corresponding to each drone are obtained.

[0113] S33. Generate a drone priority discount factor.

[0114] The hidden state of each UAV is used to record the historical task execution information of the UAV. The hidden state of each UAV and Mtask_and_env are input into the action strategy neural network NNactor to generate the corresponding UAV priority discount factor.

[0115] S34. Generate task priorities and actions.

[0116] Generate task priority according to Mtask_and_env, generate the action adrone of a single drone and the log value log_p(adrone) of the probability of selecting the current action adrone.

[0117] S35, generating joint actions and performing resource allocation;

[0118] The individual actions of all drones are combined to generate a joint action ajoint; according to the task priority of each drone, resource allocation is adjusted to ensure that high-priority tasks can be completed first; the joint action ajoint is applied to the current drone swarm environment, the status of each drone is updated, and the corresponding tasks are executed.

[0119] S36. Calculate the reward value R of each drone according to the formula i , the calculation formula is as follows:

[0120] R i =α·ω p,i +β·ω b,i ;

[0121] Among them, α and β are hyperparameters; ω p is the set of task waiting time ratios of all UAVs; ω p,i is the task waiting time ratio of UAV i; ω b is the set of mission completion efficiency ratios of all drones; ω b,i is the mission completion efficiency ratio of UAV i.

[0122] Step 4: Use the value evaluation neural network NNcritic to evaluate the value of taking the set action in the current state.

[0123] S41. Calculate the action value of each drone: input the drone information matrix Mdrone into the evaluation neural network NNcritic to calculate the value vdrone of the current action that each drone will obtain.

[0124] S42, store the relevant data in MAPPO_buffer, and generate a trajectory when the current sample completes the scheduling. The relevant data refers to all information related to the actions, state changes and rewards performed by the current drone;

[0125] make

[0126] ti={s1,s2,...,sN,a1,a2,...aN,log_p1,log_p2,...,log_pN,v1,v2,...vN};and stored in MAPPO_buffer;where sN represents the state vector; aN represents the action vector; log_pN represents the logarithmic probability; vN represents the value vector; N is the number of drones;

[0127] When the current sample completes scheduling, a trajectory {t1, t2, ..., tT} is generated.

[0128] Step 5: Construct the loss function of the action strategy and value evaluation network for parameter updating.

[0129] S51. Network parameters are updated.

[0130] The details are as follows:

[0131] When the number of trajectories in MAPPO_buffer reaches the threshold, the network parameters are updated:

[0132] The fitted action policy function and value evaluation function of NNactor and NNcritic are represented as πθ and Vω respectively, where θ and ω are the parameters of the action policy network NNactor and the value evaluation network NNcritic respectively;

[0133] The advantage function is estimated using the GAE formula, which is a weighted average of multiple value estimates, as shown in the formula:

[0134]

[0135] Among them, A t is the advantage function, defined as the reward at the current moment t minus the baseline value; t represents the current moment; t+1 represents the next moment; l represents the time interval; γ is the discount factor; λ represents the proportion of the value estimate of each step in the future, V(s t ) represents the value evaluation of the state at time t; T represents the trajectory length; r is the reward value;

[0136] The reward-to-go calculation is performed as shown in the formula for updating the parameters of the value evaluation neural network NNcritic:

[0137]

[0138] Among them, G t is the return value; that is, the sum of all rewards from the current moment to the end of the trajectory; T represents the length of the trajectory.

[0139] Define the loss functions Lπ and LV of the action strategy network and the value evaluation network, as shown in the formula:

[0140]

[0141] Among them, L π (θ) is the loss function of the action strategy network; L V (ω) is the loss function of the value evaluation network; is the expectation operator.

[0142] Through the loss functions of the action strategy network and the value evaluation network, the parameters of the action strategy network and the value evaluation network are updated by using gradient back propagation, and the parameters become θ and ω;

[0143] S52: Strategy similarity check.

[0144] After completing a network parameter update, the strategy similarity is calculated, as shown in the formula, that is, KL divergence to determine whether trajectory generation needs to be re-performed:

[0145]

[0146] Among them, πθ old is the parameter of the old version; πθ(a|s) represents the probability of selecting action a according to the strategy π determined by the parameter θ when the state s is given.

[0147] When the KL divergence is less than the preset threshold, the training is performed by introducing the scaling of the probability, that is, the loss function of the action strategy neural network becomes as shown in the formula

[0148] Among them, L π (θ) is the loss function; D is the sample distribution in the experience replay buffer; ∈ is a decimal constant, which is a very small positive number used to prevent division by zero errors.

[0149] S53, repeating steps 3 to 5 until a predetermined Epoch value is reached. An Epoch is a complete training cycle, indicating that the model has seen all training samples once.

[0150] The beneficial effects of this embodiment are as follows: the distributed dynamic coordination method of drones in this embodiment realizes efficient coordination of drone groups in complex environments through the Markov game model and neural network. This method improves task execution efficiency, optimizes resource allocation, and ensures the stability and adaptability of drone groups, and has significant technical progress and application value.

[0151] Embodiment 2:

[0152] See also Figure 2 As shown, a distributed dynamic collaborative system of a drone swarm of this embodiment includes a model building module, an environment initialization module, an action strategy generation module, a value evaluation module and a network parameter updating module.

[0153] Model building module: used to define the drone state, environment state, and mission state; generate a comprehensive state matrix for each drone.

[0154] Environment initialization module: used to collect and count mission data, initialize the drone swarm environment, build mission sample sets and simulate the environment.

[0155] Action strategy generation module: used to obtain the cluster state vector, convert it to obtain the drone and task environment state matrix, generate the drone priority discount factor, generate task priority and action, merge actions to generate joint actions, and calculate the drone reward value.

[0156] Value evaluation module: used to calculate the value of drone actions, store data and generate trajectories.

[0157] Network parameter update module: used to update action strategies and value evaluation network parameters, estimate the advantage function using the GAE formula, calculate reward-to-go, define and calculate the loss function, perform strategy similarity checks, and adjust training strategies.

[0158] The beneficial effects of this embodiment include: achieving efficient collaborative operation of drone swarms, significantly improving task execution efficiency and resource utilization through precise modeling, environment initialization, intelligent action strategy generation, action value evaluation and dynamic update of network parameters, and enhancing the adaptability and robustness of the system.

[0159] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0160] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to only specific implementation methods. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A distributed dynamic collaboration method for drone swarms, characterized in that: The method flow is as follows: Step 1: Establish a Markov game model for distributed dynamic coordination of drone swarms, including defining drone status, environment status, task status, and the environment status of each drone; Step 2: Collect real drone mission data, count the resources required for the largest mission in the mission file, and initialize the drone swarm environment; extract mission information according to the mission file, build a sample set, and replay the mission in a simulation environment to build a drone swarm simulation environment; Step 3: Generate the drone’s actions and their probabilities through the action strategy neural network NNactor; Step 4: Use the value evaluation neural network NNcritic to evaluate the value of taking the set action in the current state; Step 5: Construct the loss function of the action strategy and value evaluation network for parameter updating.

2. A distributed dynamic collaboration method for drone swarms according to claim 1, characterized in that: The drone status includes the drone's position, speed, battery power, task queue length and waiting time of the task at the head of the task queue; the environmental status includes the location of obstacles in the environment, weather conditions and communication status between drones; the task status includes the location, type, priority and deadline of the task; the environmental status of each drone is a comprehensive state matrix of each drone formed by combining task and environmental information, which is used to describe the complete state of each drone in the current environment.

3. The distributed dynamic collaboration method of a drone swarm according to claim 1, characterized in that: The step 2 is specifically as follows: Step S21, collect real drone task files, count the amount of resources required for the largest task in the task files, and initialize the drone swarm environment, where the drone swarm environment is composed of multiple drones, and the state of each drone is represented by a six-tuple {drone position, drone speed, drone battery power, task queue length, waiting time of the head task in the task queue, drone id}; Among them, the drone position includes longitude, latitude and altitude; the drone speed includes horizontal speed and vertical speed; the drone battery power indicates the current remaining battery power; the task queue length indicates the number of tasks in the drone task queue; the waiting time of the task queue head task indicates the waiting time of the first task in the task queue; the drone id indicates the unique identifier of the drone; Step S22: extract each task information in order according to the task file, perform data cleaning according to the threshold, retain the tasks with resource competition, and build a sample set; randomly select samples from the sample set and put them into the task queue for playback to build a drone swarm simulation environment.

4. The distributed dynamic collaboration method of a drone swarm according to claim 1, characterized in that: The step three is as follows: S31, obtaining the cluster state vector S of the current drone swarm simulation environment; S = [q, N]; wherein q represents the number of tasks in the current task queue, and N represents the number of drones in the current drone swarm; S32, by converting the cluster state vector S, obtaining the task and environment state matrix Mtask_and_env and the drone information matrix Mdrone corresponding to each drone; S33, generating a drone priority discount factor; The hidden state of each drone is used to record the historical task execution information of the drone, and the hidden state and Mtask_and_env of each drone are input into the action strategy neural network NNactor to generate the corresponding drone priority discount factor; S34, generating task priorities and actions; Generate task priority according to Mtask_and_env, generate action adrone of a single drone and log value log_p(adrone) of the probability of selecting the current action adrone; S35, generating joint actions and performing resource allocation; Combine the individual actions of all drones to generate a joint action ajoint; adjust resource allocation according to the task priority of each drone to ensure that high-priority tasks can be completed first; apply the joint action ajoint to the current drone swarm environment, update the status of each drone, and execute the corresponding task; S36. Calculate the reward value R of each drone according to the formula i , the calculation formula is as follows: R i =a·oh p,i +b·h b,i ; Among them, α and β are hyperparameters; ω p is the set of task waiting time ratios of all UAVs; ω p,i is the task waiting time ratio of UAV i; ω b is the set of mission completion efficiency ratios of all drones; ω b,i is the mission completion efficiency ratio of UAV i.

5. The method for distributed dynamic coordination of drone swarms according to claim 1, characterized in that: The step 4 is specifically as follows: S41, calculating the action value of each drone: inputting the drone information matrix Mdrone into the evaluation neural network NNcritic, and calculating the value vdrone of the current action that each drone will obtain; S42, storing the relevant data in MAPPO_buffer, and generating a trajectory when the current sample completes the scheduling; wherein the relevant data refers to all information related to the action, state change and reward performed by the current drone; make ti={s1,s2,...,sN,a1,a2,...aN,log_p1,log_p2,...,log_pN,v1,v2,...vN};and stored in MAPPO_buffer;where sN represents the state vector; aN represents the action vector; log_pN represents the logarithmic probability; vN represents the value vector; N is the number of drones; When the current sample completes scheduling, a trajectory {t1, t2, ..., tT} is generated.

6. The method for distributed dynamic coordination of drone swarms according to claim 1, characterized in that: The step 5 also includes updating network parameters, which are as follows: When the number of trajectories in MAPPO_buffer reaches the threshold, the network parameters are updated: The fitted action policy function and value evaluation function of NNactor and NNcritic are represented as πθ and Vω respectively, where θ and ω are the parameters of the action policy network NNactor and the value evaluation network NNcritic respectively; The advantage function is estimated using the GAE formula, which is a weighted average of multiple value estimates, as shown in the formula: Among them, A t is the advantage function, defined as the reward at the current moment t minus the baseline value; t represents the current moment; t+1 represents the next moment; l represents the time interval; γ is the discount factor; λ represents the proportion of the value estimate of each step in the future, V(s t ) represents the value evaluation of the state at time t; T represents the trajectory length; r is the reward value; The reward-to-go calculation is performed as shown in the formula, which is used to update the parameters of the value evaluation neural network NNcritic: Among them, G t is the return value; that is, the sum of all rewards from the current moment to the end of the trajectory; T represents the length of the trajectory; Define the loss functions Lπ and LV of the action strategy network and the value evaluation network, as shown in the formula: Among them, L π (θ) is the loss function of the action strategy network; L V (ω) is the loss function of the value evaluation network; is the expectation operator; Through the loss functions of the action strategy network and the value evaluation network, the parameters of the action strategy network and the value evaluation network are updated using gradient back propagation, and the parameters become θ and ω.

7. The method for distributed dynamic coordination of drone swarms according to claim 1, characterized in that: The step five also includes a strategy similarity check, which is as follows: After completing a network parameter update, the strategy similarity is calculated, as shown in the formula, that is, KL divergence to determine whether trajectory generation needs to be re-performed: Among them, πθ old is the parameter of the old version; πθ(a|s) represents the probability of selecting action a according to the strategy π determined by the parameter θ when the state s is given; When the KL divergence is less than the preset threshold, the training is performed by introducing the scaling of the probability, that is, the loss function of the action strategy neural network becomes as shown in the formula: Among them, L π (θ) is the loss function; D is the sample distribution in the experience replay buffer; ∈ is a decimal constant, which is a very small positive number used to prevent division by zero errors; c is a constant coefficient; is the expected value of the samples in the experience playback buffer D; clip is the clipping function; S53, repeat steps 3 to 5 until a predetermined epoch value is reached.

8. A distributed dynamic collaboration system for drone swarms, used to implement a distributed dynamic collaboration method for drone swarms as claimed in claim 1, characterized in that: The system includes a model building module, an environment initialization module, an action strategy generation module, a value evaluation module and a network parameter updating module; Model building module: used to define the drone state, environment state, and mission state; generate a comprehensive state matrix for each drone; Environment initialization module: used to collect and count mission data, initialize the drone swarm environment, build mission sample sets and simulate the environment; Action strategy generation module: used to obtain the cluster state vector, convert it to obtain the drone and task environment state matrix, generate the drone priority discount factor, generate task priority and action, merge actions to generate joint actions, and calculate the drone reward value; Value evaluation module: used to calculate the value of drone actions, store data and generate trajectories; Network parameter update module: used to update action strategies and value evaluation network parameters, estimate the advantage function using the GAE formula, calculate reward-to-go, define and calculate the loss function, perform strategy similarity checks, and adjust training strategies.

Images