Real-time decision-oriented dynamic task re-planning framework for uav swarm

Abstract

The application discloses a kind of unmanned aerial vehicle cluster dynamic task re-planning framework for real-time decision, can establish unmanned aerial vehicle cluster task execution scene and hierarchical optimization problem model, based on simulated annealing enhanced K-medoids clustering algorithm calculation optimal task grouping and unmanned aerial vehicle formation deployment, enhance the ability to deal with changes, solid deployment effect;Based on target-oriented belief space multi-agent reinforcement learning algorithm real-time calculation task reassignment result, with belief-oriented experience replay, multi-agent deep Q learning mechanism, improve training efficiency, accelerate strategy convergence.It first establishes task scene, hierarchical optimization model;Then with SADCK-Medoids algorithm to obtain optimal task grouping and formation deployment;After using GOBS-MARL algorithm to achieve sub-group real-time task reassignment;The framework accurately solves unmanned aerial vehicle cluster task planning and re-planning, effectively improves task execution efficiency and system stability, ensures efficient operation under complex dynamic environment.

Description

(I) Technical Field

[0001] This invention belongs to the field of reliability engineering technology, specifically relating to a dynamic task replanning framework for UAV swarms that is oriented towards real-time decision-making. (II) Background Technology

[0002] The application of drone swarms in long-term urban services is showing increasing momentum, encompassing areas such as geographic hazard monitoring, data collection, and medical supply delivery. Ensuring their continuous operation in geographically distributed environments requires prioritizing drone deployment and task allocation in mission planning to achieve efficient inter-system collaboration and mission continuity. While integrated solutions for multi-drone deployment and task allocation can effectively manage ground users in different spatial locations, drones inevitably encounter unforeseen circumstances during operation, causing dynamic demands to exceed pre-defined planning scope. This necessitates not only redesigning on-demand scheduling strategies but also enabling drones to make real-time decisions, thereby ensuring effective responses to emergencies arising from changing mission requests. These challenges highlight the necessity of building customized frameworks, supplemented by advanced methods to coordinate and resolve interdependent sub-problems.

[0003] However, current UAV swarm task replanning decision-making methods have several shortcomings. Specifically, they lack a method for task reassignment in distributed environments to handle long-term task execution in UAV swarms; they cannot achieve robustness, adaptability, and convergence to the global optimum for potentially changing long-term tasks; and they lack effective strategies to address sparse rewards and increased task complexity in extended state spaces, thus assisting decision-makers in effectively exploring and improving learning efficiency. Therefore, this invention proposes a replanning framework to address these shortcomings, enabling efficient real-time decision-making for UAV swarms in dynamic environments. (III) Summary of the Invention

[0004] The purpose of this invention is to provide a two-layer framework for UAV swarm task replanning oriented towards real-time decision-making. This framework comprehensively considers factors such as predefined task locations and dynamic requirements, and achieves effective processing of these key elements and goal attainment by developing improved algorithms for sub-problems at each layer. To achieve the above objective, this invention provides the following framework:

[0005] S100: Establishing problem scenarios and hierarchical optimization problem models for drone swarm task execution;

[0006] S101: Abstract cluster distributed task execution scenario;

[0007] S102: Establish an optimization model for the UAV swarm deployment problem based on dynamic subgroup partitioning;

[0008] S103: Establish an optimization model for the UAV task reassignment problem, taking subgroups as units;

[0009] S200: Calculates optimal task grouping and UAV formation deployment based on simulated annealing-enhanced K-medoids clustering algorithm;

[0010] S201: Initialize all task groups and the formation deployment of all drones;

[0011] S202: Use the K-medoids algorithm to group the tasks, resulting in several task subgroups;

[0012] S203: Assign and deploy drones to specific task subgroups and form formations;

[0013] S204: If the annealing conditions or the maximum number of annealing cycles are not met, return to step S201;

[0014] S300: Real-time calculation of task redistribution results based on a goal-oriented belief space multi-agent reinforcement learning algorithm;

[0015] S301: For each task subgroup and the corresponding UAV formation, initialize the agent environment, action-value function, and objective function;

[0016] S302: Using a belief-oriented post-experience replay mechanism, multi-agent training is conducted through reinforcement learning.

[0017] S303: Real-time task redistribution within UAV formations based on trained multi-agent systems.

[0018] In step S100, a problem scenario and hierarchical optimization problem model for the execution of drone cluster tasks are established, and the specific process is as follows.

[0019] A mathematical model of the task execution scenario is established, and the task execution problem of UAV swarm is established as a two-layer optimization problem model. The upper layer establishes an optimization problem model for global optimal task grouping and UAV deployment, and the lower layer establishes a real-time decision problem model for task redistribution for each group of tasks and the corresponding UAV formation.

[0020] In step S101, the cluster distributed task execution scenario is abstracted, considering that the tasks include accessing a finite set of M static tasks, denoted as Γ={τ1,ω2,…,τ M The drone swarm performing the mission consists of a total of N drones, and the drone swarm set is denoted as V = {v1, v2, ..., v...}. N For each drone v i Let i = 1, 2, ..., N, and consider modeling it as a quintuple. <li ,s i ,θ i ,c i ,f i >, where l i =(x i ,y i ,z i ), s i >0, θ i ∈(0,π), f i >0 represents the UAV's position coordinates, constant flight speed, observation angle, capability level, and maximum flight distance, respectively.

[0021] In step S102, an optimization model for the UAV swarm deployment problem based on dynamic subgroup partitioning is established. The task set is divided into K groups, denoted as {A1, A2, ..., A...}. K With maximizing the residual capacity equilibrium (BRC) as the optimization objective, the drone swarm is also divided into K groups, denoted as V = {V0, V1, V2, ..., V...}. K} represents the drone formations deployed to each task dynamic subgroup, and the drone swarm deployment scheme is used as the decision variable for this optimization problem.

[0022] In step S103, an optimization model for the UAV task reassignment problem is established, taking subgroups as units. The task reassignment problem within each UAV formation can be structured as a decentralized, partially observable Markov decision process, denoted as... The drones within each drone swarm are abstracted as intelligent agents, denoted as... Where N k This represents the number of drones in the k-th formation. γ is the reward discount factor, and the other elements are defined as follows:

[0023] State space: The environment state at time t is defined as s. t ∈S, which contains the state of each drone. <l i ,s i ,θ i ,c i ,f i > and the status of the task.

[0024] Action space: each drone v i The action at time t is defined as in The first four elements represent the drone's choice to fly east, west, south, and north, respectively, while the last element represents staying stationary.

[0025] Observation space: Due to the observation range and communication limitations of each UAV, the observation information that each agent relies on when making a decision at time t. This only includes the task location and task requirements.

[0026] State transition probability: The transition probability P satisfies P(s) t+1 |s t ,a t ):s t ×a t ×s t+1 →[0,1] indicates that in s t State selection a t Transition to state s t+1 The probability of.

[0027] Reward function: For In s t Select in status Actions will be rewarded.

[0028] Belief Space: b t ∈b represents the probability of a new task request occurring at time t, and it is calculated as follows:

[0029]

[0030] The calculation method for p is as follows:

[0031]

[0032] Target space: The space containing the final goal and intermediate goals, denoted as G, represents the increase in belief level. g is a mapping function satisfying g:(b t ,s t → G.

[0033] In step S200, the optimal task grouping and UAV formation deployment are calculated based on the simulated annealing-enhanced K-medoids clustering algorithm. The specific process is as follows.

[0034] The outer loop uses the simulated annealing algorithm. Inside each simulated annealing, the following steps are performed: after initializing all task groups and all UAV formation deployments, the K-medoids algorithm is used to group the tasks to obtain several task subgroups. Then, the UAVs are assigned to specific task subgroups and formed into formations. Once the conditions for simulated annealing are met, the deployment of UAVs in the upper layer of the framework is completed.

[0035] In step S201, the first Euclidean centroid μ1 of the initialization cluster is randomly sampled from the task set. Then, based on the distance to the known centroid, the other k-1 centroids are iteratively selected. Points closer to all targets and the known centroids are more likely to be selected as centroids. At the same time, all UAV formations and task subgroups are initialized to an empty set.

[0036] In step S202, the K-medoids algorithm is used to group the tasks, resulting in several task subgroups. The specific process is as follows.

[0037] Step 1: Calculate τ for each task j Euclidean distance d to all centers of mass jk =||τ j -μ k || 2 ;

[0038] Step 2: Find the minimum distance r = argmin among all task subgroups. k d jk , will task τ j Join the task subgroup with the smallest distance;

[0039] Step 3: Calculate the density d of task subgroup k. k If the value is greater than the threshold, the clustering decision is rejected, the second nearest task subgroup is selected, and the threshold is retested; otherwise, the Euclidean centroid μ of task subgroup k is updated. k .

[0040] In step S203, the drones are assigned and deployed to specific task subgroups and formed into formations. The specific process is as follows.

[0041] Step 1: Randomly deploy drones from the drone swarm to K task subgroups;

[0042] Step 2: For each drone v t Calculate the global residual capacity balance (BRC) once. new ;

[0043] Step 3: For the drones selected above... t Select all subgroups C in sequence k , will v t Transfer to task subgroup C in sequence k ;

[0044] Step 4: Calculate the residual capacity equilibrium BRC at this point. now And the change ΔBRC=BRC now -BRC new ;

[0045] Step 5: If ΔBRC > 0, or If the step is successful, accept the transfer from Step 3; otherwise, reject the transfer from Step 3.

[0046] In step S204, if the number of simulated annealing rounds is greater than the threshold N... I If the loop ends, the final task grouping and drone cluster deployment are obtained; otherwise, return to step S201 for a new round of iterative optimization.

[0047] In step S300, the task redistribution result is calculated in real time based on the goal-oriented belief space multi-agent reinforcement learning algorithm. The specific algorithm process is as follows.

[0048] In step S301, for each task subgroup and the corresponding UAV formation, the agent environment, action-value function and objective function are initialized. The specific steps are as follows.

[0049] Step 1: Define the task environment in which the multi-agent operates, including information such as the task area, distribution of user needs, and UAV performance parameters, to provide the basic environment for the agents to perceive and make decisions;

[0050] Step 2: Initialize the action-value function Q for each agent. π (b t ,s t ,a t )=E[U t |b t ,s t ,a t ],in This function is used to evaluate the agent in different belief states. t Environmental conditions t Take action a t The expected cumulative discount reward is assigned a reasonable initial value range during initialization so that it can be gradually optimized in subsequent learning;

[0051] Step 3: Determine the objective function for the multi-agent cooperative task;

[0052] In step S302, a multi-agent system is trained through reinforcement learning by utilizing a belief-oriented post-experience replay mechanism. The specific steps are as follows.

[0053] Step 1: Store belief-guided experiences. At each time step t, the agent selects action a based on the current strategy and observational information. t Then, the experience Tr t ={b t ,s t ,a t ,r t,b t+1 ,s t+1} Stored in the dataset Tr={Tr1,Tr2,…,Tr M In the context of belief state b, the state of belief is... t It reflects the probability of a dynamic task request existing at time step t. The environmental state includes information such as the location of the ground user and the task requirements. The reward is calculated based on the reward function.

[0054] Step 2: Experience Sampling and Update, randomly sample experience data from the dataset. t And according to the objective function g (from belief state b) t and environmental state s t The mapping function g:(b to the target space G) t ,s t → G generates new experiences in When the drone moves in the correct direction, it is greater than r. t Then, these new experiences are used to update the agent's action-value function and belief state.

[0055] Step 3: Multi-agent deep Q-learning update, based on sampling experience, using a temporal difference training method. The action-value function parameters θ are updated using gradient descent to minimize the difference between the predicted and target values, enabling the agent to learn a better policy. Simultaneously, in multi-agent environments, considering the interaction and cooperation among agents, a value decomposition network is introduced. Assuming complete cooperation among agents, i.e., sharing the same reward function, the value decomposition formula is used... satisfy Decomposing global value into local value alleviates instability in the learning process and ensures that the optimal combination of local values ​​becomes the optimal combination of global values.

[0056] In step S303, real-time task reallocation within the UAV formation is performed based on the trained multi-agent system. After the multi-agent system completes training, in actual operation, each agent reallocates tasks according to the current environmental state s. t and one's own belief state b t Based on the learned strategy π, select action a t This enables real-time task redistribution within drone formations. For example, the agent determines the drone's flight path, task execution order, and resource allocation based on factors such as task priority, drone location, and load conditions to efficiently complete tasks and adapt to dynamic changes in task requirements. When faced with sudden task requests or environmental changes, the agent can quickly adjust its decisions to ensure task continuity and system stability.

[0057] Compared with existing technologies, this invention has the following advantages: It proposes a dynamic task replanning framework for UAV swarms oriented towards real-time decision-making. This framework can establish a model to describe a two-layer task replanning problem. At the upper layer, the SADCK-Medoid algorithm is proposed for task grouping and UAV deployment, effectively addressing potential changes in requirements and environment during long-term applications, improving deployment adaptability and stability, and outperforming existing algorithms that lack dynamic adjustment and redundancy balancing. At the lower layer, the GOBS-MARL algorithm is proposed for task reassignment, significantly improving training efficiency, accelerating convergence to the optimal policy, solving the problem of multi-UAV task allocation, and surpassing existing methods constrained by sparse rewards and complex environments. This framework can effectively guide UAV swarms to efficiently complete hierarchical distributed task replanning. (iv) Description of the attached drawings

[0058] Figure 1 A flowchart of a dynamic task replanning framework for UAV swarms oriented towards real-time decision-making. (V) Detailed Implementation

[0059] Exemplary embodiments of the present invention have been described in detail below with reference to the accompanying drawings. The following description includes specific details to aid understanding, but these details should be shown as exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various instances described herein without departing from the scope and spirit of this disclosure. Furthermore, descriptions of well-known functions and structures have been omitted for clarity and brevity.

[0060] The terms and words used in the following description and claims are not limited to their literal meaning, but are intended only for the clear and consistent understanding of the inventors in carrying out the invention. Therefore, it will be clear to those skilled in the art that the following description of various exemplary embodiments of the invention is provided for illustrative purposes only and is not intended to limit the invention as defined by the appended claims and their equivalents.

[0061] Step S100: Establish a drone swarm task execution scenario and hierarchical optimization problem model, mainly considering typical task scenarios such as drone swarms providing parcel delivery services and emergency medical supplies to communities. The parameters of the task and drones are shown in Table 1.

[0062] Table 1 Parameter Settings

[0063]

[0064] For the upper-level drone deployment optimization problem, minimizing the BRC (Breakpoint Reduction) is defined as the global optimization objective. For the lower-level autonomous redistribution decision problem within the drone formation, average reward is used as the optimization objective.

[0065] Step S200: The optimal task grouping and UAV swarm deployment are calculated using a simulated annealing-enhanced K-medoids clustering algorithm, clustering 152 ground users into five task areas. Based on this, detailed deployment results are obtained. Initially, the number of ground users in each task area was 9, 17, 9, 17, and 12, respectively, resulting in the initial deployment of 4, 6, 3, 7, and 6 UAVs, with 14 UAVs remaining stationed at the base. After the first emergency, the number of ground users in each task area increased except for TR1. The deployment decision remained unchanged. After the second emergency, the increase in the number of UAVs in each task area was significant, with 6, 9, 5, 10, and 7 UAVs redeployed, respectively. Simultaneously, in the first phase using the SADCK-Medoid algorithm, the BRC converged to 19.11. In the second phase, the BRC value increased from 19.11 to 27.27, not exceeding the threshold, therefore redeployment was not performed. In the third stage, as new event sequences emerge, the BRC climbs and exceeds the threshold, but with the help of the SADCK-Medoid algorithm, it drives the BRC to converge to an acceptable level again.

[0066] Step S300: Based on the formation deployment results, the task redistribution results are calculated in real time using a goal-oriented belief space multi-agent reinforcement learning algorithm. The task redistribution results are generated after decisions are made by agents trained using the GOBS-MARL algorithm. In the initial stage, the UAVs locate themselves around the existing targets according to the strategy, increasing the probability of detecting unpredictable events. When ground user needs are updated, the UAVs execute tasks according to the BRC global target and their own capabilities. When new ground users are encountered, the redistribution mechanism is activated with the assistance of additional UAVs to ensure comprehensive coverage and achieve accurate task allocation and dynamic collaboration.

[0067] A core feature of this invention's framework is the system's ability to make deployment decisions offline. This feature allows sufficient computation time for the algorithm to perform in-depth optimization, effectively ensuring optimal long-term performance. Once the deployment process is complete, each UAV immediately enters an independent operation mode, autonomously identifying and selecting tasks based on its built-in intelligent decision-making module. The entire process requires minimal external intervention, efficiently achieving a deep integration of autonomy and flexibility in task allocation.

[0068] Of particular note is the superior performance of this invention framework in handling scriptless tasks, enabling UAVs to respond rapidly and make precise decisions in complex and ever-changing emergency scenarios. Upon the emergence of new task requirements, the system can immediately initiate a redeployment process, efficiently allocating resources to comprehensively cover the new task and ensuring the continuity and integrity of task execution. The collaborative operation between deployment and task allocation is achieved through the BRC threshold mechanism, which flexibly adjusts based on the system's real-time status, significantly improving the overall adaptability and dynamic response flexibility of the framework. This lays the foundation for the stable and efficient operation of multi-UAV systems in complex task environments.

[0069] The examples described above illustrate in detail the implementation of each part of the present invention. The specific implementation of the present invention is not limited thereto. For those skilled in the art, all obvious changes made to the method without departing from the spirit and scope of the claims are within the protection scope of the present invention.

Claims

1. A dynamic task replanning method for UAV swarms oriented towards real-time decision-making, characterized in that, Includes the following steps: S100: Establish a model for drone swarm task execution scenarios and hierarchical optimization problems; S200: Calculates optimal task grouping and UAV formation deployment based on simulated annealing-enhanced K-Medoids clustering algorithm; S300: Real-time calculation of task redistribution results based on a goal-oriented belief space multi-agent reinforcement learning algorithm; S100 includes: S101: Abstract cluster distributed task execution scenario; S102: Establish an optimization model for the UAV swarm deployment problem based on dynamic subgroup partitioning; S103: Establish an optimization model for the UAV task reassignment problem, taking subgroups as units; S200 includes: S201: Initialize all task groups and the formation deployment of all drones; S202: Use the K-Medoids clustering algorithm to group the tasks into several task subgroups; S203: Assign and deploy drones to specific task subgroups and form formations; S204: If the annealing conditions or the maximum number of annealing cycles are not met, return to step S201; The S300 includes: S301: For each task subgroup and the corresponding UAV formation, initialize the agent environment, action-value function, and objective function. The specific steps are as follows: Step 1: Define the task environment in which the multi-agent operates, including the task area, user demand distribution, and UAV performance parameters, to provide the basic environment for the agents to perceive and make decisions; Step 2: Initialize the action-value function for each agent. This function is used to evaluate the agent in different belief states. Environmental conditions Take action below The expected cumulative discount reward is assigned a reasonable initial value range during initialization so that it can be gradually optimized in subsequent learning; Step 3: Determine the objective function for the multi-agent cooperative task; S302 utilizes a belief-oriented post-experience replay mechanism to train multiple agents through reinforcement learning. S303: Real-time task redistribution within UAV formations based on trained multi-agent systems. After training, each agent, during actual operation, redistributes tasks according to the current environmental state. and one's own state of belief Based on the learned strategy Select Action This enables real-time task redistribution within drone formations.

Images