A Reinforcement Learning-Based Heterogeneous UAV Alliance Task Allocation Method
By using the SAC reinforcement learning algorithm and constraint awareness mechanism based on the Transformer architecture, an overlapping alliance forming game algorithm was designed to solve the problem of high task allocation cost in the dynamic logistics scenario of UAV swarms. This algorithm achieves fast and reliable collaborative allocation, improving the task allocation efficiency of UAV swarms and the convergence speed of the model.
Patent Information
- Application Number
- CN202611272766.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing drone swarm task allocation methods struggle to achieve fast and reliable collaborative allocation in dynamic logistics scenarios, resulting in high task allocation costs. Traditional methods also suffer from high computational complexity and communication overhead in highly dynamic scenarios, and are prone to getting trapped in local optima.
We employ the Soft Actor-Critic (SAC) reinforcement learning algorithm based on the Transformer architecture, combined with a multi-head self-attention mechanism and a constraint-aware validity masking mechanism, to design an Overlapping Alliance Formation Game Algorithm (TSAC-OCF). Through local state observation and online collaborative task allocation, we achieve fast and reliable collaborative allocation of UAV swarms.
It effectively reduces the overall delivery cost of drone swarms in dynamic logistics scenarios, improves the decision-making efficiency and model convergence speed of drone swarms, avoids local optima, and achieves efficient task allocation of drone swarms in complex environments.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] This invention belongs to the field of task allocation technology, specifically relating to a heterogeneous UAV alliance task allocation method based on reinforcement learning. Background Technology
[0002] With the rapid development of the low-altitude economy, drone swarms have become a crucial cornerstone in logistics and other fields, significantly improving people's lives. However, the massive scale and high dynamism of logistics tasks pose significant challenges to the autonomous collaboration of drone swarms. Task allocation, as a core element in achieving autonomous collaboration, aims to minimize task costs or maximize task benefits by assigning reasonable execution plans to heterogeneous drones, which is crucial for improving the overall task efficiency of the swarm. Especially in actual logistics and delivery scenarios, orders are not given all at once but emerge continuously and dynamically as the delivery process progresses, causing the number and status of tasks to be allocated to change in real time. This places higher demands on the task allocation methods for drone swarms.
[0003] Currently, existing methods for task allocation in heterogeneous drone swarms mainly include genetic algorithms, contract network algorithms, and alliance-forming game theory algorithms. While genetic algorithms can find near-optimal solutions, their long iteration cycles and high computational complexity make them unsuitable for online real-time allocation in highly dynamic scenarios. Contract network algorithms, constrained by pre-defined market bidding rules, are prone to message congestion when dealing with large-scale swarms, leading to a surge in communication overhead and a decline in system performance.
[0004] Alliance formation game theory, as an important branch of game theory, uses mathematical models to characterize how participants dynamically form alliances to maximize their interests, providing a rigorous theoretical foundation for modeling the collaborative behavior of drone swarms. However, traditional alliance formation game algorithms often rely on random search or heuristic rules to evolve and construct alliance structures. These traditional strategies not only have huge search spaces and slow convergence speeds, but also easily lead to excessively high task allocation costs and low decision-making efficiency for drone swarms, making them difficult to adapt to complex and ever-changing logistics task environments. Summary of the Invention
[0005] In view of this, to address the high delivery costs of drone swarms in dynamic logistics scenarios caused by existing technologies, this invention provides a heterogeneous drone alliance task allocation method based on reinforcement learning. This method improves the Soft Actor-Critic (SAC) reinforcement learning algorithm based on the Transformer architecture, enabling it to effectively handle a variable number of task state inputs and train a forward-looking alliance selection strategy with long-term benefits for individual drones. Furthermore, the matured strategy is embedded into an overlapping alliance task allocation framework, achieving fast and reliable collaborative allocation of drone swarms, thereby significantly reducing the overall delivery cost of swarms in dynamic logistics scenarios.
[0006] To solve the above-mentioned technical problems, the present invention is implemented as follows.
[0007] A task allocation method for heterogeneous drone alliances based on reinforcement learning, adapted to dynamic logistics scenarios; multiple drones form an alliance to collaboratively complete a task, and a single drone can join multiple alliances simultaneously; the method includes: Phase 1: Adoption Reinforcement learning for training alliances using individual drones: Step 11: Obtain local state observations of the logistics scenario at the current moment, including the motion state and available payload state of a single drone, the real-time remaining payload requirements of unfinished tasks globally, the current alliance size, and the global warehouse status; Step 12: Construct a joint feature representation based on local state observations, and input... The encoder outputs global feature embeddings; Step 13: Embed global features into the input SAC network and train the drone to learn strategies to reduce delivery costs through reinforcement learning; Phase Two: Embed the trained policy network into an online overlapping alliance allocation framework for online collaborative task allocation. Step 21: In each iteration, each drone makes a sequential decision on task allocation; when it's a drone's turn... When making decisions, based on the preceding unmanned aerial vehicles (UAVs) The updated alliance structure after action decision-making determines the local state observation of the logistics scenario at the current moment, utilizing pre-trained... The encoder and the trained policy network perform action reasoning to determine the target task; Step 22: Based on the target tasks assigned to each UAV, output the final heterogeneous UAV swarm alliance task allocation scheme.
[0008] Preferably, the local state observation is modeled as a state set. :
[0009] in, express Time Drone The set of local states observed in the current logistics scenario; For drones The drone's status characteristics include its real-time location, available payload status, and timeout risk of the current task sequence. This represents the global unfinished task status characteristic. Represents the global set of unfinished tasks. Tasks in , Representation task Task status characteristics, including task Real-time remaining payload requirements, currently allocated to execute this task The size of the alliance of drones, and the drones Spatial distance ranking; the spatial distance ranking is based on drones and The ranking is obtained by sorting the flight distances of each mission in ascending order. This represents the global repository state characteristics. Represents a collection of warehouses warehouse , For warehouse The warehouse status characteristics.
[0010] Preferably, the method for determining the timeout risk of the currently executed task sequence is as follows: for drones For all tasks in the current task sequence, take the minimum difference between the deadline and the expected arrival time of each task, and normalize it to the interval [0,1]. The smaller the timeout risk value, the higher the timeout risk of the current task sequence.
[0011] Preferably, step 12 specifically includes: Using a multilayer sensor to analyze the state features of the drone Global unfinished task status characteristics and global repository state characteristics Encode the UAV feature vector, task feature vector, and warehouse feature vector; then expand the encoded UAV feature vector. Each feature vector is concatenated with the feature vectors of each task and the warehouse to generate a joint feature representation. ; These are the global unfinished task sets. and warehouse collection The number of elements in; Joint feature representation Input a Transformer encoder and use a multi-head self-attention mechanism to extract contextual global feature embeddings that include the global topology and spatiotemporal dependencies of the current logistics scenario. .
[0012] Preferably, in step 13, when training the drone to learn strategies for reducing delivery costs through reinforcement learning, the objective function of reinforcement learning finds the global cost. Minimum task; global cost Expressed as:
[0013] in, For a global set of tasks; Representative task The load distribution vector, The total number of drones, Indicates drone In the mission The amount of load already allocated; Indicates a collection of drones; Task Comprehensive cost function It consists of three weighted parts:
[0014] In the formula, These are preset weighting coefficients; The cost of insufficient payload is determined by the task. Unmet load requirements are expressed. To ensure timely service costs, responsible for evaluating the performance of drone swarm missions. The efficiency; for collaborative tasks, the timeliness cost of service will continue to increase until the last drone arrives; The operating expenses are calculated by taking into account both the conventional energy consumption costs of the drones during missions and the potential damage costs of the drones, and are the sum of the operating expenses of all individual drones.
[0015] Preferably, the reinforcement learning process satisfies the following physical constraints: Instantaneous load constraints: per drone At any moment during the execution of a mission, it must be ensured that the amount of load it carries at any instant does not exceed its maximum load capacity. Closed-loop route constraint: The mission sequence of the UAV must meet the take-off and landing closed-loop condition that starts from the mission warehouse and ends at any warehouse.
[0016] Preferably, in step 13, the policy network of the SAC network receives global feature embeddings and outputs log-probability in the discrete action space; a constraint-aware binary mask vector is introduced at the output of the policy network. This is used to strictly limit the probability of selecting illegal tasks that violate the maximum range constraint or mission time window constraint of the drone to 0.
[0017] Preferably, in step 13, a variable-length experience playback mechanism is set: When sampling batch data containing Q transition sequences in the experience replay buffer, all sequences are padded to the maximum length within that batch. This aligns the dimensions of the batch data; simultaneously, it constructs a binary mask matrix to shield invalid padding gradients. Elements within the valid sequence length are assigned a value of 1, while elements in the filling portion are assigned a value of 0. During backpropagation, the binary mask matrix is used. Weight the loss function.
[0018] Preferably, in step 21, at the beginning of each iteration, a decision sequence for the UAV is generated using a random permutation function, and each UAV makes a task allocation decision according to the decision sequence.
[0019] Preferably, in step 21, after determining the target task for the UAV through action reasoning in each iteration, the allocation result of the UAV-target task is further confirmed based on a hybrid stabilization mechanism of "conditional addition" and "unconditional removal"; specifically: Conditional addition operation: If the selected target task Not in drones In the current execution task sequence, and add the target task. If joining a task alliance can reduce the cost of a single task, then confirm the joining operation and set the target task. Add to the currently executing task sequence; Unconditional removal operation: If the selected target task Already in drones If the drone is in the current execution task sequence, then the drone will exit the target task. The mission alliance actively releases the target mission. Occupied by the drone's payload, target mission Enter the global unfinished task set and wait for a better task alliance to be reconstructed.
[0020] Preferably, in step 21, after determining the target task for the UAV through action reasoning in each iteration, the allocation result of UAV-target task is further confirmed based on the taboo list that records the existing alliance structure. Within a set finite step size, reverse operations that would fall back to the alliance structure in the taboo list are prohibited.
[0021] Beneficial effects: (1) To address the problem that traditional fixed-length reinforcement learning networks are unsuitable for tasks with random fluctuations in scale under dynamic logistics scenarios, this invention combines the Transformer architecture with the SAC reinforcement learning algorithm, utilizing a multi-head self-attention mechanism to extract the global topology and spatiotemporal dependencies of the logistics network. This architecture breaks the dependence of traditional reinforcement learning on a fixed-dimensional state space, and can adaptively handle variable numbers of task inputs, providing forward-looking strategy support for the collaborative cooperation of heterogeneous UAV swarms in dynamic environments.
[0022] (2) In view of the characteristics of continuous changes in task allocation and quantity in logistics and delivery scenarios, this invention constructs a local state observation of the current logistics scenario based on the real-time number of tasks and the specific allocation of task loads. This local state observation is used to characterize the carrying capacity of UAVs on task loads (a UAV may carry all or part of the loads of multiple tasks) and the allocation of task loads (the same task may be carried by multiple UAVs working together). Through the design of local state observation, the real-time changes of logistics scenarios can be reflected, and it is applicable to scenarios of overlapping alliances (UAVs participate in different task alliances at the same time according to their remaining capacity).
[0023] (3) To address the problem that reinforcement learning under complex physical constraints is prone to outputting invalid actions, leading to execution failures and low learning efficiency, this invention designs a constraint-aware validity masking mechanism at the policy network output. During the action probability distribution calculation phase, this mechanism strictly limits the probability of illegal tasks that violate physical constraints such as maximum range or strict time windows to zero. Through this hard constraint embedding, not only is the task allocation scheme generated by the model absolutely feasible in terms of physical kinematics, but also useless explorations are effectively filtered out by pruning the invalid solution space, thereby significantly improving the convergence speed of the reinforcement learning model and the reliability of online inference.
[0024] (4) To address the problem that traditional heuristic or random alliance formation algorithms are prone to getting trapped in local optima, leading to high task allocation costs for UAV swarms, this invention designs a reinforcement learning-guided Overlapping Alliance Formation Game (TSAC-OCF) algorithm. An asynchronous iterative framework is used to decompose global alliance building decisions, and a pre-trained context-aware strategy replaces short-sighted heuristic rules for action reasoning. Simultaneously, a hybrid stabilization mechanism including "unconditional removal" and "conditional joining" operations is introduced, and a tabu search list is used to store searched alliance structures to avoid repeated reverse operations. This not only gives UAV swarms the flexibility to escape local optima but also effectively prevents action loop deadlock, ensuring that the alliance structure continuously evolves towards a lower-cost allocation scheme until convergence. This achieves truly efficient and stable online alliance task allocation for UAV swarms, significantly reducing the cost of UAV swarm task allocation in dynamic logistics scenarios. Attached Figure Description
[0025] Figure 1 This is a training architecture for reinforcement learning networks based on transformers; Figure 2 To reinforce learning training curves; Figure 3 A schematic diagram for a small-scale scenario; Figure 4 The image shows a comparison of Monte Carlo simulations. Detailed Implementation
[0026] This invention provides a heterogeneous UAV alliance task allocation method based on reinforcement learning, and its core technological innovations include: (1) Variable task sequence processing capability for dynamic logistics scenarios. In actual logistics and delivery scenarios, orders continuously emerge over time, causing the number of tasks to be assigned to change in real time. Traditional reinforcement learning relies on fixed-dimensional network inputs and cannot adapt to scenarios with dynamically fluctuating task numbers. This invention introduces the Transformer architecture into the SAC reinforcement learning framework and uses a self-attention mechanism to globally model the variable number of task features, enabling the trained policy to adaptively process any number of task inputs, thus solving the applicability bottleneck of traditional methods in dynamic logistics scenarios.
[0027] To address the continuously changing nature of task allocation and quantity in logistics and delivery scenarios, this invention constructs a local state observation of the current logistics scenario based on the number of tasks and the specific allocation of task payloads (drone load status and unallocated payload status). This local state observation characterizes the drone's load-bearing capacity (a drone may carry all or part of the payload for multiple tasks) and the allocation of task payloads (the same task may be carried by multiple drones collaborating). This local state observation design can reflect real-time changes in the logistics scenario and is applicable to scenarios with overlapping alliances (drones participating in different task alliances simultaneously based on their remaining capacity).
[0028] (2) Constraint-aware validity mask mechanism. When performing logistics tasks, UAVs must strictly adhere to physical constraints such as maximum range and delivery time window. Traditional reinforcement learning is prone to generating invalid actions that violate these constraints during training, leading to execution failures and reduced training efficiency. This invention introduces a validity mask at the output of the policy network, forcibly setting the probability of all actions that do not meet the physical constraints to zero, ensuring that the output task allocation scheme is always executable, while reducing invalid exploration and improving training convergence speed.
[0029] (3) Reinforcement learning-guided overlapping alliance formation game framework. Traditional alliance formation methods rely on random search or fixed rules, which are prone to getting stuck in local optima and have high allocation costs when the task scale increases. This invention embeds a trained policy network into an overlapping alliance allocation framework to replace fixed rules driving the alliance decision of UAVs; and introduces a hybrid operation mechanism of "conditional joining and unconditional leaving" and a taboo list to avoid repeated search and circular deadlock, and realize low-cost collaborative allocation of cluster tasks.
[0030] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0031] Step 1: Establish a heterogeneous UAV cluster task allocation model as shown in equations (1)-(7).
[0032] Consider a dynamic urban logistics and delivery scenario, where Heterogeneous UAVs Collaborative Execution One task. Definition For drones, For a set of tasks, For warehouse collection, This refers to the number of warehouses. The above tasks are divided into two types: pickup tasks (…). ) and delivery tasks ( ),satisfy . Each task in Can be determined by load requirements Delivery time window Characterization. It should be noted that... Orders are not all given at the initial moment, but rather emerge dynamically as the delivery process progresses. Definition for For tasks that have already occurred but not yet been completed, the drone can only respond at any given time. The number of tasks that drones need to handle varies over time.
[0033] To characterize the heterogeneous nature of drone swarms, each drone... It has the following differentiating attributes: flight speed Maximum load capacity With maximum flight range The warehouse serves as both the initial deployment location and a dynamic resupply point for the drones in the scenario. During mission execution, the drones can change their flight paths to reach the assembly point. Any warehouse in the system can reset its load status, including reloading goods for delivery tasks or unloading goods from pickup tasks.
[0034] In logistics scenarios, the goal of task allocation is to reduce overall costs. The optimization problem is shown in equation (1).
[0035] (1) in, Representative task Load / cargo allocation vector, Indicates drone In the mission The specific load allocated is specified. Additionally, the collaborative execution tasks are defined. The drone alliance is .
[0036] Task Comprehensive cost function It consists of three weighted parts: (2) In the formula, These are the preset weighting coefficients. The specific definitions of each cost item are as follows: (1) Insufficient load penalty cost Used for quantifying tasks The unmet payload requirements are shown in Equation (3).
[0037] (3) In the formula, As an indicator function, when the drone Arrive at the mission time Less than or equal to task Deadline When the time condition is met, the value is 1; otherwise, it is 0. For the task The load requirements. Note that when the mission's load requirements are fully met, Set the lower bound to 0.
[0038] (2) Service timeliness cost : Responsible for evaluating the missions performed by the drone swarm The efficiency. For collaborative tasks, the service timeliness cost will continue to increase until the last drone arrives, as shown in equation (4).
[0039] (4) In the formula, The overall task timeframe is planned (for normalization). For the task The start time, The last mission to arrive within the alliance The time required for the drone.
[0040] (3) Operating expenses The study comprehensively considered both the conventional energy consumption costs of the drone during mission execution and the potential costs of drone damage. The value of is equal to the sum of the individual costs of all drones, as shown in equation (5).
[0041] (5) In the formula, Indicates drone Flight distance between consecutive missions; For drones Complete the task The cumulative flight distance afterward; For drones Energy consumption rate per unit distance; For drones The inherent value; This is the maximum possible economic loss constant, used to normalize costs.
[0042] Based on the above objective function, in order to ensure the kinematic and logical feasibility of the UAV in the actual physical environment, the task allocation optimization problem must also strictly satisfy the following physical constraints: (4) Instantaneous load constraints: for each UAV At any point during the mission, it must be ensured that the instantaneous load carried does not exceed its maximum load capacity. .set up Indicates drone After executing the first in its sequence The instantaneous load carried by each node (a node can be a task point or a repository) is recursively updated as follows: (6) In the formula, This represents the cumulative payload requirement for all delivery missions assigned to the drone within the flight segment between the currently arrived warehouse and the next planned warehouse. Indicates drone The first in its sequence The payload of each node for unloading / picking up goods; when performing a delivery mission, the drone unloads goods, resulting in a decrease in payload; when performing a pickup mission, the drone loads items, resulting in an increase in payload; when arriving at the warehouse, the drone's payload status will be reset, and the reset amount depends on the smaller of the delivery demand for subsequent segments and its own maximum capacity.
[0043] (5) Closed-loop route constraints: Considering that the energy and range limitations of UAVs have been incorporated into the above-mentioned operating costs as soft constraints, in order to ensure the sustainable operation of the logistics system, the mission sequence of each UAV must meet the take-off and landing closed-loop conditions: (7) In the formula, Indicates drone The initial departure node in the flight path sequence. Indicates its final ending node, This represents the total number of nodes visited by the drone. This constraint requires that all drone mission routes must originate from any warehouse and ultimately return to any warehouse after completion.
[0044] Step 2: Initialize drone, warehouse, and mission parameters; the drone parameter information includes drone number, type, initial position, flight speed, maximum payload capacity, and maximum range; the warehouse parameter information includes warehouse location; the mission parameter information refers to the initial order number, location, payload requirement, and time window.
[0045] Step 3: As Figure 1 As shown, in order to enable drones to adapt to changes in the number of tasks in dynamic logistics scenarios, this invention improves the SAC (Soft Actor-Critic) reinforcement learning network based on the Transformer architecture, and obtains a forward-looking task selection strategy through single-machine training. The specific steps are as follows: Step 3.1: Obtain local state observations of the logistics scenario at the current moment, including the motion state and available payload state of a single drone, the real-time remaining payload requirements of unfinished tasks globally, the current alliance size, and the global warehouse status.
[0046] In a preferred embodiment, such as Figure 1 As shown in the dynamic local state module, in order to adapt to the random fluctuations of the task sequence over time, based on the UAV and task parameters from step two, the parameters at any given time are... drones Local state observation modeling is a state set :
[0047] in, express Time Drone The set of local states observed in the current logistics scenario; A subset of tasks available at the current moment (defined in step one) (The moment has arrived, but the task has not yet been completed). For warehouse collection. Specific features include: (1) UAV status characteristics Characterizing drones The state characteristics of the drone not only include the drone's kinematics and payload state, but also introduce the timeout risk of the current task sequence.
[0048] Kinematics includes real-time position and may also include other kinematic parameters. Payload status refers to the available payload state of the UAV. The minimum time margin criterion is preferred for assessing timeout risk. Quantify it. The calculation method is as follows: for drones For all tasks in the current task sequence, take the minimum difference between the deadline and the expected arrival time of each task, and normalize it to the interval [0,1]. The smaller the value, the higher the risk of timeout for the current task sequence.
[0049] (2) Global unfinished task status characteristics :in Representation task Task status characteristics, including task Real-time remaining payload requirements, currently allocated to execute this task The size of the drone alliance (the larger the size, the more saturated the demand, and the lower the priority), and drones The spatial distance ranking is based on the number of drones. and The ranking is obtained by sorting the flight distances of each mission in ascending order.
[0050] (3) Global warehouse status characteristics : Represents a collection of warehouses warehouse ; For warehouse The warehouse's state characteristics can be represented using static attributes, such as its location. By projecting the warehouse and the task into a unified feature space, the UAV can consider returning to the warehouse as an equivalent candidate action.
[0051] Step 3.2: Construct a joint feature representation based on local state observations, input it into the Transformer encoder, and output the global feature embedding.
[0052] In this embodiment, as Figure 1 As shown in the Transformer encoder module, the design includes an architecture comprising context feature fusion, an encoder, and a constraint-aware masking policy head, processing variable-length state inputs and outputting the probabilities of valid actions. Specifically: (1) Contextual feature fusion: Using a multilayer perceptron (MLP) to fuse the UAV state features Global unfinished task status characteristics and global repository state characteristics Projecting onto a unified high-dimensional feature space yields encoded UAV feature vectors, task feature vectors, and warehouse feature vectors. Since candidate action nodes share [a certain number of features]... One (containing all available tasks and repositories), expand the encoded drone feature vector Each vector is concatenated with the feature vectors of each task and the feature vector of the repository to generate a joint representation set. . These are the global unfinished task sets. and warehouse collection The number of elements in the data.
[0053] (2) Transformer encoding feature extraction: representing joint features Inputting a Transformer encoder block, the interaction weights between arbitrary neural network node pairs are calculated using a multi-head self-attention (MHSA) mechanism. This allows for the extraction of contextual global feature embeddings containing global topological structure and spatiotemporal dependencies, regardless of the order of the input sequence. .
[0054] (3) Constraint-Aware Validity Mask: Traditional reinforcement learning networks directly output the probability distribution over the entire action space, which cannot guarantee that the selected task meets physical constraints such as the maximum range and time window of the UAV. Therefore, the SAC network's policy (Actor) network receives the embedding of global features from the context. It outputs the logits in the discrete action space. To strictly enforce physical constraints, a binary mask vector is introduced at the output of the Actor network. This is used to strictly limit the probability of selecting illegal tasks that violate the maximum range constraint or mission time window constraint of the drone to 0.
[0055] In practice, the flight time, including the current time, mission location, drone location, and flight speed, as well as the mission loading / unloading time, can be used to calculate the mission end time to determine if the mission time window constraint has been violated. Similarly, the distance between the drone and the mission, and the total flight distance already covered by the drone, can be used to determine if the drone's maximum range constraint has been violated. For illegal missions that violate either the drone's maximum range or the mission time window constraint, their... The mask value is set to negative infinity, and the legal task value is set to 0. The final action probability distribution is calculated using the following formula: (8) Softmax is a normalization exponential function used to convert the input vector into a probability distribution.
[0056] This constraint-aware validity masking mechanism ensures that the probability of an illegal task being selected is strictly 0, thus limiting the exploration space to an absolutely feasible action domain.
[0057] Step 3.3: Conduct SAC strategy training, using reinforcement learning to train the drone to learn strategies to reduce delivery costs.
[0058] Due to the tasks in dynamic logistics scenarios The scale of the experience transfer sequence varies over time, and the length of the sequence collected at different times differs, making traditional fixed-length experience replay mechanisms unsuitable. This invention utilizes a reinforcement learning framework to train the aforementioned network. The Critic network is used to evaluate action value, and the Actor network is used to output the policy. The specific training process is as follows: (1) Critic and Actor Network Optimization: For the discrete action space, the Critic network explicitly calculates the expected value of the probability distribution output by the Actor network and minimizes the soft Bellman residual using a dual-evaluation network architecture; the Actor network updates its parameters by minimizing the KL divergence, so that the UAV selects high-value tasks while maintaining sufficient exploration entropy. In reinforcement learning training, the reward for each step is defined as the currently available task resulting from that step. The negative value of the cost is used to guide the drones to learn strategies to reduce delivery costs.
[0059] (2) Variable-length experience replay mechanism: In response to the problem of the constantly changing size of the task set in a dynamic logistics environment, the length of each experience transfer sequence depends on the number of available tasks at that moment. This results in inconsistent sequence lengths within the same batch. Therefore, when sampling mini-batch data containing Q transition sequences in the empirical replay buffer, all sequences are padded to the maximum length within that batch. This aligns the dimensions of the batch data, meeting the requirements of parallel network computation. The invalid gradients generated by the filling positions are addressed by the binary mask matrix constructed in point (3) below. The masking mechanism in the code ensures that network optimization relies solely on real-world environmental interaction data.
[0060] (3) Masking invalid padding gradients: To eliminate the interference of padding data on network optimization, a binary mask matrix is constructed. Elements are set to 1 when within the valid sequence length and to 0 when in the padding portion. During backpropagation, this mask matrix is used to weight the loss function. (9) In the formula, This is the batch average strategy loss obtained after applying mask weighting; The number of empirical sequences sampled from the empirical playback buffer in one operation; The length of the longest empirical sequence in the current batch; For experience sequence indexing; This serves as the position index for state-action data within the experience sequence; A binary mask matrix The Middle Line 1 The element in the column is set to 1 when the data at that position is actual interactive data, and to 0 when the data at that position is filler data; For the first The first in the empirical sequence The environmental state corresponding to each decision moment is used to describe the logistics delivery scenario information observed by the drone at that moment; For drones in status The task selection action executed below is used to indicate whether the drone chooses to join or leave a task alliance in the current logistics delivery scenario, or chooses to fly to a warehouse for resupply, unloading, or to end the current task sequence. For state ,action and strategy network parameters The corresponding single-sample policy loss. This mechanism ensures that the gradient comes only from effective environmental interactions, improving the convergence stability of the algorithm in dynamic environments.
[0061] Step 4: Transform the context-aware strategy learned on a single machine into a global utility for a multi-machine cluster. By embedding the well-trained strategy into an online overlapping association allocation framework, fast and stable allocation of cluster tasks is achieved. This includes the following steps: Step 4.1: To avoid resource conflicts caused by synchronous decision-making, an asynchronous iterative framework is used to decompose the generation process of the global alliance structure.
[0062] (1) Random sequence generation: in each iteration Initially, a random permutation function is used to generate the decision sequence for the UAV. This is to prevent specific drones from monopolizing high-value missions and to encourage clusters to explore diverse alliance combinations.
[0063] (2) Remaining demand update: Each UAV updates according to the decision sequence Decision-making task allocation; when it's the turn of the drone in the sequence... When making decisions, it no longer bases them on static initial load requirements. Instead, it is based on pre-emptive drones. The updated alliance structure and the observation of the currently available task set. Dynamic residual demand for each task .
[0064] (3) State Reconstruction: Unmanned Aerial Vehicle Based on real-time observation of dynamic remaining demand By combining real-time alliance feedback (such as the current alliance size), the task feature embedding is dynamically updated to construct a local state observation that reflects the precise spatiotemporal dependencies of the current iteration step. .
[0065] Step 4.2: Drone Based on the local state observation of the logistics scenario at the current moment, action reasoning is performed using a pre-trained Transformer encoder and a trained Actor network.
[0066] Local state observation The input is fed into a pre-trained Transformer policy model with frozen network parameters. A binary masking mechanism is used to filter out illegal tasks that violate physical constraints, and the probability distribution of executable actions is calculated. And determine the target task through random sampling. This mechanism effectively balances the utilization of high-probability, high-quality tasks with the exploration of potential global optimal solutions.
[0067] Step 4.3: To ensure the system continuously evolves towards a lower-cost allocation scheme, a hybrid stabilization mechanism based on "conditional joining" and "unconditional removal" is introduced for the output target task. Topology operation filtering and alliance structure evolution are performed. Furthermore, to prevent circular deadlocks, a tabu table-based deadlock prevention mechanism is introduced.
[0068] (1) Add Operation (ADD): If the selected target task Not in the current drone If the task is currently in the execution sequence, then the system will attempt to join the task alliance. The system will only accept the joining operation if and only if the operation satisfies the marginal cost change of equation (10).
[0069] (10) In the formula, , drones join in The optimal payload allocation vector before and after. Equation (10) represents the UAV joining the target mission. Joining a mission alliance can reduce the cost of a single mission.
[0070] (2) Remove operation: If the selected target task If the drone is already in the current execution sequence, it will exit the target mission. The mission alliance actively releases the payload occupied by that mission. Target mission. Enter the global set of unfinished tasks. To give the system the flexibility to escape local optima, this removal operation is unconditionally accepted, allowing the system to temporarily reduce the solution quality to reconstruct a better global union.
[0071] (3) Taboo list to prevent deadlock: Record the existing alliance structure during the search process in the taboo list. Within a set finite step size, strictly prohibit the system from performing reverse operations that would fall back to the state in the taboo list. Continuously loop the above asynchronous decision-making and structure evolution process until the maximum number of iterations is reached or the convergence condition is met, and output the final heterogeneous UAV swarm alliance task allocation scheme.
[0072] Step 5: Output the final heterogeneous UAV swarm alliance task allocation scheme. This may include, for example, the optimal (or suboptimal) flight trajectory sequence and payload allocation matrix for each UAV, and the global cost. The allocation results are quantitatively evaluated using this as an assessment indicator.
[0073] The following examples illustrate the implementation process and effects of this invention.
[0074] Example: The simulation hardware consists of an AMD Ryzen 9 7945HX 2.50 GHz processor and 32 GB of RAM, and the simulation language is C++17. This example sets up four heterogeneous drones collaboratively executing 10 logistics tasks. The scenario includes two warehouses. Given parameters such as drone locations and task locations, the overlapping alliance forming game (TSAC-OCF) algorithm, guided by reinforcement learning disclosed in this invention, is used to achieve collaborative task allocation for the drone swarm.
[0075] like Figure 1 As shown in this embodiment, the Overlapping Alliance Formation Game (TSAC-OCF) algorithm guided by reinforcement learning is disclosed. The specific implementation steps are as follows: Step 1: Consider a dynamic urban logistics delivery scenario where 4 heterogeneous drones collaborate to perform 10 tasks, and there are 2 warehouses in the scenario. Definition For drones, For a set of tasks, This is a collection of warehouses. The parameters for drones, missions, and warehouses are shown in Tables 1 and 2.
[0076] Table 1. Parameter settings for heterogeneous UAVs
[0077] In logistics scenarios, the goal of task allocation is to reduce overall costs. The optimization problem is shown in equation (11).
[0078] (11) Task Comprehensive cost function It consists of three weighted parts: (12) In the formula, These are the preset weighting coefficients, which are set to 100, 10, and 1 in this example.
[0079] Step Two: Initialize Drone, Warehouse, and Mission Parameters. The drone parameters include drone number, type, initial position, flight speed, maximum payload capacity, and maximum range. The warehouse parameters include warehouse location. The mission parameters refer to the initial order number, location, payload requirement, and time window. In this example, the parameter information for 4 drones, 2 warehouses, and 10 mission parameters are set as shown in Tables 2 and 3.
[0080] Table 2. Parameter Settings for Drones and Warehouses
[0081] Table 3 Task Parameter Settings
[0082] Step 3: Figure 2 To enhance the learning training curve, a pre-trained Transformer-SAC policy network is loaded, and initial feature states are extracted based on the current scene. To enable the drone swarm to adaptively allocate the 10 dynamically appearing logistics tasks in Table 3, the system deploys an improved SAC network model that has been trained and converged offline. The specific execution process is as follows: (1) State observation and reconstruction: for any time interval The system collected data from 4 drones listed in Table 2. to Real-time location, available payload, and time margin are used to construct the UAV state characteristics. Simultaneously observe Table 3 to Based on the remaining payload requirements and spatial location, and combined with the scale of the coalition executing the mission, mission status characteristics are constructed. And merge the two warehouses. , Location information construction .
[0083] (2) Context Feature Encoding: The above features are concatenated into a joint representation set and input into a Transformer encoder with frozen weights. A multi-head self-attention mechanism is used to extract the spatial topology and spatiotemporal dependencies between the current 4 heterogeneous UAVs and 10 tasks, and the context feature embedding is output. .
[0084] Step 4: Execute the Asynchronous Overlapping Alliance Formation Game (TSAC-OCF) for online collaborative task allocation. After obtaining the global feature embeddings, the four drones utilize the online overlapping alliance allocation framework for rapid decision-making. The specific evolutionary iterations are as follows: (1) Constraint perception and action reasoning: In each iteration, using a drone For example, the system inputs its features into the Actor network. Because The maximum flight range is limited to 30 km and the maximum payload is only 10 kg. The constraint-aware validity masking mechanism will generate a binary mask vector at the output. For tasks that are too far away or whose requirements exceed their payload capacity (e.g., remote locations). The mask value is set to negative infinity, making the probability of these illegal tasks being selected strictly 0. The final network output... The probability distribution of actions within the legal task space is determined by random sampling to identify the target task.
[0085] (2) Hybrid stability and coalition evolution: Unmanned aerial vehicles The task was selected in the first iteration. The system judges Not here In the current execution sequence, an add operation (ADD) is attempted. If The addition of reduced execution The overall cost (i.e., the sum of the load, timeliness, and operating costs of a single task in equation (12), is distinct from the global cost accumulated across all tasks). If a drone actively releases its occupied payload in order to explore a better solution, the system will unconditionally accept its removal operation (REMOVE). The updated alliance structure will serve as the real-time state input for subsequent drone decisions, without explicit notification, and the asynchronous iteration framework will automatically ensure information synchronization.
[0086] (3) Anti-deadlock iteration: Continuously record drones The evolutionary history of the alliance is appended to the taboo list. The above asynchronous reasoning and structural evolution process is continuously repeated until the global cost no longer decreases and the alliance structure tends to stabilize.
[0087] Step 5: Output the final heterogeneous UAV swarm alliance task allocation scheme and simulation result evaluation. After the iterative evolution in Step 4, the system finally outputs the optimal (or suboptimal) flight trajectory sequence and payload allocation matrix for 4 UAVs for 10 tasks. For example... Figure 3 As shown in the Gantt chart of neutron graph (a), blue represents delivery tasks, orange represents pickup tasks, and gray represents the warehouse. The four drones, based on the overlapping alliance concept, output a conflict-free task allocation scheme. Furthermore, when new tasks arise, the algorithm can dynamically adapt to the new tasks and generate new schemes to rapidly reduce costs. For example... Figure 3 As shown in neutron diagrams (b) and (c), at T=90 s, when faced with a new task T7, all four UAVs chose to prioritize the task reassignment scheme in order to execute the task within the time window constraint, which proves the effectiveness of the hybrid stabilization mechanism.
[0088] Monte Carlo simulation: To demonstrate the effectiveness of the proposed TSAC-OCF algorithm, this example compares it horizontally with optimization-type Distributed Genetic Algorithm (DGA) and market-type Contract Network Algorithm (CNP), and vertically with similar game theory algorithms (ROCF using a random strategy and HOCF using a heuristic formula). Four representative scales (4-10, 8-20, 16-40, and 32-80) are selected for Monte Carlo simulation comparison, with each algorithm running 1000 times for each scale to generate box plots.
[0089] like Figure 4As shown, the proposed algorithm has advantages in all four scales, and the advantages of the proposed algorithm are more obvious when the scale is larger. In the scale of 32-80, the proposed algorithm reduces the cost by 39.76% compared with the suboptimal algorithm (HOCF).
[0090] Based on the simulation results and analysis of the aforementioned UAV swarm task allocation example, it is evident that the reinforcement learning-guided Overlapping Alliance Formation Game (TSAC-OCF) algorithm described in this invention can improve the probability of forming a globally optimal alliance, reduce the UAV's policy space, accelerate the speed of UAV swarm collaborative task allocation, and reduce the cost of UAV swarms performing dynamic logistics delivery tasks. Therefore, this invention has strong engineering applicability and can achieve its intended purpose.
[0091] The specific embodiments described above only illustrate the design principles of the present invention. The shapes and names of the components in this description may differ and are not limited. Therefore, those skilled in the art can modify or make equivalent substitutions to the technical solutions described in the foregoing embodiments; and these modifications and substitutions do not depart from the inventive spirit and technical solutions of the present invention, and should all fall within the protection scope of the present invention.
Claims
1. A reinforcement learning-based heterogeneous UAV coalition task allocation method, suitable for dynamic logistics scenarios; multiple UAVs form a coalition to cooperatively complete a task, and a single UAV can simultaneously join multiple coalitions; characterized in that, include: Phase One: Adopt Reinforcement Learning, for single unmanned aircraft training alliance selection strategy: Step 11: Obtain local state observations of the logistics scenario at the current moment, including the motion state and available payload state of a single drone, the real-time remaining payload requirements of unfinished tasks globally, the current alliance size, and the global warehouse status; Step 12: Construct joint feature representation based on local state observations, input Encoder, output global feature embedding; Step 13: Embed global features into the input SAC network and train the drone to learn strategies to reduce delivery costs through reinforcement learning; Phase Two: Embed the trained policy network into an online overlapping alliance allocation framework for online collaborative task allocation. Step 21: Each iteration, each UAV decides task assignment in turn; when it is the turn of a UAV to make a decision, it is based on the pre-order UAV action decision updates the current coalition structure, and then determines the local state observation of the current time logistics scene, and uses the pre-trained encoder and the trained strategy network to perform action inference to determine the target task; Step 22: Based on the target tasks assigned to each UAV, output the final heterogeneous UAV swarm alliance task allocation scheme.
2. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, The local state observation is modeled as a set of states : wherein, denotes time instants unmanned aerial vehicles a set of local states observed at the current logistics scenario; a drone a drone state feature, including a real-time position of the drone, an available payload state, and a timeout risk of a current execution task sequence; a global unfinished task state feature, representing a global unfinished task set a task , characterizing a task state feature of the task , containing real-time residual load demand of the task , current allocated alliance scale of unmanned aerial vehicles performing the task , and spatial distance ranking of unmanned aerial vehicles ; the spatial distance ranking is a ranking obtained by ascending arrangement according to flight distances between unmanned aerial vehicles and tasks in the task set This represents the global repository state characteristics. Represents a collection of warehouses warehouse , For warehouse The warehouse status characteristics.
3. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, The method for determining the timeout risk of the currently executing task sequence is as follows: for drones For all tasks in the current task sequence, take the minimum difference between the deadline and the expected arrival time of each task, and normalize it to the interval [0,1]. The smaller the timeout risk value, the higher the timeout risk of the current task sequence.
4. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 2, characterized in that, Step 12 specifically includes: Using a multilayer sensor to analyze the state features of the drone Global unfinished task status characteristics and global repository state characteristics Encode the UAV feature vector, task feature vector, and warehouse feature vector; then expand the encoded UAV feature vector. Each feature vector is concatenated with the feature vectors of each task and the warehouse to generate a joint feature representation. ; These are the global unfinished task sets. and warehouse collection The number of elements in; Joint feature representation Inputting a Transformer encoder, a multi-head self-attention mechanism is used to extract contextual global feature embeddings containing the global topology and spatiotemporal dependencies of the current logistics scenario. .
5. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, In step 13, when training the drone to learn strategies to reduce delivery costs through reinforcement learning, the objective function of reinforcement learning finds the global cost. Minimum task; global cost Expressed as: in, For a global set of tasks; Representative task The load distribution vector, The total number of drones, Indicates drone In the mission The amount of load already allocated; Indicates a collection of drones; Task Comprehensive cost function It consists of three weighted parts: In the formula, These are preset weighting coefficients; The cost of insufficient payload is determined by the task. Unmet load requirements are expressed. To ensure timely service costs, responsible for evaluating the performance of drone swarm missions. The efficiency; for collaborative tasks, the timeliness cost of service will continue to increase until the last drone arrives; The operating expenses are calculated by taking into account both the conventional energy consumption costs of the drones during missions and the potential damage costs of the drones, and are the sum of the operating expenses of all individual drones.
6. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, In step 13, the policy network of the SAC network receives global feature embeddings and outputs log-probability in the discrete action space; a constraint-aware binary mask vector is introduced at the output of the policy network. This is used to strictly limit the probability of selecting illegal tasks that violate the maximum range constraint or mission time window constraint of the drone to 0.
7. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, In step 13, a variable-length experience playback mechanism is set up: When sampling batch data containing Q transition sequences in the experience replay buffer, all sequences are padded to the maximum length within that batch. This aligns the dimensions of the batch data; simultaneously, it constructs a binary mask matrix to shield invalid padding gradients. Elements within the valid sequence length are assigned a value of 1, while elements in the filling portion are assigned a value of 0. During backpropagation, the binary mask matrix is used. Weight the loss function.
8. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, In step 21, at the beginning of each iteration, a decision sequence for the UAVs is generated using a random permutation function, and each UAV makes a task allocation decision according to the decision sequence.
9. The heterogeneous UAV alliance task allocation method based on reinforcement learning as described in claim 1, characterized in that, In step 21, after determining the target task for the UAV through action reasoning in each iteration, the allocation result of UAV-target task is further confirmed based on a hybrid stabilization mechanism of "conditional joining" and "unconditional removal"; specifically: Conditional addition operation: If the selected target task Not in drones In the current execution task sequence, and add the target task. If joining a task alliance reduces the cost per task, then confirm the joining operation and set the target task. Add to the currently executing task sequence; Unconditional removal operation: If the selected target task Already in drones If the drone is in the current execution task sequence, then the drone will exit the target task. The mission alliance actively releases the target mission. Occupied by the drone's payload, target mission Enter the global unfinished task set and wait to reconstruct a better task alliance.
10. The method for heterogeneous UAV alliance task allocation based on reinforcement learning as described in claim 9, characterized in that, In step 21, after determining the target task for the UAV through action reasoning in each iteration, the allocation result of UAV-target task is further confirmed based on the taboo list that records the existing alliance structure. Within a set finite step size, reverse operations that would fall back to the alliance structure in the taboo list are prohibited.
