Unmanned aerial vehicle task prediction and distribution method based on multi-agent reinforcement learning

Abstract

The invention discloses an unmanned aerial vehicle task prediction and distribution method based on multi-agent reinforcement learning. According to the method, a task platform firstly predicts a task to appear by embedding a historical task sequence; and after the predicted task distribution and type are obtained, the platform performs different scheduling reward enabling on each region according to the current position condition of the UAV and the task demand density condition. And then, when the task arrives, the UAV serving as a single agent observes the environment condition, selects a task to be executed and notifies other UAVs to go to the task site later to complete the task. And the task platform adjusts the incentive scheduling reward during the execution period of the new round of task according to the completion condition of the UAV. According to the method, advanced scheduling of the UAV can be carried out through prediction, the requirement for matching of task types and UAV skills can be fully considered, and more reasonable and efficient task allocation is achieved.

Description

A Method for Unmanned Aerial Vehicle (UAV) Task Prediction and Allocation Based on Multi-Agent Reinforcement Learning Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) task allocation technology, and in particular to a method for UAV task prediction and allocation based on multi-agent reinforcement learning. Background Technology

[0002] With the continued opening of low-altitude airspace and the ongoing maturation of related technologies, the low-altitude economy, with unmanned aerial vehicles (UAVs) as its core carrier, is developing rapidly. UAVs are now widely used in various fields such as logistics and distribution, agricultural and forestry protection, power line inspection, emergency rescue, and urban governance. The operational model under the low-altitude economy is characterized by high frequency, high density, large scale, and diversification, with fleets of hundreds or even thousands of UAVs executing complex mission flows in urban and suburban airspace. This large-scale, networked operational approach places unprecedented demands on the level of intelligence in UAV mission management.

[0003] Task prediction and assignment is a technique that involves rationally planning and allocating resources to plan the task path for each available execution unit, assigning tasks to each unit, and having it perform the specified operations. In actual task execution, situations often arise where tasks require UAVs to possess specific skills (carrying different equipment or performing tasks in specific flight areas) to ensure successful completion. This problem is known as the multi-skill-constrained UAV scheduling problem. In this problem, task generation is often dynamic and random, and the UAV's state, including its current location and availability, changes over time. Dynamic methods need to consider the complex characteristics of real-world scenarios.

[0004] In drone mission prediction, traditional static or reactive scheduling algorithms are inefficient due to the uncertainty of dynamic environments. Furthermore, blind task allocation may lead to unnecessary resource consumption in subsequent scheduling, reducing task response speed and impacting overall mission completion efficiency. Matching drone skills with mission type is a core consideration in drone mission allocation. Drone swarms often use multi-agent reinforcement learning for mission allocation, but the algorithm's reward settings significantly influence the allocation, thus affecting the overall efficiency.

[0005] Existing technologies also include some research on multi-UAV task allocation fusion involving reinforcement learning, such as the invention patent application number 202311696323.4 entitled "Multi-UAV Task Allocation Fusion Method Based on Improved Reinforcement Learning." This invention incorporates clustering algorithms into multi-UAV task allocation, grouping similar states or actions and adding a spiral search strategy to improve the overall system performance, enabling the agent to learn and make decisions more efficiently. It addresses the problems of incomplete or duplicate task allocation that often occur during multi-UAV task allocation. However, it relies solely on a single optimization framework of clustering strategies and improved Q-learning networks. While it solves the problems of task allocation redundancy and insufficient clustering adaptability in heterogeneous UAV swarm scenarios by integrating multi-dimensional state perception and dynamic resource scheduling mechanisms, it still has shortcomings in real-time response capabilities under environmental uncertainties (such as dynamic threats and resource fluctuations), UAV range loss, and task completion rates. For example, the invention patent with application number 202410037582.4, entitled "A Method and Device for Dynamic Task Allocation of Unmanned Aerial Vehicles Based on Hierarchical Reinforcement Learning", uses hierarchical reinforcement learning to decompose complex tasks into sub-task hierarchical decisions. Although this improves the flexibility of decision-making, it has the defects of lagging collaboration between levels and limited adaptability to dynamic environments.

[0006] In summary, current task prediction and assignment systems face problems such as poor prediction performance and inability to meet the needs of multi-skill matching. There is an urgent need to explore an efficient and feasible solution, especially given the rapid development of the low-altitude economy and UAV technology. The system needs to simultaneously optimize UAV task allocation and real-time task scheduling to improve the efficiency and effectiveness of task completion in dynamic environments. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, the present invention aims to provide a method for UAV task prediction and allocation based on multi-agent reinforcement learning. In scenarios where both UAVs and tasks are dynamically changing, this method can not only perform advance scheduling of UAVs through prediction, but also fully consider the matching requirements between task types and UAV skills, thereby achieving more reasonable and efficient task allocation.

[0008] In the technical solution of this invention, the task platform first predicts upcoming tasks by embedding historical task sequences. After obtaining the predicted task distribution and type, the platform assigns different scheduling rewards to each region based on the current location of the UAV and the task demand density, providing additional rewards to UAVs that complete tasks in designated regions. Subsequently, when a task arrives, the UAV, acting as an individual agent, observes the environment, selects a task to execute, notifies other UAVs, and then proceeds to the task location to complete the task. Based on the UAVs' performance, the task platform adjusts the incentive scheduling rewards during the next round of task execution and continues to predict the next round of task occurrences, thereby achieving advance scheduling of UAVs. After a UAV completes a task, the system provides corresponding rewards and incentive scheduling rewards based on its performance and selects a new task to continue execution.

[0009] This invention adopts the following technical solution: a method for UAV task prediction and allocation based on multi-agent reinforcement learning, comprising the following steps:

[0010] Step 1: Before the process begins, the platform needs to confirm the number, skills, maximum travel distance, and load of UAVs that can participate in task allocation, and integrate the information of all UAVs into a candidate UAV set in order to match different task types.

[0011] Step 2: Before a new round of tasks arrives, the platform utilizes the reasoning capabilities of a time-series model to predict upcoming tasks based on the historical task set, including the location, type, and time of task occurrence. After obtaining the predicted task results, by analyzing the UAV situation in the current candidate UAV set and the location and density of the predicted tasks, an auxiliary reward is dynamically generated. This reward is obtained by scheduling UAVs to a certain area to obtain additional rewards, thereby regulating the number of UAVs in high-demand or low-demand areas.

[0012] Step 3: The UAV, acting as a single intelligent agent, observes the current environment and selects an executable task. It confirms whether the task type matches its own skills and whether it is within its reach. After confirmation, the UAV successfully receives the task and notifies other UAVs of the situation. Each UAV independently selects and executes the task.

[0013] Step 4: Based on the task allocation data of the new round of UAVs obtained in Step 3, the platform uses a large model to analyze the task completion status and response rate, dynamically adjusts the reward value of incentive scheduling, and continues to predict the next round of tasks.

[0014] Step 5: After completing the task, the UAV receives the corresponding reward and incentive scheduling reward, and then continues to select the next task to execute.

[0015] Furthermore, the UAV information in step 1 mainly includes: its current location, the types of skills it possesses, the current UAV's correspondence with a task, and the maximum distance it can move. After organizing all the UAV information, the platform will obtain a set of candidate UAVs.

[0016] Furthermore, the information mainly included in each task in the historical task set in step 2 includes: the geographical location where the task was generated, the time when the task appeared, the duration required to complete the task, the deadline of the task, the type of the task, and the time when the task started execution after it was assigned.

[0017] In step 2, each task has certain constraints (the time the task appears plus the time required to complete the task should be within a whole allocation period). The position of each task is fixed when it appears and does not change over time. Tasks have an appearance time and a required completion time. When a task is received by a UAV, the start time of the task is recorded. Using this information, the platform can determine the UAV set's response rate to tasks and timeout status, allowing for better allocation in the future. Simultaneously, a timeout completion is defined; while the UAV will receive a portion of the reward for completing the task, it will be subject to a timeout penalty, affecting the platform's subsequent task allocation priority and status for the UAV.

[0018] Furthermore, in step 2, the platform predicts the historical task set using a time-series model. This involves extracting skill sequences, location sequences, and timestamp sequences from the historical task set and embedding these three sequences in different ways. Next, the platform generates a prediction sequence through feature fusion and uses the time-series model to predict the possible task scenarios, including the number, location, time, and type of tasks. After obtaining the final embedded feature sequence, the time-series data is processed using a time-series model to finally obtain the number, location, time, and type of possible tasks.

[0019] Furthermore, in step 3, the observation of the current environment by a single UAV includes: the status of each UAV and its corresponding task, the current set of tasks to be assigned, and its current location. After completing the observation of the environment, the UAV will take a reasonable action and select a task. Then, through a series of confirmations, it will ensure that the task can be completed.

[0020] Furthermore, in step 4, the platform predicts the next tasks based on the completion status of the task set allocation. By judging the completion status, timeout penalties, and scheduling rewards in the previous round, the platform dynamically adjusts the parameter settings.

[0021] Furthermore, in step 5, the UAV will receive corresponding rewards after completing the current task, as well as a portion of the incentive scheduling reward. If the UAV completes a task that exceeds the time limit, it will be subject to a certain timeout penalty. After task settlement, the UAV will continue to select the next unassigned task to execute. The reward the UAV receives in a round of task allocation is calculated as the revenue from all completed tasks minus the penalty for completing a timeout task.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0023] 1. The present invention provides a method for UAV task prediction and allocation based on multi-agent reinforcement learning, which differs from traditional static scene modeling and single allocation techniques. It innovatively constructs a spatiotemporal coupling model of dual dynamic elements of task and worker, while fully considering the motion state of the task and UAV, as well as the impact of advance scheduling on task allocation efficiency. It introduces a task prediction strategy based on historical task sets, optimizes the path planning of UAV to complete tasks, and improves the task response rate and task allocation efficiency.

[0024] 2. This invention provides a UAV task prediction and allocation method based on multi-agent reinforcement learning. It constructs a multi-skill matching UAV task allocation method to address different types of actual low-altitude logistics task requirements. By scheduling UAVs with different skills, it completes tasks at different altitudes, with different loads, and with different requirements.

[0025] 3. The present invention provides a method for UAV task prediction and allocation based on multi-agent reinforcement learning, which improves the reward design in multi-agent reinforcement learning. It calculates an auxiliary reward module for optimizing UAV pre-scheduling through the demand map obtained from task prediction, and achieves reasonable allocation of UAVs and tasks by adding the auxiliary reward module to the reward design of multi-agent reinforcement learning, thereby improving the overall efficiency and benefits of task completion. Attached Figure Description

[0026] Figure 1 is a schematic diagram of the application scenario of the UAV task prediction and allocation method based on multi-agent reinforcement learning of the present invention;

[0027] Figure 2 is a flowchart of the overall process of the UAV task prediction and allocation method based on multi-agent reinforcement learning of the present invention.

[0028] Figure 3 is a schematic diagram of the UAV task prediction and allocation method based on multi-agent reinforcement learning of the present invention. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0030] The purpose of this invention is to address the shortcomings of existing technologies by providing a method for UAV task prediction and allocation based on multi-agent reinforcement learning.

[0031] Example 1

[0032] A method for UAV task prediction and allocation based on multi-agent reinforcement learning, as shown in Figures 1-3, includes the following steps:

[0033] Step 1: Initial Platform Confirmation

[0034] Step 1.1: Before the process begins, the platform needs to confirm the number of UAVs that can participate in task allocation and their skills in order to match them with different task types; the UAV information u is represented as:

[0035]

[0036] in This indicates that the UAV is at time The location, This indicates the skills possessed by the UAV. This indicates the current correspondence between the UAV and task t. This indicates the maximum distance that the UAV can travel.

[0037] Step 1.2: Next, after confirming the information of all UAVs, the platform integrates them into a candidate UAV set for subsequent task allocation. The candidate UAV set is represented as follows:

[0038] .

[0039] Based on the candidate UAV set, the platform determines the maximum distance each UAV can travel and its maximum load. Each UAV's location changes constantly as it performs a task, and each UAV also has a maximum load and maximum single-trip distance, and its skills should include the types required by the task. These are all factors the platform needs to consider when matching UAVs to tasks.

[0040] Step 2: Future Task Prediction

[0041] Step 2.1: Before a new round of tasks arrives, the platform predicts the upcoming tasks based on the historical task set, including the location, type, and time of each task. Each task t in the historical task set is represented as:

[0042]

[0043] in This indicates the geographical location where the task was generated. Indicates the time when the task occurred. This indicates the time required to complete the task. This indicates the deadline for the task. This indicates the time when the task started execution after it was assigned. This indicates the type of task, which corresponds to the skills required by the UAV.

[0044] Each task also has certain constraints, which are defined by the following formula:

[0045]

[0046] in This represents the entire allocation time for a task. The time a task appears plus the time required to complete it should fall within this entire allocation period. The position of each task is fixed upon its appearance and does not change over time. Tasks have an appearance time. And the required time to complete When a task is received by a UAV, the time when the task starts executing is recorded. These information platforms can determine the response rate and timeout status of the UAV set to tasks, so as to make better allocations in the future.

[0047] At the same time, the representation of a task completing after a timeout is defined as:

[0048]

[0049] Even if a UAV completes a task late, it will still receive a portion of the reward for completing the task, but it will also be subject to a timeout penalty, which will affect the platform's priority and allocation of tasks to the UAV in the future.

[0050] Step 2.2: The platform uses a time-series model to predict the historical task set. This involves extracting skill sequences, location sequences, and timestamp sequences from the historical task set and embedding these three sequences in different ways. The embedding calculation formula is as follows:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] in Represents an embedded skill sequence. Indicates the sine and cosine wave embedding method. This represents the learnable frequency parameters. This represents the embedded location timestamp sequence. Represents the relative time embedding matrix. These are query, key, and value matrices, respectively. This is the relative position offset function.

[0057] For skill sequences, a learnable embedding matrix is ​​used to map discrete skill identifiers to a dense vector space. For location sequences, a sine / cosine function is used to generate location codes, which provides the model with explicit location information while maintaining the continuity of relative positional relationships. For timestamp sequences, to effectively capture the temporal dependencies in UAV mission sequences, we propose an embedding encoding method based on relative temporal attention. This method considers not only absolute time information but also the relative temporal relationships between tasks, thus better modeling dynamic patterns in time series.

[0058] The platform then generates a prediction sequence through feature fusion and uses a time-series model to predict potential upcoming tasks, including the number, location, time, and type of tasks. The formula for feature fusion is as follows:

[0059]

[0060]

[0061]

[0062] in This represents the result after feature processing. This represents the final embedded sequence after feature fusion. After obtaining the final embedded feature sequence, the time series data is processed through a time series model to finally obtain the number, location, time, and type of possible tasks.

[0063] Step 2.3: After obtaining the predicted task results, the platform dynamically generates an auxiliary reward by analyzing the status of UAVs in the current candidate UAV set and the location and density of the predicted tasks. By scheduling UAVs to a certain area to obtain additional rewards, the platform can regulate the number of UAVs in high-demand or low-demand areas.

[0064] Step 3: The UAV, acting as a single intelligent agent, observes the current environment. The observation of the current environment by a single UAV is represented as follows:

[0065]

[0066] in Indicates the current UAV at time Environmental observation, This indicates the current set of tasks to be assigned;

[0067] After completing its environmental observations, the UAV will take a reasonable action and select a task, using the following formula:

[0068]

[0069] After selecting a task, the UAV must undergo a series of confirmations to ensure the task is feasible. The calculation formula is as follows:

[0070]

[0071]

[0072]

[0073] in This indicates the current location of the UAV. and This indicates the geographical location of the task. Each UAV can only accept one task at a time, and each task can only be completed by one UAV alone. The distance between the assigned UAV and the task cannot exceed the maximum distance that can be dispatched, and the type of task should correspond to the skills possessed by the UAV. After confirming this information, if execution is possible, the current UAV will officially accept the task, record the task start time, and notify other UAVs to prevent multiple UAVs from simultaneously selecting the same task.

[0074] Each UAV acts as an agent, observing the environment and selecting an unassigned task to perform.

[0075] Step 4: After a new round of UAV task allocation, the platform will obtain a new set of historical task data. Based on the allocation and completion status of this task set, predictions will be made for the next tasks, and the auxiliary reward value for incentive scheduling will be adjusted accordingly. The calculation formula is as follows:

[0076]

[0077] in Represents the reward function, , , These represent the coefficients for task completion, timeout penalty, and scheduling reward, respectively. This indicates that the agent has been penalized for exceeding the timeout limit. This represents the auxiliary reward an agent receives by being scheduled to a designated location through task dispatch.

[0078] After completing a round of task allocation, the platform will adjust the auxiliary reward value of incentive scheduling and the experience replay buffer of multi-agent reinforcement learning based on the response rate of UAV receiving tasks, the benefits of completing tasks, and the timeout of task completion, so as to improve the response rate of tasks, as well as the efficiency and benefits of completing tasks in the next round of allocation.

[0079] Step 5: After completing the current task, the UAV will receive the corresponding reward, as well as a portion of the incentive scheduling reward. If the UAV completes the task after a timeout, it will be subject to a timeout penalty. After task settlement, the UAV will continue to select the next unassigned task to execute. The reward calculation formula for the UAV in a round of task allocation is as follows:

[0080]

[0081] in This represents the reward the UAV receives in this round of task allocation, calculated as the total reward for all completed tasks minus the penalty for completing overtimed tasks.

[0082] The results will be presented below. This invention was tested using synthetic datasets K1 to K3 and a real dataset R1. The experiment involved four UAVs randomly generating tasks within a 10×20 km area over a period of time. Each UAV possessed two skills, with task types ranging from 1 to 3, and skill allocation was randomly generated within this range. The estimated completion time for each task fluctuated within a certain range.

[0083] To better demonstrate the performance of the experimental method, some metrics are defined below for testing the method.

[0084] For the task prediction part, this embodiment uses the root mean square error (RMSE) to evaluate skill prediction, location prediction, and time prediction. This represents the root mean square error of the skill sequence. This represents the root mean square error of the position sequence. This represents the root mean square error of the timestamp sequence.

[0085] For the task allocation part, the following indicators are used for evaluation in this embodiment:

[0086] Reward Value (REW): The reward value represents the number of tasks completed by the drone during the entire time period, as well as the portion completed beyond the time limit. It is an assessment of the overall task completion.

[0087] Completion Rate (CR): The percentage of tasks completed by drones out of all tasks that occur within a given time period.

[0088] Timeout Rate (TR): The percentage of all tasks completed by the drone within a given time period that took longer than the allotted time.

[0089] Response Rate (RR): For a task, the time difference between the occurrence of the task and its acceptance by the drone is used as an indicator.

[0090] The experimental results are shown in Tables 1, 2 and 3.

[0091] Table 1. Comparison of the method of the present invention with other task prediction methods

[0092]

[0093] Table 1 shows the performance of the method of this invention and other task prediction methods on task prediction. Transformer is a basic method for prediction based on sequences, while UniST is a method for prediction based on time sequences. As shown in Table 1, for task prediction, the method of this invention exhibits the best performance in skill prediction, location prediction, and time prediction, with an overall improvement of approximately 14% compared to the UniST algorithm and approximately 18% compared to the Transformer algorithm.

[0094] Table 2. Completion status of task assignment under different datasets

[0095]

[0096] Table 2 shows the performance of the proposed method in task allocation under different datasets. Dataset K1 represents a dataset with uniform task distribution that follows the actual task distribution and type ratio; dataset K2 represents a dataset with non-uniform task distribution that follows the actual task distribution and type ratio; and dataset K3 represents a dataset with uniform task distribution but where the task type is concentrated in regular city orders. As shown in Table 2, the fluctuation of various indicators of the proposed method is very small for different datasets, demonstrating strong stability and achieving good experimental results.

[0097] Table 3 Comparison with other task allocation methods

[0098]

[0099] Table 3 shows the performance of our proposed method and other task allocation methods in terms of task allocation completion. As can be seen from the data in Table 3, our proposed method achieves the best performance. For real-world datasets, the Random method exhibits significant bias due to its randomness, and the Greedy method fails to perform reasonable secondary scheduling. Compared to the well-performing EDF and MADDPG algorithms, our proposed method still offers some improvement. In the crucial REW metric, our proposed method achieves a 58% improvement over the EDF algorithm and approximately a 12% improvement over the MADDPG algorithm. Furthermore, the data indicates that the dynamic allocation method significantly outperforms the static method, achieving better allocation performance under dynamically changing task and UAV conditions.

[0100] The embodiments and implementation process of the present invention have been described in detail above with reference to the accompanying drawings and tables, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A method for UAV task prediction and allocation based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step 1: The platform initially confirms the information of participating UAVs and integrates them into a candidate UAV set; Step 2: Utilizing the reasoning capability of the time series model, the platform predicts the location, type, and time of the next task based on the historical task set, and dynamically generates an auxiliary reward for scheduling a UAV to a designated area; Step 3: The UAV, acting as a single agent in the environment, observes, selects an executable task, and notifies other UAVs; Step 4: The platform adjusts the incentive scheduling reward value based on the completion status of the new round of tasks and continues to predict the next round of tasks; Step 5: The UAV calculates the reward for completing the task, calculates the reward for the completed task and the reward for incentive scheduling, and selects the next round of tasks.

2. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, The information of the UAV mentioned in step 1 includes the UAV's dynamic location information, its skills, its correspondence with the task, and the maximum distance it can move. The platform will integrate the confirmed UAVs into a candidate set.

3. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 2, characterized in that, The specific characteristics of the historical task set in step 2 are: the geographical location where the task was generated, the time when it appeared, the duration required to complete the task, the deadline of the task, the time when the task started execution after it was assigned, and the type of task corresponding to the UAV's skills.

4. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 3, characterized in that, In step 2, the task has certain constraints. When the task is received, the start time of execution is recorded. Based on the task's response rate and timeout status, the subsequent allocation priority and status are adjusted through the large model.

5. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 1, characterized in that, In step 2, the platform uses a time series model to predict the historical task set. It embeds the skill sequence, location sequence, and timestamp sequence using three different embedding functions. The embedded sequences are then used to generate a prediction sequence through feature fusion to predict the number, location, time, and type of the next task.

6. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 5, characterized in that, In step 3, the UAV, acting as a single agent, observes the status of other UAVs in the current environment and the set of tasks to be assigned, selects a reasonable task, and performs a series of task confirmations to ensure that the task can be completed.

7. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 6, characterized in that, In step 4, based on the completion status of the task set allocation, the next tasks are predicted, and a reward function is used to represent the current system's status of task completion, timeout, and scheduling rewards.

8. The UAV task prediction and allocation method based on multi-agent reinforcement learning according to claim 7, characterized in that, In step 5, the UAV will receive corresponding rewards after completing the current task, as well as a portion of the incentive scheduling reward. If the UAV completes the task after the timeout, it will be subject to a certain timeout penalty.

Images