Priority-based uav cluster assistance task computation offloading method and device

Abstract

The application relates to a priority-based UAV cluster-assisted task computing offloading method and device. The method comprises the following steps: constructing a mobile edge computing system in cooperation with a UAV cluster, wherein the system comprises a plurality of UAVs providing computing offloading services and a plurality of ground intelligent devices; modeling a UAV cluster-assisted task computing offloading problem into a decentralized partially observable Markov decision process, wherein the process is described by observation, state, action and reward; constructing a priority-based HQMIX network to solve the UAV cluster-assisted task computing offloading problem, and obtaining a UAV cluster-assisted task offloading result, wherein the network is composed of a priority scheduling queue module, a local local Q network module and a global mixed Q network module. The method can realize the highest priority task offloading decision while guaranteeing the quality of service of the UAV cluster task.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, and in particular to a priority-based method and apparatus for calculating and offloading unmanned aerial vehicle (UAV) swarm-assisted tasks. Background Technology

[0002] In extreme application scenarios such as emergency response and disaster relief, the lack of robust basic communication network infrastructure, coupled with resource scarcity, makes it difficult to effectively support the demands of computationally intensive and latency-sensitive services. In these situations, drone swarms equipped with computing servers can serve as flexible aerial nodes, providing not only broad communication coverage for ground terminal equipment but also supplementing necessary auxiliary computing capabilities. However, the high-speed movement of drones leads to rapidly changing distances between them and ground terminals, further causing rapid fluctuations in wireless communication channel quality and connection status. Simultaneously, under limited computing power, providing indiscriminate communication and computing services to all tasks will inevitably result in a decline in service quality for resource-intensive critical services. Therefore, implementing differentiated services based on service type and assigning corresponding priorities to different tasks to achieve more efficient task scheduling has become a crucial issue that urgently needs to be addressed.

[0003] Multi-agent reinforcement learning effectively enhances the ability to handle complex and dynamic environmental problems and accelerates the learning process by enabling multiple agents to learn collaboratively or competitively through interaction. This approach not only enhances the robustness and efficiency of the system but also promotes the generation of innovative solutions. Classic algorithms such as Q-learning and Actor-Critic have been widely applied and developed in this field. However, most existing methods focus only on single-type agents and do not consider methods designed for heterogeneous agents. Furthermore, most existing methods operate under the premise of full observation, typically assuming that agents can observe global state information. In real-world scenarios, the observable information of agents is usually limited. Summary of the Invention

[0004] Therefore, it is necessary to provide a priority-based method and apparatus for unloading computational tasks in collaborative computing by UAV swarms to address the aforementioned technical problems, aiming to provide a more effective solution to the priority task scheduling problem in collaborative computational unloading of heterogeneous intelligent agents.

[0005] A priority-based method for unmanned aerial vehicle (UAV) swarm-assisted task computation and offloading, the method comprising:

[0006] Construct a mobile edge computing system for drone swarm collaboration. The system includes several drones that provide computing offloading services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of finite length to store incoming tasks.

[0007] The computational offloading problem of drone swarm-assisted tasks is modeled as a decentralized, partially observable Markov decision process, which is described by observations, states, actions, and rewards.

[0008] A priority-based HQMIX network is constructed to solve the computational offloading problem of UAV swarm assistance tasks, and the offloading results of UAV swarm assistance tasks are obtained. The HQMIX network consists of a priority scheduling queue module, a local Q network module, and a global hybrid Q network module.

[0009] The priority scheduling queue module is used to schedule and sort the newly arrived tasks in the buffer of each agent and the remaining tasks in the buffer of the previous time slot according to the task's latency and priority, and optimize to obtain the optimal task queue as the observation of each agent.

[0010] The local Q-network module contains several local Q-networks, which are the same number of agents as the number of agents. It is used to input the observations of each agent into the corresponding local Q-network for calculation and to estimate the local Q value corresponding to the possible actions of each agent.

[0011] The global hybrid Q-network module is used to combine the local Q-values ​​output by all local Q-networks into a global Q-value, and generate ground intelligent device task offloading decisions and UAV movement trajectories based on the global Q-value, which can achieve the highest priority.

[0012] In one embodiment, within the mobile edge computing system, each drone satisfies boundary, flight speed, and flight distance constraints;

[0013] Here, boundary constraints indicate that the drone's flight range does not exceed the pre-defined service area, expressed as:

[0014] ;

[0015] in, , , , Indicates the boundary of the service area; and They represent drones In the time slot The x and y coordinates of time Indicates a collection of drones and , Number of drones;

[0016] The flight speed constraint indicates that the drone's flight speed does not exceed its maximum speed, expressed as:

[0017] ;

[0018] in, and They represent drones In the time slot and time slot The horizontal coordinate at time, and , Indicates drone Maximum flight speed, The length of each time slot, For the set of all time slots, The number of time slots;

[0019] Flight distance constraint means that the distance between any two drones is not less than their minimum safe distance, expressed as:

[0020] ;

[0021] in, Indicates that in a drone swarm, excluding drones Another drone In the time slot Horizontal coordinates at time This indicates the minimum safe distance between drones.

[0022] In one embodiment, within a mobile edge computing system, the task offloading decision for ground-based intelligent devices includes processing the entire task locally or offloading the entire task to a drone for processing.

[0023] If the ground-based intelligent equipment processes the entire task locally, the ground-based intelligent equipment In the time slot The length of time is The task queue status is represented as ;in, Indicates ground-based intelligent devices In the time slot The first task in the task queue at that time One task, , and Representing tasks The number of time slots already waiting, the number of bits remaining to be calculated, and the estimated completion time. Indicates task Waiting time; Indicates ground-based intelligent devices At the start of the time slot The task generated at that time is located in the task queue at the 1st position. One location, , and These represent the total number of bits in the task, the CPU frequency required to calculate each bit, and the maximum allowed latency of the task, respectively. This represents the latency coefficient of the task. Indicates the priority of the task;

[0024] If ground-based intelligent devices offload all tasks generated to drones for processing, the drones In the time slot The length of time is The task queue state is represented as ;in, Indicates drone In the time slot The first task in the task queue at that time One task, , and Representing tasks The number of time slots already waiting, the number of bits remaining to be calculated, and the estimated completion time. Indicates task Waiting time;

[0025] Each task queue updates the order of all tasks at the beginning of each time slot, and the mobile edge computing system calculates tasks sequentially according to the order. This process continues until the drone and ground-based intelligent devices have finished processing the first task. The first task will be processed only after the first task is completed. One task.

[0026] In one embodiment, in a mobile edge computing system, a ground-based smart device With drones In the time slot The sets of tasks that can be completed on time are respectively represented as and ;in, The length of each time slot;

[0027] Ground intelligent devices With drones In the time slot The total priority for completion at any time is expressed as follows: and ;

[0028] Mobile edge computing systems in time slots The total priority for completion is expressed as follows: ;in, It represents a collection of ground-based intelligent devices. This refers to a collection of drones.

[0029] In one embodiment, the Markov decision process is described using observations, states, actions, and rewards, including:

[0030] For unmanned aerial vehicles (UAVs), the observations in the Markov decision-making process are described using the position, queue information, and priority information of each UAV, as well as the position, queue information, and priority information of the ground intelligent devices in the set of ground intelligent devices that each UAV can perceive, and the position information of other UAVs in the set of UAVs. For ground intelligent devices, the observations in the Markov decision-making process are described using the position, queue information, and priority information of each ground intelligent device, as well as the position information of each UAV in the set of UAVs that each ground intelligent device can perceive.

[0031] The state in the Markov decision-making process is described using the position, queue information, and priority information of all agents in the mobile edge computing system;

[0032] For drones, the actions in the Markov decision-making process are described using the queue information and position of each drone; for ground-based intelligent devices, the actions in the Markov decision-making process are described using the queue information and task offloading decisions of each ground-based intelligent device.

[0033] The rewards in the Markov decision process are described by the total priority of the mobile edge computing system, the total number of drone collisions, the total number of drones crossing boundaries, the scaling weight factor of priority, the scaling weight factor of collisions, and the scaling weight factor of boundary crossings.

[0034] In one embodiment, the specific process of the priority scheduling queue module in sorting tasks includes:

[0035] The tasks in the buffer corresponding to each agent are initially sorted. Specifically, they are arranged in reverse order according to the remaining waiting time of each task. Tasks that are about to time out will be prioritized at the front of the task queue, while tasks with relatively loose waiting time will be placed at the back of the task queue. For tasks with the same remaining waiting time in the buffer, they are arranged in order of priority from high to low to ensure that tasks that are about to time out and have high priority can be processed first.

[0036] Traverse the initially sorted tasks in each buffer and search for any timed-out tasks. If a timed-out task is the lowest priority task in the current task queue, it means that the task has the lowest value and is discarded directly. Otherwise, insert the timed-out task into each position before the timeout position in turn, obtain the optimal insertion position, and repeat the above steps until all tasks in each buffer do not time out. Obtain the optimal task queue in each buffer that can achieve the highest priority as the observation of each agent.

[0037] In one embodiment, the specific process of the local Q-network estimating the local Q-value includes:

[0038] The local Q-network consists of two fully connected layers and one gated recurrent unit. First, the agent's observations are input to the first fully connected layer, which outputs the agent's embedded feature vector. Then, the agent's embedded feature vector and the hidden vector from the previous time slot are input to the second gated recurrent unit, which outputs the hidden vector for the current time slot. Finally, the hidden vector for the current time slot is input to the third fully connected layer, which outputs the local Q-values ​​corresponding to all possible actions the agent can choose. The agent then adjusts its actions accordingly. - Greedy strategy and local Q value determine the action to be taken in the current time slot.

[0039] In one embodiment, the specific process by which the global hybrid Q-network module jointly generates the global Q-value includes:

[0040] The global hybrid Q network module consists of two fully connected layers. First, all local Q networks are time-slotted. The set of local Q-values ​​output at that time is a vector. Received through the first fully connected layer of the global hybrid Q network module As input, output Corresponding embedded feature vector , represented as

[0041] ;

[0042] in, , and These represent the weight matrix, bias term, and activation function of the first fully connected layer of the global hybrid Q-network module, respectively.

[0043] Will The input is fed into the second fully connected layer of the global hybrid Q network module, and the output is the global Q value, represented as...

[0044] ;

[0045] in, , and These represent the weight matrix, bias term, and activation function of the second fully connected layer of the global hybrid Q-network module, respectively. Indicates time slot The overall state of the mobile edge computing system in real time. Indicates time slot A collection of actions from drones and ground-based intelligent devices.

[0046] A priority-based drone swarm-assisted task computation offloading device, the device comprising:

[0047] The mobile edge computing system building unit is used to build a mobile edge computing system for drone swarm collaboration. The system contains several drones that provide computing offloading services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of finite length to store incoming tasks.

[0048] The unloading problem modeling unit is used to model the unloading problem of drone swarm-assisted task computation as a decentralized, partially observable Markov decision process, which is described by observations, states, actions, and rewards.

[0049] The priority-based unloading decision-making unit is used to construct a priority-based HQMIX network to solve the unloading problem of UAV swarm-assisted task computation, and obtain the unloading result of UAV swarm-assisted task. The HQMIX network consists of a priority scheduling queue module, a local Q-network module, and a global hybrid Q-network module. The priority scheduling queue module schedules and sorts newly arriving tasks in each agent's buffer and remaining tasks in the previous time slot according to task latency and priority, optimizing the task queue to serve as the observation for each agent. The local Q-network module contains several local Q-networks, matching the number of agents, and uses the observations of each agent as inputs to the corresponding local Q-network for computation, estimating the local Q-value corresponding to the possible actions chosen by each agent. The global hybrid Q-network module combines the local Q-values ​​output by all local Q-networks into a global Q-value, and generates the ground intelligent device task unloading decision and UAV movement trajectory based on the global Q-value, achieving the highest priority.

[0050] The aforementioned priority-based UAV swarm-assisted task computation offloading method and apparatus, by employing a priority-based task queue scheduling method within a constructed priority-based HQMIX network to update and sort tasks in the finite-length buffers of each agent in real time, can ensure the highest priority while guaranteeing the quality of service for the tasks. Furthermore, this HQMIX network utilizes a centralized offline training and distributed online execution architecture to perform heterogeneous multi-agent-assisted task computation offloading. Agents do not need to obtain global state information; they only rely on limited information for training, enabling the overall mobile edge computing system to obtain ground agent task offloading decisions and UAV flight trajectories that achieve the highest priority. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a priority-based drone swarm-assisted task calculation and offloading method in one embodiment.

[0052] Figure 2 This is a schematic diagram of a mobile edge computing system for drone swarm collaboration constructed in one embodiment;

[0053] Figure 3 This is a schematic diagram of a priority-based HQMIX network in one embodiment;

[0054] Figure 4 This is a schematic diagram comparing the training performance of the algorithm proposed in this application with different algorithms in one embodiment;

[0055] Figure 5 This is a schematic diagram comparing the average priority gain of this application with different algorithms under different numbers of ground intelligent devices in one embodiment;

[0056] Figure 6 This is a schematic diagram comparing the completion ratios of two types of tasks under different latency coefficients in one embodiment of the present application;

[0057] Figure 7 This is a schematic diagram comparing the completion ratios of two types of tasks under different task generation ratios in one embodiment of the present application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] In one embodiment, such as Figure 1 As shown, a priority-based method for drone swarm-assisted task computation offloading is provided, including the following steps:

[0060] Step 1: Construct a mobile edge computing system for drone swarm collaboration. The system includes several drones that provide computing offloading services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of finite length to store incoming tasks.

[0061] Specifically, mobile edge computing systems that enable drone swarm collaboration, such as Figure 2 As shown, the system includes several unmanned aerial vehicles (UAVs) providing computational offloading services and several ground intelligent devices (GUs). The set of ground intelligent devices and the set of UAVs respectively adopt... and To express, and Separately, the number of ground-based intelligent devices and the number of drones. Drones can operate throughout the entire service cycle. Provide computing offloading services for ground-based intelligent devices within its coverage area. Divided equally into There are 3 equal time slots, each with a length of 1. The set of all time slots can be represented as In each time slot At the beginning, smart devices on the ground will generate a task. All tasks are indivisible and can only be processed locally as a whole or offloaded to the drone for processing. Both drones and ground-based intelligent devices are equipped with buffers of limited length to store incoming tasks. It is particularly important to note that drones cannot obtain global environmental information about the entire network in advance; they can only detect ground-based intelligent devices and other drones within their perception range. Therefore, drones need to fly within their service area to explore task requirement information in order to improve the quality of service for the tasks.

[0062] This application establishes a three-dimensional Cartesian coordinate system to describe the position information of ground-based intelligent devices and drones. The position coordinates are drones At a fixed height Flying, its horizontal coordinate is Furthermore, each drone meets boundary, flight speed, and flight distance constraints, specifically including:

[0063] Boundary constraints indicate that the drone's flight range does not exceed the pre-defined service area, expressed as:

[0064] ;

[0065] in, , , , Indicates the boundary of the service area; and They represent drones In the time slot The x and y coordinates of time.

[0066] The flight speed constraint indicates that the drone's flight speed does not exceed its maximum speed, expressed as:

[0067] ;

[0068] in, and They represent drones In the time slot and time slot Horizontal coordinates at time Indicates drone Maximum flight speed.

[0069] Flight distance constraint means that the distance between any two drones is not less than their minimum safe distance, expressed as:

[0070] ;

[0071] in, Indicates that in a drone swarm, excluding drones Another drone In the time slot Horizontal coordinates at time This indicates the minimum safe distance between drones.

[0072] Within a mobile edge computing system, task offloading decisions for ground-based intelligent devices include either processing the entire task locally or offloading the entire task to a drone for processing. Figure 2 middle This represents ground-based intelligent devices. In the time slot The task unloading decision at that time. Represents ground-based intelligent devices The task will be computed locally. Represents ground-based intelligent devices The task will be Unload to drone The calculations are performed. Tasks arrive at the buffers of ground-based intelligent devices and drones in a queue, and each ground-based intelligent device and drone will use its maximum computing frequency to process the task at the head of the queue.

[0073] If the ground-based intelligent equipment processes the entire task locally, the ground-based intelligent equipment In the time slot The length of time is The task queue status is represented as ;in, Indicates ground-based intelligent devices In the time slot The first task in the task queue at that time One task, , and Representing tasks The number of time slots already waiting, the number of bits remaining to be calculated, and the estimated completion time. Indicates task Waiting time; Indicates ground-based intelligent devices At the start of the time slot The task generated at that time is located in the task queue at the 1st position. One location, , and These represent the total number of bits in the task, the CPU frequency required to calculate each bit, and the maximum allowable latency of the task, respectively. Tasks will be classified into two types: delay-sensitive (DS) and delay-tolerant (DT). This represents the latency coefficient of a task. The more critical the latency threshold, the larger the coefficient, reflecting the greater the importance of the task. Indicates the priority of the task.

[0074] If ground-based intelligent devices offload all tasks generated to drones for processing, the drones In the time slot The length of time is The task queue state is represented as ;in, Indicates drone In the time slot The first task in the task queue at that time One task, , and Representing tasks The number of time slots already waiting, the number of bits remaining to be calculated, and the estimated completion time. Indicates task The waiting time.

[0075] Each task queue updates the order of all tasks at the beginning of each time slot, and the mobile edge computing system calculates tasks sequentially according to the order. This process continues until the drone and ground-based intelligent devices have finished processing the first task. The first task will be processed only after the first task is completed. One task.

[0076] Specifically, let's take the queue model on the drone side as an example for explanation, task The transmission time is

[0077] ;

[0078] in, Ground-based intelligent devices With drones The transmission rate between them is calculated using the following formula:

[0079] ;

[0080] in, , , and These represent the bandwidth of the communication link between the ground intelligent device and the drone, the transmit power of the ground intelligent device, the channel gain, and the channel noise, respectively.

[0081] Computational tasks The time spent is

[0082] ;

[0083] in, It is a drone Computational speed, task The total time required to complete is

[0084] ;

[0085] in, Therefore, the number of bits remaining to be computed in this task can be calculated as follows:

[0086] ;

[0087] in, , .

[0088] Queue model of ground intelligent device terminal and The drone queue model is consistent with the time.

[0089] In mobile edge computing systems, ground-based intelligent devices With drones In the time slot The sets of tasks that can be completed on time are respectively represented as and .

[0090] Therefore, ground-based intelligent equipment With drones In the time slot The total priority for completion at any time is expressed as follows: and .

[0091] Mobile edge computing systems in time slots The total priority for completion is expressed as follows: ;in, It represents a collection of ground-based intelligent devices. This refers to a collection of drones.

[0092] Step 2: Model the unloading problem of drone swarm-assisted task computation as a decentralized, partially observable Markov decision process, which is described by observations, states, actions, and rewards.

[0093] Specifically, to ensure that the mobile edge computing system can complete the most priority tasks on time, this application establishes a system overall priority maximization problem by jointly considering the unloading decision of ground intelligent devices, the movement trajectory of the UAV, and the scheduling of tasks in the buffer. Since the UAV cannot acquire prior information about the environment and can only observe information within its perception range, the system overall priority maximization problem is transformed into a decentralized, partially observable Markov decision process. This process is described using observations, states, actions, and rewards, specifically including:

[0094] (1) Observation: For UAVs, they can only observe ground-based intelligent devices and other UAVs within their perception range. Information beyond their perception range cannot be observed. Therefore, UAVs In the time slot The observation information at that time included drones The drone's own location, queue information, priority information, and other information. The collection of ground-based intelligent devices that can be sensed Location, queue information, and priority information of ground-based intelligent devices, as well as drones. Other drones that can be sensed The location, queue information, and priority information of other drones are expressed as follows:

[0095] .

[0096] For ground-based intelligent devices In terms of time slots Real-time observation information includes smart devices on various ground surfaces. Its own location, queue information, priority information, and local ground intelligent devices The collection of drones that can be sensed The location information of the UAV is expressed as follows:

[0097] .

[0098] (2) State: State is the collection of all environmental information required by an agent. The state of a mobile edge computing system includes the location, queue information, and priority information of all agents. Specifically, time slots... The overall state of the system at that time is represented as

[0099] .

[0100] (3) Actions: For drones, in each time slot Within the buffer zone, each drone needs to prioritize all tasks and determine the execution order that achieves the highest priority. Furthermore, it needs to meticulously design flight paths to balance user load within the area, provide service to currently critical equipment, and respond to the demands of high-priority tasks. (Drone) In the time slot The actions at the time included drones The queue information and position are expressed as follows:

[0101] .

[0102] For ground-based intelligent devices, each device also needs to determine the execution order of tasks with the highest priority within its corresponding buffer. Simultaneously, the ground-based intelligent devices also need to decide whether to keep tasks for local computation or offload them to a specific drone based on the estimated available priority. In the time slot The actions at that time include ground-based intelligent devices Queue information and task unloading decisions, expressed as:

[0103] .

[0104] (4) Reward: To maximize the overall system priority while avoiding the aforementioned collision and out-of-bounds issues, the reward function is defined as the sum of the overall priority and the penalty terms for collision and out-of-bounds. Specifically, it is defined as follows:

[0105] ;

[0106] in, , , , as well as These are the total number of drone collisions, the total number of drones crossing the boundary, the priority scaling factor, the collision scaling factor, and the boundary crossing scaling factor. Tasks that time out in the queue will be discarded directly without further penalty.

[0107] Step 3: Construct a priority-based HQMIX network to solve the unloading problem of UAV swarm assistance task computation and obtain the unloading result of UAV swarm assistance task; wherein, the HQMIX network consists of a priority scheduling queue module, a local Q network module, and a global hybrid Q network module.

[0108] Priority-based HQMIX (Heterogeneous Q-mixing Networks) network architecture, such as... Figure 3 As shown.

[0109] in, Figure 3 Module (a) is the proposed priority scheduling queue module, used to schedule and sort newly arrived tasks in each agent's buffer and remaining tasks in the buffer of the previous time slot according to the task's latency and priority, optimizing to obtain the optimal task queue for each agent's observation. The specific task scheduling and sorting process is as follows:

[0110] First, the tasks in the buffer corresponding to each agent are initially sorted. Specifically, they are arranged in reverse order according to the remaining waiting time of each task. Tasks that are about to time out will be prioritized at the front of the task queue, while tasks with relatively loose waiting time will be placed at the back of the task queue. For tasks with the same remaining waiting time in the buffer, they are arranged in order of priority from high to low to ensure that tasks that are about to time out and have high priority can be processed first.

[0111] Subsequently, the tasks in each buffer are traversed after initial sorting. If there are any timed-out tasks, and the timed-out task is the lowest priority task in the current task queue, it means that the task has the lowest value and is directly discarded. Otherwise, the timed-out task is inserted into each position before the timeout position in turn to obtain the optimal insertion position. The above steps are repeated until all tasks in each buffer do not time out. The optimal task queue that can achieve the highest priority in each buffer is obtained as the observation of each agent.

[0112] In the HQMIX network architecture, the agent is limited by its observation range; therefore, a local Q-network is constructed for each agent, and several local Q-networks form a local Q-network module. Each agent inputs the information it can observe into its local Q-network to generate the corresponding local Q-value for all possible actions the agent can choose. In reality, estimating the corresponding Q-value based solely on partial observation information is extremely inaccurate. Therefore, as... Figure 3 As shown in module (b), this application constructs a neural network containing two fully connected layers and one gated recurrent unit for a local Q-network. (UAV) Taking a local Q-network as an example, the estimation of drones... The specific process for obtaining the corresponding local Q value includes:

[0113] First, the drone Observations Input is fed into the first fully connected layer, and output is the drone. Corresponding embedded feature vector Then Hidden layer vector of the previous time slot The input is obtained from the second-level gated recurrent unit, and the output is the hidden vector of the current time slot. ; will be The input is fed into the third fully connected layer, and the output is the drone. Local Q-values ​​for all possible actions And drones according to - Greedy strategy and local Q-value selection of the action to take in the current time slot .

[0114] Figure 3 Module (c) is the global hybrid Q-network module, consisting of two fully connected layers. This module is used to combine the local Q-values ​​output by all local Q-networks into a global Q-value, and generate, based on the global Q-value, the ground intelligent device task offloading decision and UAV movement trajectory that achieve the highest priority. The specific process of the global hybrid Q-network module in generating the global Q-value includes:

[0115] First, all local Q networks in time slots The set of local Q-values ​​output at that time is a vector. Received through the first fully connected layer of the global hybrid Q network module As input, output Corresponding embedded feature vector , represented as

[0116] ;

[0117] in, , and These represent the weight matrix, bias term, and activation function of the first fully connected layer of the global hybrid Q-network module, respectively.

[0118] Finally, The input is fed into the second fully connected layer of the global hybrid Q network module, and the output is the global Q value, represented as...

[0119] ;

[0120] in, , and These represent the weight matrix, bias term, and activation function of the second fully connected layer of the global hybrid Q-network module, respectively. Indicates time slot The overall state of the mobile edge computing system in real time. Indicates time slot This refers to the set of actions of two heterogeneous intelligent agents: drones and ground-based intelligent devices. Among them, the global hybrid Q-network module, based on global state information... Obtain the weights and bias parameters of your own network, and specifically use ReLU and absolute value activation functions.

[0121] Furthermore, the global hybrid Q-network module updates network parameters by minimizing the difference between the global Q-value and the target Q-value, thus aiding local Q-network decision-making. To ensure consistency between the optimal action in the local policy and the joint policy, the following monotonicity must be guaranteed:

[0122] ;

[0123] .

[0124] To verify the beneficial effects of this application, further simulation experiments were conducted. First, the algorithm proposed in this application (Priority-HQMIX) was compared with Priority-QMIX (a priority-based hybrid Q-network), PS-HQMIX (a priority-based heterogeneous hybrid Q-network), EDF-VDN (a minimum deadline-first value decomposition network), and FCFS-VDN (a first-come, first-served value decomposition network). Figure 4 The diagram illustrates a comparison of the training performance of the proposed algorithm with different algorithms, such as... Figure 4 As shown, the total number of priorities that the algorithm proposed in this application can obtain is far higher than that of other algorithms, and according to Figure 4 As shown in the convergence curve, this algorithm can converge quickly and smoothly.

[0125] Figure 5 The diagram illustrates a comparison of the average priority gain of this application with different numbers of ground-based intelligent devices and different algorithms, as shown below. Figure 5 As shown, the algorithm proposed in this application can achieve the highest priority gain under different numbers of ground-based intelligent devices. Figure 5This demonstrates that the algorithm proposed in this application exhibits strong robustness and adaptability when handling different user loads.

[0126] Figure 6 This illustration shows a comparison of the completion rates of DS and DT tasks under different latency coefficients in this application, such as... Figure 6 As shown, due to the strict latency requirements of DS tasks, the completion rate of DS tasks is significantly lower than that of DT tasks when the weights are the same. As the weight of DS tasks increases, their importance in the system also increases, and they can obtain more computing resources, thereby improving their completion rate. However, as DS tasks become increasingly important, system resources become insufficient, leading to a decrease in the completion rates of both DS and DT tasks.

[0127] Figure 7 This illustration shows a comparison of the completion ratios of DS and DT tasks under different task generation proportions, as shown in the diagram. Figure 7 As shown, with the increase in the proportion of DS tasks, the completion rate of DS tasks decreases due to limited computing resources. Simultaneously, when the proportion of DS tasks is below 50%, the system's computing resources are preferentially allocated to DS tasks, leading to a decrease in the completion rate of DT tasks. Once the proportion of DS tasks exceeds 50%, the increase in the number of DS tasks overwhelms the system's computing resources to meet their demands. Because DT tasks have more lenient latency requirements, their completion rate begins to rise and eventually surpasses that of DS tasks.

[0128] In summary, compared to existing technologies, this application, through its priority-based HQMIX network, employs a priority-based task queue scheduling method to update and sort tasks in the finite-length buffers of each agent in real time. This achieves the highest priority while ensuring the quality of service for tasks. Furthermore, this HQMIX network utilizes a centralized offline training and distributed online execution architecture to offload heterogeneous multi-agent collaborative task computation. Agents do not need to obtain global state information; they are trained using only limited information within a restricted scope. This enables the overall mobile edge computing system to acquire ground agent task offloading decisions and UAV flight trajectories that achieve the highest priority. Simulation results demonstrate that the proposed method can maximize the number of priorities, converges faster than existing benchmark schemes, and obtains the highest long-term reward.

[0129] In one embodiment, a priority-based drone swarm-assisted task computation offloading device is provided, comprising:

[0130] The mobile edge computing system building unit is used to build a mobile edge computing system for drone swarm collaboration. The system contains several drones that provide computing offloading services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of finite length to store incoming tasks.

[0131] The unloading problem modeling unit is used to model the unloading problem of drone swarm-assisted task computation as a decentralized, partially observable Markov decision process, which is described by observations, states, actions, and rewards.

[0132] The priority-based unloading decision-making unit is used to construct a priority-based HQMIX network to solve the unloading problem of UAV swarm-assisted task computation, and obtain the unloading result of UAV swarm-assisted task. The HQMIX network consists of a priority scheduling queue module, a local Q-network module, and a global hybrid Q-network module. The priority scheduling queue module schedules and sorts newly arriving tasks in each agent's buffer and remaining tasks in the previous time slot according to task latency and priority, optimizing the task queue to serve as the observation for each agent. The local Q-network module contains several local Q-networks, matching the number of agents, and uses the observations of each agent as inputs to the corresponding local Q-network for computation, estimating the local Q-value corresponding to the possible actions chosen by each agent. The global hybrid Q-network module combines the local Q-values ​​output by all local Q-networks into a global Q-value, and generates the ground intelligent device task unloading decision and UAV movement trajectory based on the global Q-value, achieving the highest priority.

[0133] Specific limitations regarding the priority-based UAV swarm-assisted task computation offloading device can be found in the limitations of the priority-based UAV swarm-assisted task computation offloading method described above, and will not be repeated here. Each module in the aforementioned priority-based UAV swarm-assisted task computation offloading device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0134] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0135] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A priority-based method for unmanned aerial vehicle (UAV) swarm-assisted task computation and offloading, characterized in that, The method includes: A mobile edge computing system for drone swarm collaboration is constructed. The system includes several drones that provide computing offloading services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of finite length to store incoming tasks. The computational offloading problem of drone swarm-assisted tasks is modeled as a decentralized, partially observable Markov decision process, which is described by observations, states, actions, and rewards. A priority-based HQMIX network is constructed to solve the unloading problem of UAV swarm assistance task computation, and the unloading result of UAV swarm assistance task is obtained; wherein, the HQMIX network consists of a priority scheduling queue module, a local Q network module, and a global hybrid Q network module; The priority scheduling queue module is used to sort and schedule newly arrived tasks in the buffer of each agent and the remaining tasks in the buffer of the previous time slot according to the task's delay and priority, and optimize to obtain the optimal task queue as the observation of each agent. The local Q-network module contains several local Q-networks, which are the same number of agents as the number of agents. It is used to input the observations of each agent into the corresponding local Q-network for calculation and to estimate the local Q value corresponding to the action that each agent may choose. The global hybrid Q-network module is used to combine the local Q-values ​​output by all local Q-networks into a global Q-value, and generate ground intelligent device task offloading decisions and UAV movement trajectories based on the global Q-value, which can achieve the highest priority. Within the mobile edge computing system, the task offloading decision for ground-based intelligent devices includes either processing the entire task locally or offloading the entire task to a drone for processing. If the ground-based intelligent equipment processes the entire task locally, the ground-based intelligent equipment In the time slot The length of time is The task queue status is represented as ;in, Indicates ground-based intelligent devices In the time slot The first task in the task queue at that time One task, , and Representing tasks The number of time slots already waiting, the number of bits remaining to be calculated, and the estimated completion time. Indicates task Waiting time; Indicates ground-based intelligent devices At the start of the time slot The task generated at that time is located in the task queue at the 1st position. One location, , and These represent the total number of bits in the task, the CPU frequency required to calculate each bit, and the maximum allowed latency of the task, respectively. This represents the latency coefficient of the task. Indicates the priority of the task; If ground-based intelligent devices offload all tasks generated to drones for processing, the drones In the time slot The length of time is The task queue state is represented as ;in, Indicates drone In the time slot The first task in the task queue at that time One task, , and Representing tasks The number of time slots already waiting, the number of bits remaining to be calculated, and the estimated completion time. Indicates task Waiting time; Each task queue updates the order of all tasks at the beginning of each time slot, and the mobile edge computing system calculates tasks sequentially according to the task order. This process continues until the UAV and ground-based intelligent devices have finished processing the first task. The first task will be processed only after the first task is completed. One task; In the mobile edge computing system, ground-based intelligent devices With drones In the time slot The sets of tasks that can be completed on time are respectively represented as and ;in, The length of each time slot; Ground intelligent devices With drones In the time slot The total priority for completion at any time is expressed as follows: and ; The mobile edge computing system in time slots The total priority for completion is expressed as follows: ;in, It represents a collection of ground-based intelligent devices. This refers to a collection of drones.

2. The method according to claim 1, characterized in that, Within the mobile edge computing system, each UAV satisfies boundary, flight speed, and flight distance constraints; Here, boundary constraints indicate that the drone's flight range does not exceed the pre-defined service area, expressed as: ； in, , , , Indicates the boundary of the service area; and They represent drones In the time slot x and y coordinates of time Indicates a collection of drones and , Number of drones; The flight speed constraint indicates that the drone's flight speed does not exceed its maximum speed, expressed as: ； in, and They represent drones In the time slot and time slot The horizontal coordinate at time, and , Indicates drone Maximum flight speed, The length of each time slot, For the set of all time slots, The number of time slots; Flight distance constraint means that the distance between any two drones is not less than their minimum safe distance, expressed as: ； in, Indicates that in a drone swarm, excluding drones Another drone In the time slot Horizontal coordinates at time This indicates the minimum safe distance between drones.

3. The method according to claim 1, characterized in that, The Markov decision process is described using observations, states, actions, and rewards, including: For unmanned aerial vehicles (UAVs), the observations in the Markov decision-making process are described using the position, queue information, and priority information of each UAV, as well as the position, queue information, and priority information of the ground intelligent devices in the set of ground intelligent devices that each UAV can perceive, and the position, queue information, and priority information of other UAVs in the set of UAVs that each ground intelligent device can perceive. For ground intelligent devices, the observations in the Markov decision-making process are described using the position, queue information, and priority information of each ground intelligent device, as well as the position information of each UAV in the set of UAVs that each ground intelligent device can perceive. The state in the Markov decision process is described using the position, queue information, and priority information of all agents in the mobile edge computing system; For unmanned aerial vehicles (UAVs), the actions in the Markov decision-making process are described using the queue information and position of each UAV; for ground-based intelligent devices, the actions in the Markov decision-making process are described using the queue information and task offloading decisions of each ground-based intelligent device. The rewards in the Markov decision process are described by the total priority of the mobile edge computing system, the total number of drone collisions, the total number of drones crossing boundaries, the scaling weight factor of the priority, the scaling weight factor of the collision, and the scaling weight factor of the boundary crossing.

4. The method according to claim 1, characterized in that, The specific process of task scheduling and sorting by the priority scheduling queue module includes: The tasks in the buffer corresponding to each agent are initially sorted. Specifically, they are arranged in reverse order according to the remaining waiting time of each task. Tasks that are about to time out will be prioritized at the front of the task queue, while tasks with relatively loose waiting time will be placed at the back of the task queue. For tasks with the same remaining waiting time in the buffer, they are arranged in order of priority from high to low to ensure that tasks that are about to time out and have high priority can be processed first. Traverse the initially sorted tasks in each buffer and search for any timed-out tasks. If a timed-out task is the lowest priority task in the current task queue, it means that the task has the lowest value and is discarded directly. Otherwise, insert the timed-out task into each position before the timeout position in turn, obtain the optimal insertion position, and repeat the above steps until all tasks in each buffer do not time out. Obtain the optimal task queue in each buffer that can achieve the highest priority as the observation of each agent.

5. The method according to claim 1, characterized in that, The specific process of estimating the local Q-value using the local Q-network includes: The local Q-network consists of two fully connected layers and one gated recurrent unit. First, the agent's observations are input to the first fully connected layer, which outputs the agent's corresponding embedded feature vector. Then, the agent's embedded feature vector and the hidden vector from the previous time slot are input to the second gated recurrent unit, which outputs the hidden vector for the current time slot. Finally, the hidden vector for the current time slot is input to the third fully connected layer, which outputs the local Q-values ​​corresponding to all possible actions the agent might choose. The agent then adjusts its local Q-values ​​according to the... - Greedy strategy and local Q value determine the action to be taken in the current time slot.

6. The method according to claim 1, characterized in that, The specific process by which the global hybrid Q-network modules jointly generate the global Q-value includes: The global hybrid Q-network module consists of two fully connected layers. First, all local Q-networks are time-slotted. The set of local Q-values ​​output at that time is a vector. Received through the first fully connected layer of the global hybrid Q network module As input, output Corresponding embedded feature vector , represented as ； in, , and These represent the weight matrix, bias term, and activation function of the first fully connected layer of the global hybrid Q-network module, respectively. Will The input is fed into the second fully connected layer of the global hybrid Q network module, and the output is the global Q value, represented as... ； in, , and These represent the weight matrix, bias term, and activation function of the second fully connected layer of the global hybrid Q-network module, respectively. Indicates time slot The overall state of the mobile edge computing system at that time. Indicates time slot A collection of actions from drones and ground-based intelligent devices.

7. A priority-based unmanned aerial vehicle (UAV) swarm-assisted task computation and offloading device based on the method of any one of claims 1-6, characterized in that, The device includes: A mobile edge computing system construction unit is used to build a mobile edge computing system for drone swarm collaboration. The system includes several drones that provide computing offloading services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of finite length to store incoming tasks. The unloading problem modeling unit is used to model the unloading problem of drone swarm-assisted task computation as a decentralized, partially observable Markov decision process, which is described by observations, states, actions, and rewards. A priority-based unloading decision-making unit is used to construct a priority-based HQMIX network to solve the unloading problem of UAV swarm-assisted task computation, and obtain the unloading result of the UAV swarm-assisted task. The HQMIX network consists of a priority scheduling queue module, a local Q-network module, and a global hybrid Q-network module. The priority scheduling queue module is used to schedule and sort newly arriving tasks in the buffers of each agent and the remaining tasks in the buffers of the previous time slot according to the task's latency and priority, optimizing to obtain the optimal task queue as the observation for each agent. The local Q-network module contains several local Q-networks equal to the number of agents, used to input the observations of each agent into the corresponding local Q-network for calculation, estimating the local Q-value corresponding to the possible actions selected by each agent. The global hybrid Q-network module is used to combine the local Q-values ​​output by all local Q-networks into a global Q-value, and generate the ground intelligent device task unloading decision and UAV movement trajectory that achieves the highest priority based on the global Q-value.

Images