Priority-based unmanned aerial vehicle cluster assisted task calculation unloading method and device

Abstract

The invention relates to a priority-based unmanned aerial vehicle cluster assisted task calculation unloading method and device. The method comprises the following steps: constructing an unmanned aerial vehicle cluster collaborative mobile edge computing system, wherein the system comprises a plurality of unmanned aerial vehicles for providing computing unloading services and a plurality of ground intelligent devices; the method comprises the following steps: modeling an unmanned aerial vehicle cluster assisted task calculation unloading problem into a decentralized partially observable Markov decision process, wherein the process is described by adopting observation, state, action and reward; a priority-based HQMIX network is constructed to solve an unmanned aerial vehicle cluster assisted task calculation unloading problem, an unmanned aerial vehicle cluster assisted task unloading result is obtained, and the network is composed of a priority scheduling queue module, a local Q network module and a global mixed Q network module. According to the method, the task service quality of the unmanned aerial vehicle cluster can be guaranteed while the task unloading decision with the highest priority is realized.

Description

Technical Field

[0001] The present application relates to the field of wireless communication technology, and in particular to a priority-based unmanned aerial vehicle cluster assisted task calculation offloading method and device. Background Art

[0002] In extreme application scenarios such as emergency response and emergency rescue, due to the lack of stable basic communication network facilities and the scarcity of resources, it is difficult to effectively support the needs of computing-intensive and delay-sensitive services. At this time, a drone cluster equipped with a computing server can serve as a flexible aerial node, not only providing extensive communication coverage for ground terminal equipment, but also supplementing the necessary auxiliary computing capabilities. However, the high-speed movement of drones causes the distance between them and the ground terminal to change rapidly, which further causes the rapid fluctuation of the quality and connection status of wireless communication channels. At the same time, under limited computing power conditions, if indiscriminate communication and computing services are provided for all tasks, it will inevitably lead to a decline in service quality for critical services with high resource requirements. In view of this, it has become an important issue to be solved urgently to implement differentiated services according to business types and give different tasks corresponding priorities to achieve more efficient task scheduling.

[0003] Multi-agent reinforcement learning effectively improves the ability to handle complex and dynamic environmental problems and accelerates the learning process by enabling multiple agents to learn collaboratively or competitively in interaction. This approach not only enhances the robustness and efficiency of the system, but also promotes the generation of innovative solutions. Classic algorithms such as Q-learning and Actor-Critic have been widely used and developed in this field. However, most existing methods only focus on a single type of agent and do not consider the design of methods for heterogeneous agents. In addition, most existing methods are carried out under the premise of full observation, usually assuming that the agent can observe global state information. In real scenarios, the information observable by the agent is usually limited. Summary of the invention

[0004] Based on this, it is necessary to provide a priority-based drone cluster assisted task computing offloading method and device to address the above technical problems, aiming to provide a more effective solution to the priority task scheduling problem of heterogeneous intelligent agents during collaborative computing offloading.

[0005] A priority-based unmanned aerial vehicle cluster assisted task computing offloading method, the method comprising: Build a mobile edge computing system coordinated by drone clusters. The system includes several drones that provide computing offload services and several ground intelligent devices. Drones and ground intelligent devices are collectively referred to as agents. The observation range of each agent is limited and each agent is equipped with a buffer of limited length to store incoming tasks. The problem of unloading computational tasks assisted by drone swarms is modeled as a decentralized partially observable Markov decision process, which is described by observations, states, actions, and rewards. A priority-based HQMIX network is constructed to solve the problem of unloading the computation of UAV cluster-assisted tasks, and the unloading results of UAV cluster-assisted tasks are obtained; the HQMIX network consists of a priority scheduling queue module, a local Q network module, and a global hybrid Q network module; Among them, the priority scheduling queue module is used to schedule and sort the newly arrived tasks in the buffer of each agent and the remaining tasks in the buffer in the previous time slot according to the delay and priority of the tasks, and optimize the optimal task queue as the observation of each agent; The local Q network module contains a number of local Q networks that match the number of agents, which are used to input the observations of each agent into the corresponding local Q network for calculation and estimate the local Q value corresponding to the action that each agent may choose; The global hybrid Q network module is used to combine the local Q values ​​output by all local Q networks into a global Q value, and generate the ground intelligent device task offloading decision and UAV movement trajectory that can achieve the highest priority based on the global Q value.

[0006] In one of the embodiments, in a mobile edge computing system, each drone satisfies boundary, flight speed, and flight distance constraints; Among them, the boundary constraint means that the flight range of the drone does not exceed the preset service area, and the expression is ; in, , , , Indicates the boundaries of the service area; and Respectively represent drones In time slot The horizontal and vertical coordinates of represents a collection of drones and , is the number of drones; The flight speed constraint means that the flight speed of the drone does not exceed its maximum speed. The expression is: ; in, and Respectively represent drones In time slot and time slot The horizontal coordinate at time , and , Indicates drone The maximum flight speed, For the length of each time slot, is the set of all time slots, is the number of time slots; The flight distance constraint means that the distance between any two drones is not less than their minimum safety distance, which is expressed as ; in, Indicates that the drone cluster is excluding drones Another drone outside In time slot The horizontal coordinate at time Indicates the minimum safe distance between drones.

[0007] In one of the embodiments, in the mobile edge computing system, the task offloading decision of the ground intelligent device includes processing the entire task locally or offloading the entire task to the drone for processing; If the ground intelligent device processes the entire task locally, the ground intelligent device In time slot The length of The task queue status is represented by ;in, Indicates ground intelligent equipment In time slot The first task in the task queue at tasks, , and Respectively represent tasks The number of time slots waited, the number of bits remaining to be calculated, and the estimated completion time, Indicates the task Waiting time; Indicates ground intelligent equipment Start at time slot The task generated when is located in the task queue Locations, , and They represent the total number of bits of the task, the CPU frequency required to calculate each bit, and the maximum allowed delay of the task. represents the delay coefficient of the task, Indicates the priority of the task; If the ground intelligent equipment offloads the generated tasks to the drone for processing, the drone In time slot The length of The task queue status is represented as ;in, Indicates drone In time slot The first task in the task queue at tasks, , and Respectively represent tasks The number of time slots waited, the number of bits remaining to be calculated, and the estimated completion time, Representation Task Waiting time; Each task queue re-updates the order of all tasks at the beginning of each time slot, and the mobile edge computing system calculates the tasks in the order of task arrangement. The first task will be processed after task.

[0008] In one embodiment, in a mobile edge computing system, a ground smart device With drones In time slot The set of tasks that can be completed on time are expressed as and ;in, is the length of each time slot; Ground smart devices With drones In time slot The total priority of completing and ; Mobile edge computing system in time slot The total priority of completing ;in, Represents a collection of ground smart devices. Represents a collection of drones.

[0009] In one embodiment, the Markov decision process is described using observations, states, actions, and rewards, including: For drones, the observations in the Markov decision process are described by the position, queue information, priority information of each drone itself, and the position, queue information, and priority information of the ground intelligent devices in the ground intelligent device set and other drones in the other drone sets that each drone can perceive; for ground intelligent devices, the observations in the Markov decision process are described by the position, queue information, priority information of each ground intelligent device itself, and the position information of each drone in the drone set that each ground intelligent device can perceive; The state in the Markov decision process is described by the location, queue information, and priority information of all agents in the mobile edge computing system; For drones, the actions in the Markov decision process are described by the queue information and position of each drone; for ground-based smart devices, the actions in the Markov decision process are described by the queue information and task offloading decisions of each ground-based smart device; The reward in the Markov decision process is described by the total priority of the mobile edge computing system, the total number of drone collisions, the total number of drone out-of-bounds violations, the scaling weight factor of the priority, the scaling weight factor of the collision, and the scaling weight factor of the out-of-bounds violation.

[0010] In one embodiment, the specific process of the priority scheduling queue module performing task scheduling and sorting includes: Perform a preliminary sorting of the tasks in the buffer corresponding to each agent. Specifically, they are sorted in reverse order according to the remaining waiting delay of each task. Tasks that are about to time out will be prioritized at the front of the task queue, and tasks with relatively loose delays will be prioritized at the end of the task queue. For tasks with equal remaining waiting delays in the buffer, they are sorted in order from high to low priority to ensure that tasks that are about to time out and have high priority can be processed first. Traverse the tasks in each buffer after preliminary sorting, and search for any timed-out tasks. If the timed-out task is the lowest priority task in the task queue of this section, it means that the value of the task is the lowest, and the task is directly discarded; otherwise, insert the timed-out task into each position before the timed-out position in turn, obtain the optimal insertion position, and repeat the above steps until all tasks in each buffer do not time out, and obtain the optimal task queue of each buffer that can achieve the highest priority as the observation of each intelligent agent.

[0011] In one embodiment, the specific process of estimating the local Q value by the local partial Q network includes: The local local Q network consists of two fully connected layers and one gated recurrent unit. First, the observation of the agent is input into the first fully connected layer, and the embedded feature vector corresponding to the agent is output. Then, the embedded feature vector corresponding to the agent and the hidden vector of the previous time slot are input into the second gated recurrent unit to obtain the hidden vector of the current time slot. The hidden vector of the current time slot is input into the third fully connected layer, and the local Q values ​​corresponding to all actions that the agent may choose are output. - The greedy strategy uses the local Q value to select the action to take in the current time slot.

[0012] In one embodiment, the specific process of the global hybrid Q network module jointly generating the global Q value includes: The global hybrid Q network module consists of two fully connected layers. First, all local local Q networks are connected in time slots. The local Q value set output when is a vector , received through the first fully connected layer of the global hybrid Q network module As input, output The corresponding embedding feature vector , expressed as ; in, , and They represent the weight matrix, bias term, and activation function of the first fully connected layer of the global hybrid Q network module respectively; Will Input to the second fully connected layer of the global hybrid Q network module, and output the global Q value, expressed as ; in, , and They represent the weight matrix, bias term, and activation function of the second fully connected layer of the global hybrid Q network module, respectively. Indicates time slot The overall state of the mobile edge computing system, Indicates time slot The action collection of drones and ground intelligent devices.

[0013] A priority-based unmanned aerial vehicle cluster assisted task computing offloading device, the device comprising: The mobile edge computing system construction unit is used to build a mobile edge computing system for drone cluster collaboration. The system includes several drones that provide computing offload services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of limited length to store incoming tasks. The offloading problem modeling unit is used to model the UAV cluster assisted task computing offloading problem as a decentralized partially observable Markov decision process, which is described by observation, state, action and reward; The priority-based offloading decision solving unit is used to construct a priority-based HQMIX network to solve the problem of unmanned aerial vehicle cluster assisted task computing offloading, and obtain the unmanned aerial vehicle cluster assisted task offloading result; wherein, the HQMIX network is composed of a priority scheduling queue module, a local local Q network module and a global hybrid Q network module; wherein, the priority scheduling queue module is used to schedule and sort the newly arrived tasks in the buffer of each intelligent agent and the remaining tasks in the buffer in the previous time slot according to the delay and priority of the task, and optimize the optimal task queue as the observation of each intelligent agent; the local local Q network module contains a number of local local Q networks consistent with the number of intelligent agents, which is used to input the observation of each intelligent agent into the corresponding local local Q network for calculation, and estimate the local Q value corresponding to the action that each intelligent agent may choose; the global hybrid Q network module is used to combine the local Q values ​​output by all local local Q networks into a global Q value, and generate the ground intelligent device task offloading decision and the UAV movement trajectory that can achieve the highest priority based on the global Q value.

[0014] The above-mentioned priority-based drone cluster assisted task computing offloading method and device can achieve the highest priority while ensuring the service quality of the task by using a priority-based task queue scheduling method in the constructed priority-based HQMIX network to update and sort the tasks in the limited length buffer of each agent in real time. In addition, the HQMIX network uses centralized offline training and distributed online execution architecture to perform heterogeneous multi-agent assisted task computing offloading. The agent does not need to obtain global state information, but only relies on partial information within a limited range for training, so that the overall mobile edge computing system can obtain the ground agent task offloading decision and drone flight trajectory that can achieve the highest priority. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The figure is a flowchart of a method for offloading calculation of a priority-based drone cluster-assisted task in one embodiment; Figure 2 A schematic diagram of a mobile edge computing system for drone cluster collaboration constructed in one embodiment; Figure 3 is a schematic diagram of a priority-based HQMIX network in one embodiment; Figure 4 A schematic diagram showing a comparison of the training performance of the algorithm proposed in the present application and different algorithms in one embodiment; Figure 5 A schematic diagram showing a comparison between average priority gains of the present application and different algorithms under different numbers of ground intelligent devices in one embodiment; Figure 6 A schematic diagram showing a comparison of completion ratios of two types of tasks under different delay coefficients of the present application in one embodiment; Figure 7 This is a schematic diagram comparing the completion ratios of two types of tasks under different task generation quantity ratios of the present application in an embodiment. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0017] In one embodiment, Figure 1 As shown, a priority-based UAV cluster assisted task computing offloading method is provided, comprising the following steps: Step 1: Build a mobile edge computing system coordinated by drone clusters. The system includes several drones that provide computing offload services and several ground intelligent devices. Drones and ground intelligent devices are collectively referred to as agents. The observation range of each agent is limited and each agent is equipped with a buffer of limited length to store incoming tasks.

[0018] Specifically, the mobile edge computing system of drone cluster collaboration is as follows: Figure 2 As shown in the figure, the system includes several unmanned aerial vehicles (UAVs) and several ground intelligent devices (GUs) that provide computing offloading services. The ground intelligent device set and the UAV set are respectively and To express, and The number of ground smart devices and the number of drones are respectively. Provide computing offloading services to ground intelligent devices within its coverage area. Divide equally into equal time slots, each of which is , the set of all time slots can be expressed as In each time slot At the beginning, all smart devices on the ground will have a task All tasks are inseparable and can only be processed locally or offloaded to drones for processing. Both drones and ground-based smart devices are equipped with a limited length buffer to store incoming tasks. It should be noted that drones cannot obtain the global environmental information of the entire network in advance and can only detect ground-based smart devices and other drones within their perception range. Therefore, drones need to fly within the service area to explore task demand information to improve the service quality of the task.

[0019] This application establishes a three-dimensional Cartesian coordinate system to describe the location information of ground smart devices and drones. The position coordinates are , drones At a fixed height Flight, its horizontal coordinate is In addition, each drone meets the boundary, flight speed and flight distance constraints, including: The boundary constraint means that the flight range of the drone does not exceed the preset service area, and the expression is ; in, , , , Indicates the boundaries of the service area; and Respectively represent drones In time slot The horizontal and vertical coordinates of time.

[0020] The flight speed constraint means that the flight speed of the drone does not exceed its maximum speed. The expression is: ; in, and Respectively represent drones In time slot and time slot The horizontal coordinate at time Indicates drone Maximum flight speed.

[0021] The flight distance constraint means that the distance between any two drones is not less than their minimum safety distance, which is expressed as ; in, Indicates that the drone cluster is excluding drones Another drone outside In time slot The horizontal coordinate at time Indicates the minimum safe distance between drones.

[0022] In the mobile edge computing system, the task offloading decision of the ground intelligent device includes processing the entire task locally or offloading the entire task to the drone for processing. Figure 2 middle Represents ground-based intelligent devices In time slot The task offloading decision at the time of Represents ground-based intelligent equipment The task will be calculated locally. Represents ground-based intelligent equipment The task will Offloading to drone The tasks arrive at the buffer of the ground intelligent device and the drone in the form of a queue, and each ground intelligent device and drone will call its maximum computing frequency to process the task at the head of the queue.

[0023] If the ground intelligent device processes the entire task locally, the ground intelligent device In time slot The length of The task queue status is represented by ;in, Indicates ground intelligent equipment In time slot The first task in the task queue at tasks, , and Respectively represent tasks The number of time slots waited, the number of bits remaining to be calculated, and the estimated completion time, Indicates the task Waiting time; Indicates ground intelligent equipment Start at time slot The task generated when is located in the task queue Locations, , and They represent the total number of bits of the task, the CPU frequency required to calculate each bit, and the maximum allowable delay of the task, respectively. Tasks will be classified into two types: delay-sensitive (DS) and delay-tolerant (DT); Indicates the delay coefficient of the task. The higher the delay threshold, the larger the coefficient of the task, which reflects that the task is more important. Indicates the priority of the task.

[0024] If the ground intelligent equipment offloads the generated tasks to the drone for processing, the drone In time slot The length of The task queue status is represented as ;in, Indicates drone In time slot The first task in the task queue at tasks, , and Respectively represent tasks The number of time slots waited, the number of bits remaining to be calculated, and the estimated completion time, Representation Task waiting time.

[0025] Each task queue re-updates the order of all tasks at the beginning of each time slot, and the mobile edge computing system calculates the tasks in the order of task arrangement. The first task will be processed after task.

[0026] Specifically, take the queue model on the drone side as an example to illustrate the task The transmission time is ; in, It is a ground-based intelligent device With drones The transmission rate between is calculated as ; in, , , and They respectively represent the bandwidth of the communication link between the ground intelligent device and the UAV, the rated transmission power of the ground intelligent device, the channel gain and the channel noise.

[0027] Computational tasks The time spent is ; in, It's a drone The calculation rate of the task The total time required to complete is ; in, Therefore, the number of bits remaining to be calculated for this task can be calculated as ; in, , .

[0028] Queue model and The UAV queue model is consistent with that of

[0029] In the mobile edge computing system, ground smart devices With drones In time slot The set of tasks that can be completed on time are expressed as and .

[0030] Therefore, ground intelligent equipment With drones In time slot The total priority of completing and .

[0031] Mobile edge computing system in time slot The total priority of completing ;in, Represents a collection of ground smart devices. Represents a collection of drones.

[0032] Step 2: Model the UAV cluster-assisted task computation offloading problem as a decentralized partially observable Markov decision process, which is described by observations, states, actions, and rewards.

[0033] Specifically, in order to ensure that the system mobile edge computing system can complete the most priority tasks on time, this application establishes the system total priority maximization problem by combining the unloading decision of ground intelligent devices, the movement trajectory of drones, and the scheduling of tasks in the buffer zone. Since drones cannot obtain prior information in the environment and can only observe information within the perception range, the problem of maximizing the total priority in the system is converted into a decentralized partially observable Markov decision process, which is described by observation, state, action, and reward, specifically including: (1) Observation: UAVs can only observe ground-based intelligent devices and other UAVs within their sensing range. Information beyond their sensing range cannot be observed. In time slot The observation information at that time includes drones Its own location, queue information, priority information, drone A collection of ground-based intelligent devices that can be sensed The location, queue information, and priority information of ground-based smart devices, as well as drones A collection of other drones that can be sensed The positions, queue information and priority information of other drones in the equation are expressed as .

[0034] For ground-based smart devices In terms of time slot The observation information at that time includes the smart devices on the ground Its own location, queue information, priority information, and various ground smart devices A collection of drones that can be sensed The position information of the UAV is expressed as .

[0035] (2) State: The state is the collection of all the environmental information required by the agent. The state of the mobile edge computing system includes the location, queue information, and priority information of all agents. Specifically, the time slot The overall state of the system is expressed as .

[0036] (3) Action: For the drone, in each time slot Each drone needs to sort all the tasks in the buffer and determine the execution order that can achieve the highest priority. In addition, the flight path needs to be cleverly designed to balance the user load in the area, provide services for the current important equipment, and respond to the needs of high-priority tasks. In time slot Actions including drones The queue information and position of .

[0037] For ground-based smart devices, each ground-based smart device also needs to determine the execution order of the tasks that can obtain the highest priority in its corresponding buffer. At the same time, the ground-based smart device also needs to decide whether to keep the task for local calculation or offload it to a drone for calculation based on the estimated priority that can be obtained. Ground-based smart devices In time slot The actions include ground intelligent devices The queue information and task offloading decision of .

[0038] (4) Reward: In order to maximize the total priority of the system and avoid the collision and out-of-bounds problems mentioned above, the reward function is defined as the sum of the total priority and the penalty items for collision and out-of-bounds. The specific definition is ; in, , , , as well as They are the total number of drone collisions, the total number of drone out-of-bounds, the priority scaling factor, the collision scaling factor, and the out-of-bounds scaling factor. Timed-out tasks in the queue will be discarded directly and will no longer be used as a penalty item.

[0039] Step 3, construct a priority-based HQMIX network to solve the UAV cluster assisted task computing offloading problem and obtain the UAV cluster assisted task offloading result; wherein, the HQMIX network consists of a priority scheduling queue module, a local local Q network module and a global hybrid Q network module.

[0040] The priority-based HQMIX (Heterogeneous Q-mixing Networks) network architecture is as follows Figure 3 shown.

[0041] in, Figure 3 The (a) module in the figure is the proposed priority scheduling queue module, which is used to schedule and sort the newly arrived tasks in the buffer of each agent and the remaining tasks in the buffer in the previous time slot according to the delay and priority of the task, and optimize the optimal task queue as the observation of each agent. The specific task scheduling and sorting process is: First, the tasks in the buffer corresponding to each agent are preliminarily sorted. Specifically, they are sorted in reverse order according to the remaining waiting delay of each task. Tasks that are about to time out will be prioritized at the front of the task queue, and tasks with relatively loose delays will be placed at the end of the task queue. For tasks with equal remaining waiting delays in the buffer, they are sorted in order from high to low priority to ensure that tasks that are about to time out and have high priority can be processed first. Subsequently, the tasks in each buffer that have been preliminarily sorted are traversed to search for any timed-out tasks. If the timed-out task is the lowest priority task in the task queue of this section, it means that the value of the task is the lowest and the task is discarded directly. Otherwise, the timed-out task is inserted into each position before the timed-out position in turn to obtain the optimal insertion position. The above steps are executed repeatedly until all tasks in each buffer do not time out, and the optimal task queue that can achieve the highest priority in each buffer is obtained as the observation of each intelligent agent.

[0042] In the HQMIX network architecture, the agent is limited by the observation range, so a local Q network is constructed for each agent, and several local Q networks form a local Q network module. Each agent inputs the information it can observe into the local Q network to generate the corresponding local Q value for all the actions that the current agent may choose. In fact, it is extremely inaccurate to estimate the corresponding Q value based on only partial observation information. For this reason, Figure 3 As shown in module (b), this application constructs a neural network containing two fully connected layers and one gated recurrent unit for the local Q network. As an example, the local Q network estimates the drone The specific process of the corresponding local Q value includes: First, the drone Observation Input to the first fully connected layer, output drone The corresponding embedding feature vector ; then and the hidden vector of the previous time slot Input to the second layer gated recurrent unit to obtain and output the hidden layer vector of the current time slot ; will be Input to the third fully connected layer, and output the drone The local Q values ​​corresponding to all possible actions , and drones according to - Greedy strategy and local Q value select the action to take in the current time slot .

[0043] Figure 3 The (c) module in the figure is the global hybrid Q network module, which consists of two fully connected layers. This module is used to combine the local Q values ​​output by all local Q networks into a global Q value, and generate the ground intelligent device task offloading decision and drone movement trajectory that can achieve the highest priority based on the global Q value. The specific process of the global hybrid Q network module jointly generating the global Q value includes: First, all local partial Q networks are set in time slot The local Q value set output when is a vector , received through the first fully connected layer of the global hybrid Q network module As input, output The corresponding embedding feature vector , expressed as ; in, , and They represent the weight matrix, bias term, and activation function of the first fully connected layer of the global hybrid Q network module respectively; Finally, Input to the second fully connected layer of the global hybrid Q network module, and output the global Q value, expressed as ; in, , and They represent the weight matrix, bias term, and activation function of the second fully connected layer of the global hybrid Q network module, respectively. Indicates time slot The overall state of the mobile edge computing system, Indicates time slot The global hybrid Q network module is based on the global state information. Get the weight and bias parameters of its own network. The activation function specifically uses ReLu and absolute value activation function.

[0044] Furthermore, the global hybrid Q network module updates the network parameters by minimizing the difference between the global Q value and the target Q, helping the local Q network to make decisions. In order to ensure the consistency of the optimal actions in the local strategy and the joint strategy, the following monotonicity needs to be guaranteed: ; .

[0045] To verify the beneficial effects of this application, simulation experiments were further conducted. First, the algorithm proposed in this application (Priority-HQMIX) was compared with Priority-QMIX (priority-based hybrid Q network), PS-HQMIX (heterogeneous hybrid Q network based on priority scheduling), EDF-VDN (value decomposition network based on minimum deadline priority), and FCFS-VDN (value decomposition network based on first-come, first-served) algorithms. Figure 4 A schematic diagram showing the training performance comparison between the algorithm proposed in this application and different algorithms is shown, Figure 4 As shown, the total number of priorities that the algorithm proposed in this application can obtain is much higher than other algorithms, and according to Figure 4 The convergence curve shown in shows that this algorithm can converge quickly and smoothly.

[0046] Figure 5 The following diagram shows the comparison between the average priority gain of this application and different algorithms under different numbers of ground intelligent devices. Figure 5 As shown, the algorithm proposed in this application can obtain the highest priority gain under different scales of ground intelligent devices. Figure 5 It shows that the algorithm proposed in this application has strong robustness and adaptability when dealing with different user loads.

[0047] Figure 6 The following is a schematic diagram showing the comparison of the completion ratios of the two types of tasks, DS tasks and DT tasks, under different delay coefficients in this application. Figure 6 As shown in Figure 1, due to the strict delay requirements of DS tasks, when the weights are the same, the completion rate of DS tasks is significantly lower than that of DT tasks. As the weights of DS tasks increase, their importance in the system also increases, and they can obtain more computing resources, thereby improving the completion rate. However, as DS tasks become more and more important, system resources will become insufficient, resulting in a decrease in the completion rate of DS tasks and DT tasks.

[0048] Figure 7 The following is a schematic diagram showing the comparison of the completion ratios of the two types of tasks, DS tasks and DT tasks, under different ratios of the number of tasks generated in this application. Figure 7 As shown in the figure, as the proportion of DS tasks increases, the completion rate of DS tasks decreases due to limited computing resources. At the same time, when the proportion of DS tasks is lower than 50%, the system's computing resources are preferentially allocated to DS tasks, resulting in a decrease in the completion rate of DT tasks. Once the proportion of DS tasks exceeds 50%, the number of DS tasks increases, and the system's computing resources cannot meet the needs of DS tasks. Since DT tasks have more relaxed requirements on latency, their completion ratios begin to rise and eventually exceed those of DS tasks.

[0049] In summary, compared with the prior art, the present application uses a priority-based task queue scheduling method in the constructed priority-based HQMIX network to update and sort the tasks in the limited-length buffer of each agent in real time, which can achieve the highest priority while ensuring the service quality of the task. In addition, the HQMIX network uses centralized offline training and distributed online execution architecture to perform heterogeneous multi-agent assisted task computation offloading. The agent does not need to obtain global state information, but only relies on partial information within a limited range for training, so that the overall mobile edge computing system can obtain the highest priority ground agent task offloading decision and drone flight trajectory. And the simulation results show that the method proposed in this application can maximize the priority, converge faster than the existing benchmark scheme, and obtain the highest long-term reward.

[0050] In one embodiment, a priority-based UAV cluster assisted task computing offloading device is provided, comprising: The mobile edge computing system construction unit is used to build a mobile edge computing system for drone cluster collaboration. The system includes several drones that provide computing offload services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of limited length to store incoming tasks. The offloading problem modeling unit is used to model the UAV cluster assisted task computing offloading problem as a decentralized partially observable Markov decision process, which is described by observation, state, action and reward; The priority-based offloading decision solving unit is used to construct a priority-based HQMIX network to solve the problem of unmanned aerial vehicle cluster assisted task computing offloading, and obtain the unmanned aerial vehicle cluster assisted task offloading result; wherein, the HQMIX network is composed of a priority scheduling queue module, a local local Q network module and a global hybrid Q network module; wherein, the priority scheduling queue module is used to schedule and sort the newly arrived tasks in the buffer of each intelligent agent and the remaining tasks in the buffer in the previous time slot according to the delay and priority of the task, and optimize the optimal task queue as the observation of each intelligent agent; the local local Q network module contains a number of local local Q networks consistent with the number of intelligent agents, which is used to input the observation of each intelligent agent into the corresponding local local Q network for calculation, and estimate the local Q value corresponding to the action that each intelligent agent may choose; the global hybrid Q network module is used to combine the local Q values ​​output by all local local Q networks into a global Q value, and generate the ground intelligent device task offloading decision and the UAV movement trajectory that can achieve the highest priority based on the global Q value.

[0051] For the specific limitations of the priority-based drone swarm assisted task calculation unloading device, please refer to the limitations of the priority-based drone swarm assisted task calculation unloading method above, which will not be repeated here. Each module in the above-mentioned priority-based drone swarm assisted task calculation unloading device can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0052] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0053] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A priority-based UAV cluster-assisted task computing offloading method, characterized in that: The method comprises: Construct a mobile edge computing system coordinated by drone clusters, which includes several drones that provide computing offload services and several ground intelligent devices. The drones and ground intelligent devices are collectively referred to as agents. The observation range of each agent is limited and each agent is equipped with a buffer of limited length for storing incoming tasks. The problem of unloading computational tasks assisted by drone swarms is modeled as a decentralized partially observable Markov decision process, which is described by observations, states, actions, and rewards. Construct a priority-based HQMIX network to solve the problem of unloading the calculation of UAV cluster-assisted tasks, and obtain the unloading results of UAV cluster-assisted tasks; wherein the HQMIX network consists of a priority scheduling queue module, a local Q network module, and a global hybrid Q network module; The priority scheduling queue module is used to schedule and sort the newly arrived tasks in the buffer of each agent and the remaining tasks in the buffer in the previous time slot according to the delay and priority of the tasks, and optimize the optimal task queue as the observation of each agent; The local local Q network module includes a number of local local Q networks that is consistent with the number of agents, and is used to input the observation of each agent into the corresponding local local Q network for calculation, and estimate the local Q value corresponding to the action that each agent may choose; The global hybrid Q network module is used to combine the local Q values ​​output by all local partial Q networks into a global Q value, and generate a ground intelligent device task offloading decision and a UAV movement trajectory that can achieve the highest priority based on the global Q value.

2. The method according to claim 1, characterized in that In the mobile edge computing system, each drone satisfies boundary, flight speed and flight distance constraints; Among them, the boundary constraint means that the flight range of the drone does not exceed the preset service area, and the expression is ； in, , , , Indicates the boundaries of the service area; and Respectively represent drones In time slot The horizontal and vertical coordinates of represents a collection of drones and , is the number of drones; The flight speed constraint means that the flight speed of the drone does not exceed its maximum speed. The expression is: ； in, and Respectively represent drones In time slot and time slot The horizontal coordinate at time , and , Indicates drone The maximum flight speed, For the length of each time slot, is the set of all time slots, is the number of time slots; The flight distance constraint means that the distance between any two drones is not less than their minimum safety distance, which can be expressed as ； in, Indicates that the drone cluster is excluding drones Another drone outside In time slot The horizontal coordinate at time Indicates the minimum safe distance between drones.

3. The method according to claim 2, characterized in that In the mobile edge computing system, the task offloading decision of the ground intelligent device includes processing the entire task locally or offloading the entire task to the drone for processing; If the ground intelligent device processes the entire task locally, the ground intelligent device In time slot The length of The task queue status is represented by ;in, Indicates ground intelligent equipment In time slot The first task in the task queue at tasks, , and Respectively represent tasks The number of time slots waited, the number of bits remaining to be calculated, and the estimated completion time, Indicates the task Waiting time; Indicates ground intelligent equipment Start at time slot The task generated when is located in the task queue Locations, , and They represent the total number of bits of the task, the CPU frequency required to calculate each bit, and the maximum allowed delay of the task. represents the delay coefficient of the task, Indicates the priority of the task; If the ground intelligent equipment offloads the generated tasks to the drone for processing, the drone In time slot The length of The task queue status is represented as ;in, Indicates drone In time slot The first task in the task queue at tasks, , and Respectively represent tasks The number of time slots waited, the number of bits remaining to be calculated, and the estimated completion time, Indicates the task Waiting time; Each task queue re-updates the order of all tasks at the beginning of each time slot, and the mobile edge computing system calculates the tasks in the order of task arrangement. The first task will be processed after task.

4. The method according to claim 3, characterized in that In the mobile edge computing system, ground intelligent devices With drones In time slot The set of tasks that can be completed on time are expressed as and ;in, is the length of each time slot; Ground smart devices With drones In time slot The total priority of completing and ; The mobile edge computing system in the time slot The total priority of completing ;in, Represents a collection of ground smart devices. Represents a collection of drones.

5. The method according to claim 1, characterized in that The Markov decision process is described using observations, states, actions, and rewards, including: For drones, the observations in the Markov decision process are described by the position, queue information, priority information of each drone itself, and the position, queue information, and priority information of the ground smart devices in the ground smart device set and other drones in the other drone sets that each drone can perceive; for ground smart devices, the observations in the Markov decision process are described by the position, queue information, priority information of each ground smart device itself, and the position information of each drone in the drone set that each ground smart device can perceive; The state in the Markov decision process is described by the location, queue information, and priority information of all agents in the mobile edge computing system; For drones, the actions in the Markov decision process are described by the queue information and position of each drone; for ground smart devices, the actions in the Markov decision process are described by the queue information and task offloading decisions of each ground smart device; The reward in the Markov decision process is described by the total priority of the mobile edge computing system, the total number of drone collisions, the total number of drone cross-borders, the scaling weight factor of the priority, the scaling weight factor of the collision, and the scaling weight factor of the cross-border.

6. The method according to claim 1, characterized in that The specific process of the priority scheduling queue module performing task scheduling and sorting includes: Perform a preliminary sorting of the tasks in the buffer corresponding to each agent. Specifically, they are sorted in reverse order according to the remaining waiting delay of each task. Tasks that are about to time out will be prioritized at the front of the task queue, and tasks with relatively loose delays will be prioritized at the end of the task queue. For tasks with equal remaining waiting delays in the buffer, they are sorted in order from high to low priority to ensure that tasks that are about to time out and have high priority can be processed first. Traverse the tasks in each buffer after preliminary sorting, and search for any timed-out tasks. If the timed-out task is the lowest priority task in the task queue of this section, it means that the value of the task is the lowest, and the task is directly discarded; otherwise, insert the timed-out task into each position before the timed-out position in turn, obtain the optimal insertion position, and repeat the above steps until all tasks in each buffer do not time out, and obtain the optimal task queue of each buffer that can achieve the highest priority as the observation of each intelligent agent.

7. The method according to claim 1, characterized in that The specific process of estimating the local Q value by the local partial Q network includes: The local local Q network consists of two fully connected layers and one gated recurrent unit. First, the observation of the agent is input into the first fully connected layer, and the embedded feature vector corresponding to the agent is output. Then, the embedded feature vector corresponding to the agent and the hidden vector of the previous time slot are input into the second gated recurrent unit to obtain the hidden vector of the current time slot. The hidden vector of the current time slot is input into the third fully connected layer, and the local Q values ​​corresponding to all actions that the agent may choose are output. - The greedy strategy uses the local Q value to select the action to take in the current time slot.

8. The method according to claim 1, characterized in that The specific process of the global hybrid Q network module jointly generating the global Q value includes: The global hybrid Q network module consists of two fully connected layers. First, all local Q networks are connected in time slots. The local Q value set output when is a vector , received through the first fully connected layer of the global hybrid Q network module As input, output The corresponding embedding feature vector , expressed as ； in, , and They represent the weight matrix, bias term, and activation function of the first fully connected layer of the global hybrid Q network module respectively; Will Input to the second fully connected layer of the global hybrid Q network module, and output the global Q value, expressed as ； in, , and They represent the weight matrix, bias term, and activation function of the second fully connected layer of the global hybrid Q network module, respectively. Indicates time slot The overall state of the mobile edge computing system is Indicates time slot The action collection of drones and ground intelligent devices.

9. A priority-based UAV cluster-assisted task computing and unloading device, characterized in that: The device comprises: A mobile edge computing system construction unit is used to construct a mobile edge computing system coordinated by a drone cluster. The system includes a number of drones that provide computing offload services and a number of ground intelligent devices. The drones and ground intelligent devices are collectively referred to as intelligent agents. The observation range of each intelligent agent is limited and each intelligent agent is equipped with a buffer of limited length for storing incoming tasks. The offloading problem modeling unit is used to model the UAV cluster assisted task computing offloading problem as a decentralized partially observable Markov decision process, which is described by observation, state, action and reward; The priority-based offloading decision solving unit is used to construct a priority-based HQMIX network to solve the unloading problem of UAV cluster assisted task calculation, and obtain the unloading result of UAV cluster assisted task; wherein, the HQMIX network is composed of a priority scheduling queue module, a local local Q network module and a global hybrid Q network module; wherein, the priority scheduling queue module is used to schedule and sort the newly arrived tasks in the buffer of each intelligent agent and the remaining tasks in the buffer in the previous time slot according to the delay and priority of the task, and optimize the optimal task queue as the observation of each intelligent agent; the local local Q network module contains a number of local local Q networks consistent with the number of intelligent agents, which are used to input the observation of each intelligent agent into the corresponding local local Q network for calculation, and estimate the local Q value corresponding to the action that each intelligent agent may choose; the global hybrid Q network module is used to combine the local Q values ​​output by all local local Q networks into a global Q value, and generate the ground intelligent device task offloading decision and UAV movement trajectory that can achieve the highest priority according to the global Q value.

Images