Hierarchical edge intelligence task scheduling method and device based on deep reinforcement learning

Abstract

The application relates to the technical field of mobile edge computing and unmanned aerial vehicle communication, in particular to a hierarchical edge intelligent task scheduling method and device based on deep reinforcement learning, which comprises the following steps: constructing a hierarchical collaborative computing architecture composed of a back-end mobile edge computing server, an unmanned aerial vehicle and a robot; introducing a self-attention mechanism in an Actor network; designing an auxiliary branch in a Critic network in terms of a self-supervision mechanism; using self-supervision signals such as system bottleneck information and worst-case delay performance to guide policy learning; and improving feature generalization capability through state cost prediction auxiliary tasks. In the online training process, load balancing constraints and delay penalty terms are introduced in the optimization of the policy network, and the joint scheduling of local computing and edge computing resources is realized by combining a double-layer action space design. The application effectively reduces the average time delay and energy consumption of task processing in a dynamic environment, significantly suppresses system performance fluctuations, and improves the long-term operation stability and robustness in a heterogeneous edge computing environment.

Description

Technical Field

[0001] This invention relates to the fields of mobile edge computing and UAV communication technology, specifically to a hierarchical edge intelligent task scheduling method and apparatus based on deep reinforcement learning. Background Technology

[0002] In drone-assisted mobile edge server systems, robots, drones, and edge servers collaborate to efficiently complete environmental perception, task processing, and decision execution. This collaborative computing architecture, by integrating heterogeneous resources such as ground robots, local computing units, aerial relay nodes, and edge servers, significantly improves the system's computing efficiency and environmental adaptability, and has broad application prospects in scenarios such as intelligent inspection, disaster relief, and unmanned operations.

[0003] However, in real-world dynamic environments, robot-drone-mobile edge server collaborative computing systems still face key challenges such as high dynamism, high heterogeneity, and resource constraints, specifically manifested in:

[0004] (1) The problem of uneven resource allocation is prominent: In actual operation, due to the differences in robot computing power, task scale and communication conditions, the demand for computing resources among different nodes is significantly uneven. Some task-intensive robots may be in a state of resource shortage for a long time, while other nodes are underutilized. This imbalance not only reduces the overall resource utilization efficiency of the system, but may also lead to serious task queuing at some nodes, increase system latency, and cause local computing bottlenecks, affecting the stable operation of the system.

[0005] (2) Task scheduling is difficult to adapt to dynamic environmental changes: Traditional task scheduling methods are usually based on static or semi-dynamic strategies, which are difficult to adapt to the rapid changes in task load and node status in complex environments. In practical application scenarios, robot movement, sudden task occurrences, and changes in UAV position can all change the system topology and resource distribution in a short period of time. Fixed or lagging scheduling strategies are difficult to respond to these changes in a timely manner, resulting in unreasonable task allocation, thereby reducing the real-time performance and scheduling efficiency of the system.

[0006] (3) Complex Coupling Between Communication and Computation: In this collaborative system, task offloading not only depends on computing resources but is also significantly affected by the quality of the communication link. As a relay node, the communication capability of the UAV directly determines the task transmission efficiency. However, existing studies often simplify the communication model or ignore the coupling between communication and computation, failing to fully consider factors such as link bandwidth limitations, transmission delays, and data congestion. Such simplifications can lead to discrepancies between the scheduling strategy and the actual system performance in real-world environments, reducing the practical usability of the algorithm.

[0007] (4) Lack of collaborative optimization mechanism for multi-dimensional resources: Most current task scheduling methods focus on a single resource dimension, neglecting the collaborative relationship between computing, communication, and system load. In real-world systems, task processing performance is often determined by multiple resources. For example, when a large number of tasks are simultaneously offloaded to a mobile edge server, if communication link limitations are not considered, data transmission congestion may occur, preventing computing resources from being fully utilized. Furthermore, existing methods generally lack global optimization capabilities and dynamic adjustment mechanisms, making it difficult to achieve collaborative scheduling and optimal allocation of multiple resources in complex environments.

[0008] (5) Insufficient system performance stability and robustness: Due to the randomness of task arrival and the continuous changes in system state, traditional scheduling methods often only focus on average performance indicators, while ignoring worst-case latency and system performance fluctuations. This can lead to a significant decrease in system performance under high load or sudden events, affecting the reliability of task execution. In practical applications, especially in scenarios with high real-time requirements, performance fluctuations can have serious consequences, thus requiring a more stable and robust scheduling strategy.

[0009] In summary, existing robot-drone-mobile edge server collaborative computing systems still have many shortcomings in task scheduling and resource allocation. There is an urgent need for an intelligent scheduling method that can comprehensively consider multi-dimensional resources, adapt to dynamic environmental changes, and have global optimization capabilities, so as to achieve a synergistic improvement in system performance and stability. Summary of the Invention

[0010] The purpose of this invention is to provide a hierarchical edge intelligent task scheduling method and device based on deep reinforcement learning. By oriented towards a robot-UAV-mobile edge computing server collaborative computing architecture, it realizes intelligent task offloading and load balancing in a multi-layer heterogeneous system, thereby solving the technical problems of insufficient modeling capability, limited resource collaborative optimization level, and difficulty in adapting to dynamic environmental changes in existing UAV-assisted mobile edge computing systems.

[0011] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0012] The first aspect is a hierarchical edge intelligent task scheduling method based on deep reinforcement learning, which includes the following steps:

[0013] S100. The system's task quantity and task information, the number of robots and mobile edge servers, the computing power of robots and mobile edge servers, the energy storage of robots and drones are determined by the central scheduler of the drone.

[0014] S200: The robot generates computationally intensive perception tasks in complex environments. Some of these tasks are uploaded to an aerial drone via a wireless link. The drone acts as a relay node to further forward the aggregated tasks to a mobile edge server.

[0015] S300, Construct a joint system state vector that includes the robot's local load, the drone relay queue status, and the mobile edge server's computing load;

[0016] S400 adopts a scheduling algorithm based on SS-DDPG, introduces a self-attention mechanism in the Actor network to explicitly model the global dependencies between heterogeneous nodes, and introduces a self-supervised auxiliary branch in the Critic network to predict the real-time cost of the system.

[0017] S500 determines the robot's unloading decision and the proportion of tasks that the drone will allocate to each mobile edge computing server node based on the two-layer action vector output by the Actor network, thereby realizing hierarchical task scheduling.

[0018] S600 constructs a reward function that includes load balancing constraints and action smoothness penalties based on the latency and energy consumption feedback generated by task execution, and continuously updates the SS-DDPG network parameters to achieve adaptive optimization in dynamic environments.

[0019] Preferably, in step S100, the robot is used to randomly distribute and generate perception tasks within a high exploration area, and has limited mobile computing and communication capabilities; the drone is used to hover at a fixed altitude, and as an aerial relay node, it is responsible for collecting the robot's tasks and caching and forwarding them, without directly participating in the computation; the mobile edge server is used to provide high-performance computing services for the drone and process the computing tasks forwarded by the drone.

[0020] Preferably, step S200 specifically includes the following steps:

[0021] S201, Discretize the system service cycle as Each of equal length The time slots are defined, and the set of time slots is denoted as... Each robot generates a Poisson function in each time slot. The number of tasks, the set of task sizes is denoted as . ;

[0022] S202. The UAV operates at high altitude, avoiding obstacles to reach the optimal relay position, and then establishes a communication link with the mobile edge computing node. Therefore, we set the communication between the UAV and all robots to be based on line-of-sight (LoS) communication, and its channel adopts the free space path loss model;

[0023] S203. During the task unloading phase, the robot chooses to unload the task to the drone via the uplink, and then the drone schedules the task to the mobile edge computing node or executes it locally.

[0024] A value of 1 indicates that the robot will offload the computational task to the mobile edge server for execution, while a value of 0 indicates that the robot will choose to perform local computation.

[0025] Preferably, the channel described in step S202 adopts a free-space path loss model, and its channel gain... for: ;

[0026] in, Represents robots With mobile edge server k in time slot The instantaneous horizontal position, where ; Indicates the horizontal position of the drone, where ; This indicates that the drone's flight altitude is fixed; This represents the channel efficiency factor.

[0027] Preferably, the joint system state vector in step S300 This includes: the robot's task computation scale and local cumulative load, the drone's relay workload, and the computational load of each mobile edge computing server node; specifically, it is expressed as:

[0028] ;

[0029] in, and These represent the computational scale and cumulative local load of the robot task, respectively. This indicates the relay queue load of the drone. This represents the computational load of the k-th mobile edge computing server node.

[0030] Preferably, step S400 specifically includes the following steps:

[0031] S401. Introducing a self-attention mechanism into the Actor network:

[0032] The system state features are embedded and mapped. By calculating the correlation weights between the query matrix, key matrix, and value matrix, the features of each mobile edge computing server node are weighted and fused. The calculation formula is as follows:

[0033] ;

[0034] Where S is the state embedding matrix, , , d is the learnable parameter matrix, and d is the feature dimension; this mechanism enables the Actor network to dynamically learn the importance of each computing node to the current scheduling decision.

[0035] S402. Introduce a self-supervised auxiliary branch in the Critic network:

[0036] The Critic network includes a Q-value main branch and a system cost prediction auxiliary branch; the auxiliary branch minimizes the predicted real-time system cost. Compared with the actual calculated system cost The mean squared error between the two is used to regularize feature learning, thereby improving the training stability of the policy network in non-stationary environments.

[0037] Preferably, step S402 specifically includes:

[0038] The self-supervised Critic network consists of two branches:

[0039] Main branch estimation state-action value function And combine it with self-attention to better capture joint state-action features;

[0040] The auxiliary branch learns the system cost characteristics through self-supervised learning, and the prediction target is: This branch does not directly participate in policy updates, but rather serves as a regularization of the Critic representation, enhancing feature generalization and training stability.

[0041] Strategy and Value Update: The Critic network is updated by minimizing a combined loss function, which includes strategies for... Time Difference TD Error and Instantaneous System Cost Predictor The self-supervised loss; the TD objective of the Q-value branch is defined as: ;

[0042] in, It is a discount factor; It is the target Critic network;

[0043] The combined Critic loss function is:

[0044] .

[0045] Preferably, step S500 specifically includes the following steps:

[0046] S501. Since the total computational task size is different for each robot, let the task size set... Represents robots The size of each computational task, then in the time slot ,robot The total size of the generated computational task is expressed as ;

[0047] S502, Order Represents robots In the time slot The uninstallation decision, then when When the task is completed, the computation is offloaded to the drone; otherwise, the task is processed locally.

[0048] Given state The SS-DDPG agent outputs a continuous action vector. ,in Indicates a tendency to uninstall. This represents the continuous MEC selection score. Normalized MEC preferences are obtained as follows:

[0049] ;

[0050] S503. Although SS-DDPG operates in a continuous action space, task offloading is inherently discrete. To address this mismatch, the agent's output is transformed into an efficient scheduling decision through deterministic projection. First, each consecutive offloading tendency is mapped to a binary decision:

[0051] ;

[0052] in It's the uninstallation threshold. Then, if... Then the robot Assign to the MEC server with the largest normalization preference:

[0053] ;

[0054] This projection is applied only during environmental interaction and reward evaluation. During network optimization, it is applied to continuous agent outputs. and The gradient is computed, which avoids backpropagation via non-differentiable thresholds and argmax operations. In practice, exploration noise is added before projection, so the agent can still explore near the binary decision boundary, and this transformation does not compromise the stability of training.

[0055] Preferably, step S600 specifically includes the following steps:

[0056] S601, Executing scheduling actions Afterwards, the system completes task unloading, resulting in corresponding computational latency and energy consumption, thus yielding the total computational latency. and time slot Total energy consumption at the location ;

[0057] Define the normalized cost function as follows: ;

[0058] in, , This represents the normalized data. Indicates the weighting parameter;

[0059] Considering system performance and scheduling stability, an instantaneous reward function is constructed that includes load balancing constraints and action smoothness penalties: ;

[0060] in This represents the system cost, including latency and energy consumption. This represents the MEC queue load vector. Penalize load imbalances between MEC servers; this reward design ensures that the scheduling strategy maintains smooth and stable task offloading decisions in the two-level action space while minimizing computational latency and energy consumption.

[0061] S602. Update the target network via soft update:

[0062] ;

[0063] in, It is a soft update coefficient.

[0064] Secondly, this application provides a computing device, including a memory, a processor, and a computer program stored in the memory; when the processor calls and executes the computer program, it can complete each step in the above-mentioned hierarchical edge intelligent task scheduling method based on deep reinforcement learning.

[0065] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the steps of the above-described hierarchical edge intelligent task scheduling method based on deep reinforcement learning.

[0066] Fourthly, this application provides a computer program product, including a computer program, which, when loaded and executed by a processor, is used to implement the steps in the above-described hierarchical edge intelligent task scheduling method based on deep reinforcement learning.

[0067] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0068] (1) Achieving Global Collaborative Task Scheduling Optimization: Traditional methods often employ local or independent decision-making strategies, lacking global coordination capabilities and easily leading to low resource allocation efficiency. This invention constructs a unified decision-making framework based on deep reinforcement learning, achieving global optimization of task offloading in a multi-layer collaborative architecture of "robot-drone-mobile edge server". By adaptively allocating tasks among different computing nodes, the overall task processing latency of the system is effectively reduced, and the system operating efficiency is improved.

[0069] (2) Optimizing Task Processing Latency and System Energy Consumption: Existing methods typically focus only on a single performance indicator, making it difficult to balance latency and energy consumption. This invention introduces both task processing latency and energy consumption indicators into the reward function design, comprehensively modeling the local computing energy consumption, communication transmission energy consumption, and edge computing overhead during task execution to achieve multi-objective collaborative optimization. While ensuring timely task completion, it effectively reduces the overall system energy consumption and improves energy utilization efficiency.

[0070] (3) Enhancing the state modeling and decision-making capabilities of complex systems: Traditional methods are difficult to characterize the global dependencies in multi-node systems. This invention introduces a self-attention mechanism to jointly model the robot's task load, UAV scheduling status, and MEC server queue information, which can capture the correlation between different nodes in the system, thereby improving the policy network's perception capability and decision-making accuracy in complex dynamic environments.

[0071] (4) Enhancing Algorithm Stability and Robustness: To address the problem of unstable training in dynamic environments using traditional reinforcement learning methods, this invention introduces a self-supervised learning mechanism. By constructing an auxiliary optimization objective, the strategy is guided to focus on system bottlenecks and performance under extreme conditions, thereby reducing training oscillations and improving convergence stability. Even with significant fluctuations in task load, it can still maintain superior scheduling performance.

[0072] (5) Achieving collaborative optimization of multi-dimensional resources: Existing methods often only optimize a single resource, making it difficult to fully utilize the overall system performance. This invention comprehensively considers computing resources, communication resources, and energy consumption to establish a multi-dimensional resource collaborative optimization mechanism, realizing joint scheduling of resources during the task offloading decision-making process, effectively avoiding communication congestion and computing overload problems.

[0073] (6) Possesses excellent dynamic adaptability: Compared to static or rule-driven scheduling methods, this invention is based on a reinforcement learning framework and can dynamically update its strategy through continuous interaction with the environment. When task arrival rate, node load, or system structure changes, it can quickly adjust the task allocation strategy to maintain the efficient operation of the system in complex and dynamic environments.

[0074] In summary, this invention comprehensively considers task processing latency and system energy consumption, and achieves joint optimization of multi-dimensional resources such as computing and communication in a multi-layered heterogeneous collaborative computing environment. Through self-attention and self-supervision enhancement mechanisms, it realizes dynamic adaptation and global optimization of task scheduling strategies, thereby significantly improving system resource utilization efficiency and load balancing capabilities, and meeting the application requirements for low latency and low energy consumption in complex dynamic scenarios. Attached Figure Description

[0075] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0076] Figure 1 This is an application environment diagram of the hierarchical edge intelligent task scheduling method and device based on deep reinforcement learning according to the present invention.

[0077] Figure 2 This is a flowchart of the task scheduling process of the present invention;

[0078] Figure 3 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0079] Figure 4 This is a flowchart of the SS-DDPG algorithm in this invention;

[0080] Figure 5 This is a comparison chart of the total system energy consumption of the SS-DDPG algorithm in this invention with traditional DDPG, SAC, and Random algorithms under different task arrival rates;

[0081] Figure 6 This is a comparison chart of the average execution time of different algorithms in this invention under different task arrival rates;

[0082] Figure 7 This is a comparison chart of the final task completion time points for each robot under different algorithms in this invention. Detailed Implementation

[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0084] Please see Figures 1-7 The present invention provides the following technical solution:

[0085] Example 1: The hierarchical edge intelligent task scheduling device based on deep reinforcement learning provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown.

[0086] The robot 102, drone 104, and mobile edge server 106 interact with each other via a wireless communication link. The data storage system can store task information, node status information, and deep reinforcement learning model parameters generated during system operation. This data storage system can be set up independently, integrated into the mobile edge server 106, or deployed on a cloud server or other computing nodes.

[0087] Robot 102 is used to collect perception data in complex environments and generate computationally intensive tasks based on its computing power, remaining energy, and task requirements. Robot 102 can upload some computational tasks to UAV 104 via a wireless link, where UAV 104 acts as an aerial relay node to cache, aggregate, and forward the tasks. UAV 104 further distributes the received tasks to the corresponding mobile edge server 106 for processing based on the current network status and edge server load.

[0088] The robot 102 sends the collected system status information to the mobile edge server 106. This system status information includes the robot's task computation scale, local cumulative computation load, UAV relay queue status, and the computation load status of each mobile edge server. Upon receiving this status information, the mobile edge server 106 models the entire robot-UAV-MEC system as a deep reinforcement learning environment and utilizes the robot task scheduling process as an intelligent decision-making process to construct a hierarchical task scheduling model based on SS-DDPG.

[0089] During task scheduling, the mobile edge server 106 acts as the central control node. It inputs the current global state into the Actor network, extracts the correlation features between different computing nodes through a self-attention mechanism, and generates task offloading decisions and task allocation ratios. Simultaneously, the Critic network utilizes the real-time cost features of the self-supervised auxiliary branch learning system to evaluate the value of the scheduling strategy. Through continuous interaction, it obtains feedback on task execution latency, energy consumption, and load balancing status, and updates the SS-DDPG network parameters, achieving adaptive scheduling optimization of robot computing tasks in complex dynamic environments.

[0090] Example 2: Combination Figure 1-3 As shown, the hierarchical edge adaptive task scheduling method based on deep reinforcement learning includes the following specific steps:

[0091] S100. At the beginning, the central scheduling unit on the UAV side first obtains the task scale and task attribute information in the current scene (i.e., the tasks that the system did not finish processing in the previous time slice, not the tasks in the current time slice), determines the number of robot nodes participating in the calculation, the number of mobile edge servers, and collects the computing power parameters of each robot and edge server. At the same time, it obtains the remaining energy status of the robots and UAVs to provide basic data support for subsequent scheduling decisions.

[0092] In step S100, at the beginning, the central scheduler needs to determine the following information:

[0093] The S101 system consists of 10 robots, 1 drone, and 3 mobile edge computing servers. The robots are responsible for generating computationally intensive tasks related to environmental perception, collaborative decision-making, and motion control. Due to the limited energy capacity of the drone, it primarily functions as a relay node, collecting tasks unloaded by the robots and forwarding them to the more powerful servers for processing.

[0094] S102. The local CPU frequency of each robot is set in the range of [1.5, 2.0] GHz, while the CPU frequency of the mobile edge server is configured between [9.0, 11.0] GHz, which reflects the significant advantage of the edge side in computing power.

[0095] S200: The robot generates computationally intensive perception tasks in complex environments. Some tasks are uploaded to drones in the air via wireless links. The drones act as relay nodes to further forward the aggregated tasks to mobile edge servers.

[0096] Step S200 includes the following steps:

[0097] S201, We discretize the system service cycle as follows: Each of equal length The time slots are defined, and the set of time slots is denoted as... Each robot generates a Poisson function in each time slot. Number of tasks, average arrival rate per time slot One task. The task data size is... Uniformly distributed within the Mbits range. The set of task sizes is denoted as... ;

[0098] S202. The UAV operates at high altitude, able to avoid obstacles and reach the optimal relay position, then establishes a communication link with the mobile edge computing node. Therefore, we assume that the communication established between the UAV and all robots is based on line-of-sight (LoS) communication. The channel adopts a free-space path loss model;

[0099] S203, The channel adopts a free-space path loss model, and the channel gain... for:

[0100] ;

[0101] Among them, robots With mobile edge server k in time slot The instantaneous horizontal position can be expressed as ,in The horizontal position of the drone is denoted as... ,in The drone's flight altitude is fixed at ;

[0102] S204. During the task unloading phase, the robot can choose to unload the task to the drone via the uplink, and the drone will then schedule the task to a mobile edge computing node or execute it locally. A value of 1 indicates that the robot will unload the computing task to the mobile edge server for execution, while a value of 0 indicates that the robot will choose local computing.

[0103] S300: Construct a joint system state vector that includes the robot's local load, the drone relay queue status, and the mobile edge server's computing load.

[0104] In step S300, the joint system state vector st includes the robot's task computation scale and local cumulative load, the UAV's relay workload, and the computational load of each mobile edge computing server node, specifically represented as follows: ;

[0105] in, and These represent the computational scale and cumulative local load of the robot task, respectively. This indicates the relay queue load of the drone. This represents the computational load of the k-th mobile edge computing server node.

[0106] S400 employs a scheduling algorithm based on SS-DDPG, introduces a self-attention mechanism in the Actor network to explicitly model the global dependencies between heterogeneous nodes, and introduces a self-supervised auxiliary branch in the Critic network to predict the system's real-time cost.

[0107] Step S400 includes the following steps:

[0108] S401. The self-attention mechanism introduced in the Actor network specifically involves: embedding and mapping the system state features; calculating the correlation weights between the query matrix, key matrix, and value matrix; and weighted fusing the features of each mobile edge computing server node. The calculation formula is as follows:

[0109] ;

[0110] Where S is the state embedding matrix, WQ, WK, WV are learnable parameter matrices, and d is the feature dimension; this mechanism enables the Actor network to dynamically learn the importance of each computing node to the current scheduling decision.

[0111] S402, The introduction of a self-supervised auxiliary branch in the Critic network specifically involves: the Critic network comprising a Q-value main branch and a system cost prediction auxiliary branch; the auxiliary branch minimizes the predicted real-time system cost. Compared with the actual calculated system cost The mean squared error between the two sides is used to regularize feature learning, thereby improving the training stability of the policy network in non-stationary environments. The Critic network consists of two branches: the main branch estimates the state-action function. Auxiliary branch predicts instantaneous system cost The Q-value branch also incorporates self-attention to better capture joint state-action features;

[0112] The auxiliary branch learns the system cost characteristics through self-supervised learning, and the prediction target is: ;

[0113] This branch does not directly participate in policy updates, but rather serves as a regularization of the Critic representation, enhancing feature generalization and training stability.

[0114] Strategy and Value Update: The Critic network is updated by minimizing a combined loss function, which includes strategies for... Time Difference (TD) Error and Instantaneous System Cost Predictor The self-supervised loss. The TD objective of the Q-value branch is defined as:

[0115] ;

[0116] in, It is a discount factor; It is the target Critic network.

[0117] The combined Critic loss function is given by the following equation:

[0118] ;

[0119] S500 determines the robot's unloading decision and the proportion of tasks that the drone will allocate to each mobile edge computing server node based on the two-layer action vector output by the Actor network, thereby realizing hierarchical task scheduling.

[0120] Step S500 specifically includes the following:

[0121] S501. Since the total computational task size is different for each robot, we let the task size set be... Represents robots The size of each computational task within a time slot. ,robot The total size of the generated computational task is expressed as ;

[0122] S502, Order Represents robots In the time slot The uninstallation decision, then when When the task is completed, the computation is offloaded to the drone; otherwise, the task is processed locally.

[0123] Given state The SS-DDPG agent outputs a continuous action vector. ,in Indicates a tendency to uninstall. This represents the continuous MEC selection score. Normalized MEC preferences are obtained as follows:

[0124] ;

[0125] S503. Although SS-DDPG operates in a continuous action space, task offloading is inherently discrete. To address this mismatch, the agent's output is transformed into an efficient scheduling decision through deterministic projection. First, each consecutive offloading tendency is mapped to a binary decision:

[0126] ;

[0127] in It's the uninstallation threshold. Then, if... Then the robot Assign to the MEC server with the largest normalization preference:

[0128] ;

[0129] This projection is applied only during environmental interaction and reward evaluation. During network optimization, it is applied to continuous agent outputs. and The gradient is computed, which avoids backpropagation via non-differentiable thresholds and argmax operations. In practice, exploration noise is added before projection, so the agent can still explore near the binary decision boundary, and this transformation does not compromise the stability of training.

[0130] S600 constructs a reward function that includes load balancing constraints and action smoothness penalties based on the latency and energy consumption feedback generated by task execution, and continuously updates the SS-DDPG network parameters to achieve adaptive optimization in dynamic environments.

[0131] Step S600 specifically includes the following:

[0132] S601, Executing scheduling actions Afterwards, the system completes task unloading, resulting in corresponding computational latency and energy consumption, thus yielding the total computational latency. and time slot Total energy consumption at the location ;

[0133] Define the normalized cost function as follows: ;

[0134] in, , This represents the normalized data. Indicates the weighting parameter;

[0135] Further considering system performance and scheduling stability, an instantaneous reward function is constructed that includes load balancing constraints and action smoothness penalties: ;

[0136] in, This represents the system cost, including latency and energy consumption. This represents the MEC queue load vector. Penalize load imbalances between MEC servers; this reward design ensures that the scheduling strategy maintains smooth and stable task offloading decisions in the two-level action space while minimizing computational latency and energy consumption.

[0137] S602. Update the target network via soft update:

[0138] ;

[0139] in, It is a soft update coefficient.

[0140] The performance of the embodiments of the present invention is compared with that of traditional DDPG, SAC algorithm, and Random strategy:

[0141] To comprehensively evaluate the performance of a load balancing method for a robot-drone-mobile edge server collaborative computing system based on deep reinforcement learning, this study compares and analyzes it from multiple dimensions, including random task offloading strategy, traditional DDPG algorithm, and SAC algorithm, and conducts an in-depth evaluation from three aspects: energy consumption, average execution time, and the final task completion time of multiple robots.

[0142] like Figure 5 The total energy consumption comparison is shown below: the average energy consumption of different algorithms was analyzed under different task arrival rates (10, 12, 14, and 16 tasks per time slice). It can be observed that as the task arrival rate increases, the average energy consumption of all algorithms increases due to the increased computational demands. Among all methods, the SS-DDPG algorithm consistently maintains the lowest energy consumption at all task arrival rates, demonstrating its superior energy efficiency in dynamic load environments. The DDPG algorithm performs second best, slightly higher than SS-DDPG, while the SAC algorithm has higher energy consumption, and the stochastic strategy exhibits the highest energy consumption level under high task arrival rates. These results verify that the proposed SS-DDPG framework can effectively optimize energy use efficiency while ensuring stable performance.

[0143] like Figure 6 The comparison of average computation time shows that the average execution time of all methods increases with the increase in task arrival rate, reflecting the increased system load. However, the SS-DDPG algorithm consistently achieves the lowest execution time under all task arrival rate conditions. DDPG and SAC also outperform the random strategy, but their execution time is still higher than SS-DDPG. These results demonstrate that the proposed SS-DDPG-based scheduling framework can not only reduce task response latency under normal conditions but also maintain stable performance under high-load scenarios.

[0144] like Figure 7 The comparison of the final completion times of the tasks shown is important to note. Here, "execution time" refers to the end-to-end completion time of all tasks after the robot and mobile edge server have collaborated (i.e., the final completion time of the task), not just the time spent on local computation. The results show that, under the same task arrival rate, the local computation method has the longest overall completion time because all tasks must be completed on the robot. The proposed SS-DDPG algorithm, however, exhibits superior end-to-end latency performance, achieving a more stable and moderate task completion time. This advantage stems from SS-DDPG's adaptive task scheduling mechanism, which dynamically unloads and distributes tasks between the robot and edge computing nodes based on system status. This effectively reduces the computational pressure on the robot, achieves load balancing, and avoids the overall completion time delay caused by task accumulation in a single robot.

[0145] Example 3: In an exemplary embodiment, a computer device is provided that can be used to implement a hierarchical edge intelligent task scheduling method based on deep reinforcement learning. The computer device can be a server, an edge control node, or a terminal device with computing capabilities, and its internal structure is shown in the schematic diagram below. Figure 3As shown, this computer device includes a processor, memory, input / output interfaces, and a communication interface. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface is connected to the system bus via the input / output interfaces to enable data exchange between devices.

[0146] The processor is used to perform computational operations such as task scheduling, reinforcement learning model training, and policy decision-making, and generates task offloading and resource allocation decisions based on the scheduling policy output by the SS-DDPG network. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores operating system, computer programs, and task scheduling model-related data, while the internal memory provides temporary data storage space for computer program execution.

[0147] The database in the computer device stores system status information generated during hierarchical edge intelligent task scheduling, including robot task size, local computing load, UAV relay queue status, and mobile edge server computing load. It also stores experience data and network parameter information from the reinforcement learning training process. The input / output interface enables data transmission between the processor and external devices, while the communication interface facilitates communication between the robot, UAV, and mobile edge server to complete task status acquisition, task unloading, and scheduling result feedback.

[0148] When the computer program is executed by the processor, it enables the computer device to perform a hierarchical edge intelligent task scheduling method based on deep reinforcement learning. This method includes constructing a joint system state space, generating hierarchical task scheduling actions using an Actor network, performing value evaluation through a Critic network, enhancing state feature extraction capabilities using a self-attention mechanism, and optimizing the value function estimation process through self-supervised auxiliary branches, thereby achieving efficient scheduling of computing tasks in a dynamic environment.

[0149] Figure 3 The computer device structure shown is only for illustrating the part of the structure related to the technical solution of this application and does not constitute a limitation on the specific form of the computer device. In actual application, the computer device can also add or remove some functional modules as needed, or combine and adjust existing modules.

[0150] Example 4: In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, causes the processor to execute the hierarchical edge intelligent task scheduling method based on deep reinforcement learning described in the above embodiment.

[0151] Example 5: In an exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the hierarchical edge intelligent task scheduling method based on deep reinforcement learning described in the above embodiments.

[0152] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hierarchical edge intelligent task scheduling method based on deep reinforcement learning, characterized in that, Includes the following steps: S100. The system's task quantity and task information, the number of robots and mobile edge servers, the computing power of robots and mobile edge servers, the energy storage of robots and drones are determined by the central scheduler of the drone. S200: The robot generates computationally intensive perception tasks in complex environments. Some of these tasks are uploaded to an aerial drone via a wireless link. The drone acts as a relay node to further forward the aggregated tasks to a mobile edge server. S300, Construct a joint system state vector that includes the robot's local load, the drone relay queue status, and the mobile edge server's computing load; S400 adopts a scheduling algorithm based on SS-DDPG, introduces a self-attention mechanism in the Actor network to explicitly model the global dependencies between heterogeneous nodes, and introduces a self-supervised auxiliary branch in the Critic network to predict the real-time cost of the system. S500 determines the robot's unloading decision and the proportion of tasks that the drone will allocate to each mobile edge computing server node based on the two-layer action vector output by the Actor network, thereby realizing hierarchical task scheduling. S600 constructs a reward function that includes load balancing constraints and action smoothness penalties based on the latency and energy consumption feedback generated by task execution, and continuously updates the SS-DDPG network parameters to achieve adaptive optimization in dynamic environments.

2. The hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, In step S100, the robot is used to randomly distribute and generate perception tasks within a high exploration area, and has limited mobile computing and communication capabilities; the drone is used to hover at a fixed altitude, and as an aerial relay node, it is responsible for collecting the robot's tasks and caching and forwarding them, without directly participating in the computation; the mobile edge server is used to provide high-performance computing services for the drone and process the computing tasks forwarded by the drone.

3. The hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Step S200 specifically includes the following steps: S201, Discretize the system service cycle as Each of equal length The time slots are defined, and the set of time slots is denoted as... Each robot generates a Poisson function in each time slot. The number of tasks, the set of task sizes is denoted as . ; S202. The communication established between the UAV and all robots is based on line-of-sight communication, and the channel adopts the free space path loss model. The free space path loss model has a channel gain. for: ； in, Represents robots With mobile edge server k in time slot The instantaneous horizontal position, where ; Indicates the horizontal position of the drone, where ; This indicates that the drone's flight altitude is fixed; Indicates the channel efficiency factor; S203. During the task unloading phase, the robot chooses to unload the task to the drone via the uplink, and then the drone schedules the task to the mobile edge computing node or executes it locally. A value of 1 indicates that the robot will offload the computational task to the mobile edge server for execution, while a value of 0 indicates that the robot will choose to perform local computation.

4. The hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, The joint system state vector described in step S300 This includes: the robot's task computation scale and local cumulative load, the drone's relay workload, and the computational load of each mobile edge computing server node; specifically, it is expressed as: ； in, and These represent the computational scale and cumulative local load of the robot task, respectively. This indicates the relay queue load of the drone. This represents the computational load of the k-th mobile edge computing server node.

5. The hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Step S400 specifically includes the following steps: S401. Introducing a self-attention mechanism into the Actor network: The system state features are embedded and mapped. By calculating the correlation weights between the query matrix, key matrix, and value matrix, the features of each mobile edge computing server node are weighted and fused. The calculation formula is as follows: ； Where S is the state embedding matrix, , , Let d be the learnable parameter matrix, and d be the feature dimension. S402. Introduce a self-supervised auxiliary branch in the Critic network: The Critic network includes a Q-value main branch and a system cost prediction auxiliary branch; the auxiliary branch minimizes the predicted real-time system cost. Compared with the actual calculated system cost The mean squared error between them is used to regularize feature learning; The self-supervised Critic network consists of two branches: Main branch estimation state-action value function And combine it with self-attention to better capture joint state-action features; The auxiliary branch learns the system cost characteristics through self-supervised learning, and the prediction target is: ; Strategy and Value Update: The Critic network is updated by minimizing a combined loss function, which includes strategies for... Time Difference TD Error and Instantaneous System Cost Predictor The self-supervised loss; the TD objective of the Q-value branch is defined as: ; in, It is a discount factor; It is the target Critic network; The combined Critic loss function is: 。 6. The hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Step S500 specifically includes the following steps: S501, Order Task Size Set Represents robots The size of each computational task, then in the time slot ,robot The total size of the generated computational task is expressed as ; S502, Order Represents robots In the time slot The uninstallation decision, then when When the task is completed, the computation is offloaded to the drone; otherwise, the task is processed locally. Given state The SS-DDPG agent outputs a continuous action vector. ; in, Indicates a tendency to uninstall. Represents consecutive MEC selection scores; The normalized MEC preference is obtained according to the formula: ; S503. Convert agent outputs into effective scheduling decisions through deterministic projection; First, map each consecutive unloading tendency to a binary decision: ; in, It is the uninstallation threshold; Secondly, if , will the robot Assign to the MEC server with the largest normalization preference: ； This projection is only applied during environmental interaction and reward evaluation.

7. The hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Step S600 specifically includes the following steps: S601, Executing scheduling actions Afterwards, the system completes task unloading, resulting in corresponding computational latency and energy consumption, thus yielding the total computational latency. and time slot Total energy consumption at the location ; Define the normalized cost function as follows: ; in, , This represents the normalized data. Indicates the weighting parameter; Further construct an instantaneous reward function that includes load balancing constraints and action smoothness penalties: ； in, This represents the system cost, including latency and energy consumption. This represents the MEC queue load vector. Penalize load imbalances between MEC servers; S602. Update the target network via soft update: ； in, It is a soft update coefficient.

8. A task scheduling apparatus for implementing the hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in any one of claims 1-7, characterized in that, include: The system includes a robot (102), a drone (104), a mobile edge server (106), and a data storage system; wherein the robot (102), the drone (104), and the mobile edge server (106) interact with each other via a wireless communication link. The data storage system is used to store task information, node status information, and deep reinforcement learning model parameter data generated during system operation; its configuration methods include independent configuration, integration into a mobile edge server (106), or deployment in a cloud server or other computing nodes; The robot (102) is used to collect perception tasks in complex environments and generate computationally intensive tasks based on its own computing power, remaining energy and task requirements. The robot (102) uploads some of the computing tasks to the drone (104) via a wireless link. The drone (104) acts as an air relay node to cache, aggregate and forward the tasks. The drone (104) further distributes the received tasks to the corresponding mobile edge server (106) for processing based on the current network status and edge server load. The robot (102) sends the collected system status information to the mobile edge server (106). The system status information includes the robot task computing scale, local cumulative computing load, UAV relay queue status, and computing load status of each mobile edge server. After receiving the above system status information, the mobile edge server (106) models the entire robot-UAV-MEC system as a deep reinforcement learning environment and uses the robot task scheduling process as an intelligent decision-making process to construct a hierarchical task scheduling model based on SS-DDPG. During task scheduling, the mobile edge server (106) acts as a central control node. It inputs the current global state into the Actor network, extracts the correlation features between different computing nodes by introducing a self-attention mechanism, and generates task unloading decisions and task allocation ratios. At the same time, the Critic network uses the real-time cost features of the self-supervised auxiliary branch learning system to evaluate the value of the scheduling strategy. By continuously interacting to obtain feedback on task execution latency, energy consumption, and load balancing status, and updating the SS-DDPG network parameters, adaptive scheduling optimization of robot computing tasks in complex dynamic environments is achieved.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory; when the processor calls and executes the computer program, it is able to complete the steps of the hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when run on a processor, causes the processor to execute the steps of the hierarchical edge intelligent task scheduling method based on deep reinforcement learning as described in any one of claims 1-7.

Images