An edge computing task scheduling method based on multi-agent reinforcement learning
By creating reinforcement learning agents on mobile devices, optimizing task scheduling decisions, the problems of high load and latency in mobile edge computing are solved, and better latency, drop rate and bandwidth utilization performance are achieved.
Patent Information
- Application Number
- CN202411127783.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-16
AI Technical Summary
In mobile edge computing, when multiple terminal nodes unload tasks to one edge node at the same time, it leads to high load and delay. Traditional reinforcement learning algorithms are difficult to effectively handle large-scale discrete action space, affecting task sensitivity and network link utilization.
Create a reinforcement learning agent on each mobile device that contains E-GAT agent and scheduling decision agent. By calculating the status information and cost values of each agent, optimize task scheduling decisions, select scheduling actions that minimize delays, and balance long-term and short-term Reward.
It effectively solves the problem of large-scale discrete action space, improves the performance of delay, task drop rate and bandwidth utilization. Experimental data show that it is better than existing methods in terms of delay, task drop rate and bandwidth utilization.
Smart Images

Figure CN118656195B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an edge computing task scheduling method based on multi-agent reinforcement learning, and belongs to the technical field of edge computing. Background Art
[0002] With the rapid development of short-distance communication technology, network transmission efficiency and bandwidth levels have been significantly improved, making short-distance wireless data transmission more convenient and faster. Mobile edge computing (MEC) offloads tasks generated by terminal devices to the edge of the network for processing through short-distance wireless transmission technology, showing excellent low latency and high computing power, and is regarded as an effective solution for offloading and scheduling of computing-intensive tasks.
[0003] Although mobile edge computing has obvious advantages over cloud computing in many scenarios, since the edge layer is mostly composed of intelligent wireless base stations with limited computing power, when facing the problem of task offloading of terminal devices, if a large number of terminal nodes offload tasks to an edge node at the same time, the edge node will reach a higher load level, which will affect the time-sensitive requirements of the task to a certain extent and will lead to an increase in packet loss rate due to exceeding the deadline of the task.
[0004] In the MEC system, how to select appropriate edge servers globally for task offloading so as to achieve optimal latency is a PSPACE-hard problem. The uncertainty of task arrival and network status also poses great challenges to task offloading. In the face of this multi-objective optimization problem, traditional optimization techniques are difficult to obtain good results. Therefore, how to optimally schedule and offload tasks is the core issue of edge computing.
[0005] In the common distributed scheduling algorithm of MEC system, each terminal node will train a reinforcement learning agent, and each agent needs to make task offloading decisions for the generated tasks. Assume that each terminal node has L tasks arriving at the same time, and the edge layer node has N optional nodes for offloading. Then the action space is (N+1) L Traditional reinforcement learning algorithms, such as the value-based Deep Q-learning (DQN) reinforcement learning algorithm and the policy-based Actor-Critic (AC) architecture reinforcement learning algorithm, cannot handle the above large-scale discrete action space well. Summary of the invention
[0006] Technical problem to be solved by the invention
[0007] Aiming at the time sensitivity of tasks and network link utilization issues in mobile edge computing scenarios, the present invention proposes an edge computing task scheduling method based on multi-agent reinforcement learning.
[0008] Technical Solution
[0009] In order to achieve the above object, the technical solution provided by the present invention is:
[0010] An edge computing task scheduling method based on multi-agent reinforcement learning includes the following steps:
[0011] Step 1: Create a reinforcement learning agent on each mobile device, where the reinforcement learning agent includes an E-GAT agent and a scheduling decision agent.
[0012] Step 2: Calculate the E-GAT agent status information of mobile device m at time slice t , and scheduling decision agent status information .
[0013] Step 3: At the beginning of each time slice t, the mobile device m calculates the state information of the E-GAT agent. and scheduling decision agent status information Make scheduling decisions for tasks at this moment:
[0014] For a certain scheduling policy A, the cost values of the generated E-GAT agent and the scheduling decision agent are calculated as follows:
[0015] At the end of each time slice t, the mobile device m calculates the cost based on the task completion status returned by the environment. m :
[0016]
[0017] Where i is the task number, L is the total number of tasks, represents the state of a certain E-GAT agent or scheduling decision agent at time t calculated in step 2, , indicating whether the i-th task is scheduled at time t, = 0 means that task i is computed locally. If = 1 means that task i is scheduled to the edge node for computation. Represents task i according to state s m The delay caused by the interaction between the task scheduling decision and the environment.
[0018] Step 4: For each node m, execute Cost mMinimal scheduling action :
[0019]
[0020] Where E[ ] represents the mathematical expectation, γ is the discount factor used to balance the long-term and short-term rewards in the reinforcement learning process, 0<γ<1.
[0021] Furthermore, in step 2, the status information of the E-GAT agent of the mobile device m in time slice t Including the CPU utilization and network link utilization of the mobile device where the agent is located.
[0022] Furthermore, in step 2, the state information of the scheduling decision agent of mobile device m in time slice t include , E m (t), H m (t) ,
[0023] in:
[0024] E m (t) represents all task information generated on the mobile device at the beginning of time slice t, including the task size and the waiting time for processing after the task is generated.
[0025] H m (t) represents the amount of tasks scheduled to the edge node in the mobile device m in the period before time slice t, H m (t) is a The matrix, matrix elements Represents an edge node In from Start The load level of the edge node at a certain time is the utilization rate of the edge node hardware equipment and network bandwidth.
[0026] Furthermore, in step 3 The calculation method is:
[0027] If task i is processed before the specified processing time T, then
[0028] in represents the processing completion time of task i on node m within time slice t;
[0029] If a task is not processed within the specified processing time T, then
[0030]
[0031] Where C is a preset constant;
[0032] If the number of tasks generated on terminal device m in time slice t is less than L, then
[0033] .
[0034] Beneficial Effects
[0035] The method of the present invention automatically analyzes the complex state characteristics generated by the interaction between the terminal device and the environment through a reinforcement learning method, optimizes the scheduling action according to the state information at the beginning of each time slice, and performs the unloading processing of related tasks according to the scheduling action. It effectively solves the problem of large-scale discrete action space in mobile edge computing;
[0036] Experimental data show that the method of the present invention has better performance than existing methods in terms of latency, task discard rate, bandwidth utilization and convergence performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is a flow chart of the method of the present invention;
[0038] Figure 2 Result diagram of simulation experiment of the method of the present invention. DETAILED DESCRIPTION
[0039] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and specific implementation methods.
[0040] The method of the present invention takes into account that there are several edge nodes and mobile terminal nodes in the MEC scenario, multiple terminal nodes are deployed around an edge node, and each node in the edge layer is also connected through a high-speed network. The mobile terminal node will generate at most L tasks with probability in the time slice t. The tasks will be processed locally or uploaded to the edge layer for processing according to the selection of the terminal device.
[0041] Each mobile terminal node has several threads, and each thread corresponds to a queue. The tasks generated by the terminal device will enter the thread queue for processing according to the FIFO principle. The terminal node will process or schedule the tasks according to the selection of the reinforcement learning agent on itself, and the tasks that need to be scheduled will be further forwarded or processed after reaching the edge layer.
[0042] At the beginning of each time slice, the terminal device will have multiple tasks with a certain probability. Each task has its own information such as task size, task number, etc. The task will first enter the thread queue of the terminal device for processing. When the task exceeds the maximum deadline, it will be discarded.
[0043] For tasks that need to be scheduled, after the terminal device completes the transmission, the task will enter the pre-processing queue of the edge node directly connected to it. The edge node determines whether to process or forward the task based on the edge node number of the final scheduling destination of the task.
[0044] There are three queues on each edge node in the edge layer (each edge node maintains a preprocessing queue, a computing queue, and a transmission queue for each node directly connected to it). At the beginning of each time interval, the edge node will take the task from the preprocessing queue, and determine whether to process or forward it at this edge node based on the task information, and put it into the corresponding computing queue or transmission queue.
[0045] like Figure 1 As shown, the edge computing task scheduling method based on multi-agent reinforcement learning of the present invention includes the following steps:
[0046] 1) Create a reinforcement learning agent on each mobile device. The reinforcement learning agent consists of two sub-agents: E-GAT agent and scheduling decision agent. The E-GAT (Edge-Featured Graph Attention Network) agent is an agent that uses E-GAT to extract useful latent spatial feature information in the scene and uses it as the state of the scheduling decision agent. Its role is to capture and summarize the relationship and information of network nodes for subsequent decision-making.
[0047] 2) Calculate the E-GAT agent status information of mobile device m at time slice t , and scheduling decision agent status information ,
[0048] in Including the CPU utilization and network link utilization of the mobile device where the agent is located;
[0049] include , E m (t), H m (t)
[0050] in:
[0051] E m (t) represents all task information generated on the mobile device at the beginning of time slice t. In this embodiment, the task information includes the task size and the waiting time for processing after the task is generated;
[0052] H m(t) represents the amount of tasks scheduled to the edge node in the mobile device m in the period before time slice t, H m (t) is a The matrix, matrix element H m (t) i, j Represents an edge node In from Start The load level of a time node. In this embodiment, the load level is the utilization rate of edge node hardware devices and network bandwidth.
[0053] 3) At the beginning of each time slice t, mobile device m calculates the state of the E-GAT agent according to the state information of the E-GAT agent. and scheduling decision agent status information Make scheduling decisions for tasks at this moment:
[0054] For a certain scheduling policy A, the cost values of the generated E-GAT agent and the scheduling decision agent are calculated as follows:
[0055] At the end of each time slice t, the mobile device m calculates the cost based on the task completion status returned by the environment. m :
[0056]
[0057] Where i is the task number, L is the total number of tasks, represents the state of an E-GAT agent at time t, Indicates the status of the i-th task appearing in the m-th time slice at time t, , indicating whether the i-th task in the m-th time slice is scheduled at time t. If = 0, the task is calculated locally. If =1, then task i is scheduled to the edge node for calculation. Represents task i according to state s m The delay caused by the interaction between the task scheduling decision and the environment.
[0058] In the above formula The calculation method is:
[0059] If task i is processed before the specified processing time T, then
[0060]
[0061] in represents the processing completion time of task i on node m within time slice t,
[0062] If a task is not processed within the specified processing time T, then
[0063]
[0064] Wherein C is a preset constant. In this embodiment, C=30.
[0065] If the number of tasks generated on terminal device m in time slice t is less than L, then
[0066]
[0067] 4) For each node m, execute Cost m Minimal scheduling action :
[0068]
[0069] Where E[ ] represents the mathematical expectation, γ is the discount factor used to balance the long-term return and short-term return in the reinforcement learning process, 0<γ<1, and in this embodiment, γ= 0.9.
[0070] In the scenario considered by the present invention, multiple end nodes are connected to one edge layer node, and there is a topological connection between the edge nodes. The terminal device is connected to the edge layer node through a wireless network with a default network bandwidth of 10Mbps, and the edge layer nodes are connected to each other by using a high-speed wired network with a bandwidth of 100Mbps. In order to simplify the complexity of the simulation, the terminal devices are specified to have the same processing and transmission capabilities, and the edge layer nodes also have the same storage and transmission capabilities. By default, each terminal device is connected to all edge layer nodes.
[0071] Figure 2 (a) shows the delay time of the method of the present invention for different numbers of terminal nodes. It can be seen that compared with other existing methods, the delay of the algorithm proposed by the present invention is relatively low; Figure 2 (b) shows that for different numbers of terminal nodes, the proposed method also maintains a lower task abandonment rate than other methods; Figure 2 (c) is a simulation of the relationship between network bandwidth utilization and the number of terminal nodes. The method of the present invention maintains a higher bandwidth utilization compared with other methods. Figure 2 (d) shows the convergence performance of each algorithm as the number of training rounds increases. It can be seen intuitively that the method of the present invention achieves better reward performance than the existing method after convergence.
[0072] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.
Claims
1. A method for scheduling edge computing tasks based on multi-agent reinforcement learning, characterized in that: The steps include: Step S1, creating a reinforcement learning agent on each mobile device, wherein the reinforcement learning agent includes an E-GATagent and a scheduling decision agent; Step S2: Calculate the E-GAT agent status information of mobile device m in time slice t , and scheduling decision agent status information ; Step S3: At the beginning of each time slice t, the mobile device m calculates the state of the E-GAT agent according to the state information of the E-GAT agent. and scheduling decision agent status information Make scheduling decisions for tasks at this moment: For a certain scheduling policy A, the cost values of the generated E-GAT agent and the scheduling decision agent are calculated as follows: At the end of each time slice t, the mobile device m calculates the cost based on the task completion status returned by the environment. m : Where i is the task number, L is the total number of tasks, represents the state of a certain E-GAT agent or scheduling decision agent at time t calculated in step S2, , indicating whether the i-th task is scheduled at time t, =0 means that task i is calculated locally. = 1 means that task i is scheduled to the edge node for computation. Represents task i according to state s m The delay caused by the interaction between the task scheduling decision and the environment; Step S4: For each node m, execute Cost m Minimal scheduling action : Where E[ ] represents the mathematical expectation, γ is the discount factor used to balance the long-term and short-term returns in the reinforcement learning process, 0<γ<1, T The processing time specified for the task.
2. The edge computing task scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S2, the status information of the E-GAT agent of the mobile device m in time slice t Including the CPU utilization and network link utilization of the mobile device where the agent is located.
3. The edge computing task scheduling method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: In step S2, the status information of the scheduling decision agent of the mobile device m in time slice t include , E m (t), H m (t), in: E m (t) represents all task information generated on the mobile device at the beginning of time slice t, including task size and waiting time after task generation; H m (t) represents the amount of tasks scheduled to the edge node in the mobile device m in the period before time slice t, H m (t) is a The matrix, matrix element H m (t) i, j Represents an edge node In from Start The load level of the edge node at a certain time is the utilization rate of the edge node hardware equipment and network bandwidth.
4. The edge computing task scheduling method based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S3 The calculation method is: If task i is processed before the specified processing time T, then in represents the processing completion time of task i on node m within time slice t; If a task is not processed within the specified processing time T, then Where C is a preset constant; If the number of tasks generated on terminal device m in time slice t is less than L, then .
