Task scheduling methods based on Actor-Critic deep reinforcement learning in mobile edge computing

Abstract

This invention provides a task scheduling method based on Actor-Critic deep reinforcement learning for mobile edge computing. First, the task scheduling problem in mobile edge computing is formally defined. Second, a task scheduling method combining deep reinforcement learning and a masking mechanism is proposed. Finally, the proposed system environment and scheduling method are implemented, and extensive experiments are conducted to demonstrate the effectiveness of the method. Experimental results show that the method can cope with the dynamic and ever-changing MEC environment, approximate the optimal task scheduling strategy, and effectively improve service quality.

Description

Technical Field

[0001] This invention belongs to the fields of Internet of Things and mobile edge computing technology, specifically relating to a task scheduling method based on Actor-Critic deep reinforcement learning in mobile edge computing. Background Technology

[0002] With the rapid development of the Internet of Things (IoT), various smart devices are constantly emerging and integrating into all aspects of people's lives. According to an IDC report, it is estimated that by 2025, more than 41.6 billion smart devices will be deployed globally, providing users with various smart applications such as autonomous driving, virtual reality, and smart homes. Typically, these emerging smart applications have high requirements for device performance and generate large amounts of data. When facing these smart applications, the traditional centralized cloud computing model exposes problems such as network congestion, high latency, and high overhead, making it difficult to meet their high performance and real-time requirements. To alleviate this problem, Mobile Edge Computing (MEC) deploys computing and storage resources at the network edge, allowing users to offload computing tasks from smart applications to MEC nodes equipped with servers for processing. Compared to cloud computing environments, task scheduling in MEC environments faces the following challenges:

[0003] (1) The load conditions in the MEC environment are usually closely related to the user's work and lifestyle, which results in uneven load distribution of different servers in the MEC node at different times;

[0004] (2) MEC systems are typically designed for real-time or interactive scenarios, which requires the system to respond to user requests as quickly as possible and maintain a short response time even under high load.

[0005] (3) MEC nodes have limited resources. Compared with cloud data centers, more reasonable task scheduling is required to make efficient use of limited resources.

[0006] The aforementioned factors pose significant challenges to task scheduling. Fixed patterns or unreasonable task scheduling strategies not only severely degrade Quality of Service (QoS) but also incur excessive system overhead. Therefore, when facing dynamic and ever-changing MEC environments, it is necessary to comprehensively consider uncertainties such as system state and user needs to design an adaptive and efficient task scheduling strategy. While most existing works focus on the joint optimization of task scheduling and computation offloading, they emphasize computation offloading. After offloading user tasks to MEC nodes, resources are typically allocated evenly to the offloaded tasks or some rule-based resource allocation strategies are used. These works do not comprehensively consider the differences in task requirements and the availability of MEC resources to formulate a more reasonable task scheduling strategy, which may lead to underutilization of MEC resources or a decline in QoS. Although a few works attempt to solve the task scheduling problem in MEC environments from a more granular perspective, most of them adopt heuristic methods. These methods usually rely on prior system knowledge and show good performance in task scheduling problems under specific environments, but they reveal the disadvantage of low efficiency and cannot adapt to the complex task scheduling problems in dynamic and ever-changing MEC environments. At the same time, when faced with the task scheduling problem in such a dynamic and ever-changing MEC environment, it is difficult to accurately model it using traditional mathematical models.

[0007] As an emerging and important branch of machine learning, reinforcement learning (RL) has been applied to resource optimization problems in cloud and MEC environments. RL agents, by interacting with the unknown environment and making decisions with the goal of maximizing long-term rewards, can be seen as a potentially feasible solution to optimizing task scheduling problems in dynamic and ever-changing MEC environments. In MEC environments, the limited resources of individual edge nodes can lead to some user tasks failing to complete on time. To alleviate this problem, the agent needs to formulate appropriate task scheduling strategies to better meet the resource requirements for task processing. By providing negative feedback to operations that do not meet resource constraints, RL-based methods can learn effective task scheduling strategies. However, in dynamic MEC environments, user tasks may continuously arrive, and the proportion of actions that do not meet resource constraints is relatively large. This leads to frequent triggering of illegal actions and low training efficiency for the RL agent in the early stages of training. Summary of the Invention

[0008] In recent years, with the rapid development of 5G communication technology, various emerging intelligent applications (such as facial recognition, AR / VR, and autonomous driving) have emerged. These intelligent applications exhibit characteristics such as computational intensity and latency sensitivity, while the limited computing power of mobile devices restricts their further development and popularization. To alleviate this problem, this invention proposes a task scheduling method based on Actor-Critic deep reinforcement learning in mobile edge computing. First, the task scheduling problem in mobile edge computing is formally defined. Second, a task scheduling method combining deep reinforcement learning and masking mechanisms is proposed. Finally, the proposed system environment and scheduling method are implemented, and extensive experiments are conducted to demonstrate the effectiveness of the method. Experimental results show that the method of this invention can cope with the dynamic and ever-changing MEC environment, approximate the optimal task scheduling strategy, and effectively improve the quality of service.

[0009] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0010] A task scheduling method based on Actor-Critic deep reinforcement learning for mobile edge computing is proposed. For task scheduling models in MEC environments, a state space, action space, and reward function are defined, and the optimization problem is formalized as a Markov decision process. An adaptive task scheduling method based on proximal policy optimization is adopted. Based on the Actor-Critic DRL framework, a combination of constraint policy update magnitude and masking mechanism is used to schedule user tasks and prevent agents from making actions that violate system constraints.

[0011] Furthermore, in the task scheduling model under the MEC environment, the user's smart device connects to the base station and offloads tasks to the MEC node. Tasks enter the task queue one by one according to the offloading time, and the server equipped in the MEC node provides computing services.

[0012] The servers equipped on a MEC node are denoted as set V = {v1, v2, ..., v...} n The amount of computing and memory resources provided by each server is denoted as v. i ={r i cpu ,r i mem The tasks unloaded from the smart device to the MEC node are defined as a task queue, denoted as: J = {job1, job2, job3, ..., job} m}, where m is the number of tasks; each task is defined as: job i ={t i ,r i ,l i ,e i}, where t iThe time r represents the time when a task arrives in the task queue. i Indicates the resources required for the task, l i Indicates the maximum tolerable delay for the task, e i This indicates the task execution time; the scheduler allocates tasks to the appropriate servers based on the resource requests of different tasks in the current MEC system and the resource usage of the servers; at the same time, each server records its resource usage at each time step;

[0013] The task scheduling process is as follows: When a user-unloaded task arrives at the MEC node, the system attempts to add the task to the task queue to wait for execution. Due to limited MEC resources, the length of the task queue in the system is also limited. If the task queue is full, no new tasks can be added. In this case, the task will be judged as failing to execute. If the waiting time of the task exceeds its maximum tolerable delay, it will also cause the task to fail to execute. At time t, if the scheduler schedules the job... i Assigned to the MEC server, the server will allocate available resources to the task to meet its resource requirements; in t+e i At any time, when job i Once execution is complete, the MEC node will send the execution result back to the user and record the response time of the task. i In addition, the scheduler needs to determine whether each task in the queue meets the scheduling conditions. It selects tasks from seqLen tasks to be scheduled and schedules them to servers in the MEC nodes. Therefore, the computational complexity is (n+1). seqLen To reduce computational complexity and enable the scheduler to efficiently formulate task scheduling strategies, the server is treated as a resource pool, and resource fragmentation between servers during task scheduling is ignored. At any given time, the scheduler allows multiple scheduling operations, selecting one task from the list of tasks to be scheduled each time, until a waiting signal is sent to the server.

[0014] Furthermore, the optimization objective of the task scheduling model in the MEC environment is to reduce task response time and increase task completion rate; on the one hand, the average relative waiting time of completed tasks is used to indirectly evaluate the task response time, which is defined as:

[0015]

[0016] Among them, J c This indicates that the task sequence has been completed;

[0017] On the other hand, the task failure rate is used to indirectly assess the task completion rate, which is defined as:

[0018]

[0019] The optimization objective obtained is:

[0020] min(w T T+w F F)

[0021] Among them, w T and w F These represent the weights of different optimization metrics.

[0022] Furthermore, the scheduler is viewed as a DRL agent, and the MEC system is viewed as the environment; in each time slot, the DRL agent, according to the task scheduling policy π, operates within the current MEC system state s. t The following task scheduling action a is to be executed. t Subsequently, the environment provides a feedback reward signal r. t And enter the next state s t+1 The state space, action space, and reward function are defined as follows:

[0023] State space: The state space S represents the set of all possible states observed by the DRL agent; in time slot t, s t ∈S contains the current task queue status and available resource status in the system, and is defined as follows:

[0024] s t =(c t ,m t job 1′ job 2′ ,...,job n′ )

[0025] Among them, c t and m t These represent the available computing and memory resources in the resource pool, respectively, and the job. i′ This indicates a task that is waiting to be executed in the task queue;

[0026] Action space: In time slot t, action a t This refers to selecting a task from the task queue for scheduling, termination, or issuing a waiting signal, and is defined as follows:

[0027] A={a t |a t ∈{-1,0,1,...,seqLen}}

[0028] Where, when a t When a = -1, the DRL agent terminates execution time exceeding the maximum tolerable latency task; when a tWhen the value is 0, the DRL agent does not perform task scheduling, and unscheduled tasks wait in the queue. In other cases, the DRL agent selects a task from the task queue and allocates resources to it. The environment calculates and provides an immediate reward for the current action based on the reward function. When performing a scheduling operation, the scheduler allocates a virtual machine with the corresponding resources from the resource pool based on the resources required by the task. When performing a waiting operation, if there is a completed task, the resources occupied by the virtual machine will be released. When transitioning to the next system state, the available resources in the resource pool or the number of tasks in the queue may increase. If the available resources in the system are insufficient to complete a task with minimum resource requirements, a waiting operation can only be performed until a task is completed and resources are released.

[0029] Reward function: When an action is performed in time slot t, the environment provides a corresponding immediate reward; based on the optimization objective, the reward function is defined as:

[0030]

[0031] Where, loss t w represents the number of task execution failures. loss and w response Let n and l represent the weights of task failure rate and relative task response time, respectively. To avoid slow training and tortuous gradient descent caused by numerical differences, n and l are... a t is used for normalization to improve the convergence speed of the method; it should be noted that in order to maximize the cumulative reward, the DRL agent needs to comprehensively consider the immediate reward and the future reward to select the appropriate action.

[0032] Furthermore, in the aforementioned adaptive task scheduling method based on near-end strategy optimization:

[0033] First, initialize the Actor network parameters θ, the Critic network parameters μ, the number of algorithm iterations P, the number of time slots T, and the number of training rounds E. In each iteration round, obtain the initial state and initialize the experience pool. In each time slot t, obtain the state s through a masking mechanism. t The set of valid actions, i.e., the set of actions that satisfy the resource constraints; then, the state s t Input the Actor network and calculate the logical value for each action; then, based on the set of valid actions, use the mask and softmax functions respectively to set the probability of illegal actions to 0 and output the probability distribution of valid actions; the mask function of the masking mechanism is defined as:

[0034]

[0035] Where c is a constant, and when c is sufficiently small, softmax outputs the illegal action π. iThe probability of obtaining an action is 0; after obtaining the probability distribution of effective actions, random samples are then taken from the distribution and the corresponding actions are performed in the environment; subsequently, the immediate reward r is calculated according to the formula of the reward function. t And obtain the next state s t+1 Simultaneously, the state transition process of the current time slot is stored in the experience replay pool m as a training sample. The temporal difference residual generated in each time slot is calculated and used for subsequent optimization of the Critic network, which is defined as:

[0036]

[0037] Where r(s) t ,a t ) represents the instant reward, and γ represents the discount factor. For strategy π θ In state s t The expected reward that can be obtained at that time;

[0038] Fitting s using an Actor network t and a t The mapping relationship between these elements is used to explore the optimal Actor network, ensuring that an appropriate action is chosen in each state to maximize cumulative reward. To achieve this goal, the old policy is first temporarily stored in π. k Next, in each training epoch, the loss function of the Actor network is calculated, defined as:

[0039] L(s,a,θ k ,θ)=min(R(θ k Res t ,clip(R(θ k ),1-λ,1+λ)Res t )

[0040] in,

[0041]

[0042] The clip function is used to constrain R(θ). k This avoids fluctuations or policy mutations in the learning process caused by excessive differences between the old and new strategies, making the training process more stable. λ is used to control the shear interval (i.e., [1-λ, 1+λ]).

[0043] Next, the Actor network parameters are updated using the gradient ascent method. The policy gradient is calculated as follows:

[0044]

[0045] To accurately calculate the Q value corresponding to each state, a Critic network is used to fit Q(s).t First, use the Actor network to output the state s. t+1 Below Next, the Critic network loss is calculated; then, the Critic network loss is minimized using gradient descent and subsequently updated.

[0046] The task scheduling strategy is updated iteratively. In each iteration, T training samples are sampled according to the current task scheduling strategy, and the parameters of the Actor network and Critic network are continuously updated using the gradient update method. Finally, the training continues until convergence and the optimal task scheduling strategy is output.

[0047] Based on the above design, the basic working principle of the strategy provided by this invention can be obtained as follows:

[0048] (1) The agent makes decisions based on the user's task request and the load status of the MEC server, scheduling the task to a certain MEC server for execution, or letting the task be in a waiting state.

[0049] (2) The MEC server receives the decision of the agent, allocates the resources required for the task to be executed and starts to execute the corresponding task. When the task ends, it returns the execution result to the user and releases the resources occupied by the task.

[0050] (3) During the task scheduling process, the status of each time slot, the actions taken, the rewards obtained, and the new status transitioned to are recorded. Based on the above information, the intelligent system generates the corresponding task scheduling strategy.

[0051] Compared to existing technologies, this invention and its preferred scheme first formalize task scheduling in the MEC system into an MDP and define the state space, action space, and reward function in the DRL problem accordingly. Next, an Adaptive Task Scheduling (TSAC) method based on proximal policy optimization is designed, improving the stability and performance of the method by constraining the policy update magnitude and designing a masking mechanism. Finally, the feasibility and effectiveness of the proposed TSAC method are verified through extensive simulation experiments using a real-world dataset from a Google cluster. Experimental results show that, compared to four other benchmark methods, the TSAC method achieves lower task response time and higher task completion rate under different system loads. Furthermore, compared to advanced deep reinforcement learning methods such as Actor-Critic and DQN, the TSAC method exhibits faster and more stable convergence. Attached Figure Description

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0053] Figure 1This is a schematic diagram of a task scheduling model for an MEC system according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the task scheduling process according to an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram providing an overview of the TSAC method according to an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram comparing the convergence of different parameters in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram comparing the convergence of different methods in embodiments of the present invention;

[0058] Figure 6 This is a schematic diagram illustrating the impact of different system loads on the performance of various methods in embodiments of the present invention, wherein:

[0059] (a) The impact of different system loads on task failure rate;

[0060] (b) The impact of different system loads on average relative waiting time;

[0061] Figure 7 This is a schematic diagram illustrating the effectiveness evaluation of the masking mechanism in an embodiment of the present invention. Detailed Implementation

[0062] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0063] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0064] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0065] To address the problems of existing technologies, this invention proposes a task scheduling method (TSAC) based on Actor-Critic deep reinforcement learning (DRL) in an MEC environment. For the task scheduling model in the MEC environment, a state space, action space, and reward function are defined, and the optimization problem is formalized as a Markov Decision Process (MDP). Then, an adaptive task scheduling method based on proximal policy optimization is designed. Based on the Actor-Critic DRL framework, the proposed TSAC method, combined with a masking mechanism, can reasonably schedule user tasks and prevent the agent from making actions that violate system constraints.

[0066] like Figure 1 As shown, this invention designs a task scheduling model for MEC systems. This model consists of one MEC node and several intelligent devices. The MEC node is equipped with one base station (BS), one task queue, one scheduler, and n servers. User intelligent devices access the BS via 5G or Wi-Fi and offload tasks to the MEC node. These tasks are sequentially entered into the task queue according to their offloading time, and the servers on the MEC node provide computing services. The symbols and their definitions involved in this invention are shown in Table 1.

[0067] Table 1. Symbols and their definitions

[0068]

[0069]

[0070] The servers equipped on a MEC node are denoted as set V = {v1, v2, ..., v...} n The amount of computing (i.e., CPU) and memory resources provided by each server is denoted as v. i ={r i cpu ,r i mem Tasks unloaded from smart devices to MEC nodes are defined as a task queue, denoted as: J = {job1, job2, job3, ..., job...} m}, where m is the number of tasks. Each task is defined as: job i ={t i ,r i ,l i,e i}, where t i The time r represents the time when a task arrives in the task queue. i Indicates the resources required for the task, l i Indicates the maximum tolerable delay for the task, e i This indicates the task execution time. The scheduler allocates tasks to appropriate servers based on the resource requests of different tasks in the current MEC system and the server's resource usage. Simultaneously, each server records its resource usage at each time step.

[0071] The general process of task scheduling is as follows: Figure 2 As shown. When a user-unloaded task arrives at the MEC node, the system first attempts to add the task to the task queue to await execution. Due to limited MEC resources, the task queue length is also limited. If the task queue is full, no new tasks can be added. In this case, the task will be deemed to have failed. If the task's waiting time exceeds its maximum tolerable delay, it will also lead to task execution failure. At time t, if the scheduler assigns the job... i Assigned to the MEC server, the server will allocate available resources to the task to meet its resource requirements. (t+e) i At any time, when job i Once execution is complete, the MEC node will send the execution result back to the user and record the response time of the task. i Furthermore, the scheduler needs to determine whether each task in the queue meets the scheduling conditions. It selects tasks from the seqLen tasks to schedule to the servers in the MEC nodes, hence the computational complexity is (n+1). seqLen To reduce computational complexity and enable the scheduler to efficiently formulate task scheduling strategies, this invention treats the server as a resource pool, disregarding resource fragmentation that occurs between servers during task scheduling. At any given time, the scheduler can perform multiple scheduling operations, selecting one task from the pool of tasks to be scheduled each time, until a wait signal is sent to the server.

[0072] Based on the proposed model, the optimization objective of this invention is to reduce task response time and increase task completion rate. On one hand, this invention indirectly evaluates task response time using the average relative waiting time of completed tasks, defined as:

[0073]

[0074] Among them, J c This indicates that the task sequence has been completed.

[0075] On the other hand, this invention uses the task failure rate to indirectly evaluate the task completion rate, which is defined as:

[0076]

[0077] Therefore, the optimization objective of this invention is:

[0078] min(w T T+w F F) (3)

[0079] Among them, w T and w F These represent the weights of different optimization metrics.

[0080] Based on the proposed system model and optimization objective, this invention proposes a novel Task Scheduling with Actor-Critic deep reinforcement learning (TSAC) method, aiming to reduce task response time and improve task completion rate. Figure 3 As shown, the proposed TSAC method treats the scheduler as a DRL agent and the MEC system as the environment. In each time slot, the DRL agent, according to the task scheduling policy π, operates within the current MEC system state s. t The following task scheduling action a is to be executed. t Subsequently, the environment will provide a feedback reward signal r. t And enter the next state s t+1 This process can be viewed as an MDP. Accordingly, the state space, action space, and reward function are defined as follows:

[0081] State space: The state space S represents the set of all possible states that the DRL agent can observe. In time slot t, s t ∈S contains the current task queue status and available resource status in the system, and is defined as follows:

[0082] s t =(c t ,m t job 1′ job 2′ ,...,job n′ (4)

[0083] Among them, c t and m t These represent the available computing and memory resources in the resource pool, respectively, and the job. i′ This indicates a task that is waiting to be executed in the task queue.

[0084] Action space: In time slot t, action a t This refers to selecting a task from the task queue for scheduling, termination, or issuing a waiting signal, and is defined as follows:

[0085] A={a t |a t ∈{-1,0,1,...,seqLen}} (5)

[0086] Where, when a t When a = -1, the DRL agent terminates execution time exceeding the maximum tolerable latency task; when a t When the value is 0, the DRL agent does not perform task scheduling; unscheduled tasks wait in the queue. In other cases, the DRL agent selects a task from the task queue and allocates resources to it. The environment calculates and provides an immediate reward for the current action based on the reward function. During scheduling, the scheduler allocates a virtual machine with the required resources from the resource pool. When waiting, if a completed task exists, the resources occupied by the virtual machine are released. Upon transitioning to the next system state, the available resources in the resource pool or the number of tasks in the queue may increase. If the available resources in the system are insufficient to complete a task with minimum resource requirements, a waiting operation is necessary until a task completes and releases resources.

[0087] Reward function: When an action is performed in time slot t, the environment provides a corresponding immediate reward. According to the optimization objective of this invention, the reward function is defined as:

[0088]

[0089] Where, loss t w represents the number of task execution failures. loss and w response These represent the weights of the task execution failure rate and the relative response time, respectively. To avoid slow training and tortuous gradient descent caused by numerical differences, n and l are... a `t` is used for normalization to improve the convergence speed of the method. It's important to note that to maximize cumulative reward, the DRL agent needs to consider both immediate and future rewards when selecting appropriate actions.

[0090] Based on the above definition, the proposed TSAC method will solve the complex task scheduling problem in a dynamic and ever-changing MEC environment. Its key steps are shown in Algorithm 1.

[0091] The TSAC method, based on the Actor-Critic proximal policy optimization framework, combines value-based RL and policy-based RL, enabling the scheduler to accurately respond to environmental states and make optimal task scheduling decisions. On one hand, through Temporal-Difference Learning (TD learning), value-based RL obtains the expected reward for the current state. On the other hand, policy-based RL parameterizes the policy and, guided by value-based RL, updates the network parameters through policy gradient optimization, thereby obtaining the optimal policy.

[0092] First, initialize the Actor network parameters θ, the Critic network parameters μ, the number of algorithm iterations P, the number of time slots T, and the number of training rounds E (lines 1-2). In each iteration round, obtain the initial state and initialize the experience pool (lines 4-5). In each time slot t, obtain the state s using the masking mechanism designed in this invention. t The set of valid actions (line 7) is the set of actions that satisfy the resource constraints. This masking mechanism effectively prevents the DRL agent from executing actions that violate system constraints and improves the training speed of the algorithm. Next, the state s t Input the Actor network and compute the logistic value for each action (line 8). Then, based on the set of valid actions, use the mask and softmax functions to set the probability of illegal actions to 0 and output the probability distribution of valid actions (line 9). The mask function is defined as:

[0093]

[0094] Where c is a constant, and when c is sufficiently small, softmax outputs the illegal action π. i The probability is 0. After obtaining the probability distribution of valid actions, random samples are then taken from it (line 10) and the corresponding actions are performed in the environment. Subsequently, the immediate reward r is calculated according to Formula 6. t And obtain the next state s t+1 (Line 11). Simultaneously, the state transition process of the current time slot is stored in the experience replay pool m as training samples. The temporal difference residual generated in each time slot is calculated (Line 14) for subsequent optimization of the Critic network, defined as:

[0095]

[0096] Where r(s) t ,a t ) represents the instant reward, and γ represents the discount factor. For strategy π θ In state s t The expected reward that can be obtained at that time.

[0097] In the TSAC method, the Actor network is used to fit s t and a t The mapping relationship between these elements aims to explore the optimal Actor network, ensuring that appropriate actions are chosen in each state to maximize cumulative rewards. To achieve this goal, the TSAC method first temporarily stores the old policy in π. k (Line 15) Next, in each training epoch, the loss function of the Actor network is calculated (Line 17), defined as:

[0098] L(s,a,θ k ,θ)=min(R(θ k Res t ,clip(R(θ k ),1-λ,1+λ)Res t (9)

[0099] in,

[0100]

[0101] The clip function is used to constrain R(θ). k This avoids fluctuations or policy mutations in the learning process caused by excessive differences between the old and new strategies, making the training process more stable. λ is used to control the shear interval (i.e., [1-λ, 1+λ]).

[0102] Next, the TSAC method updates the Actor network parameters using the gradient ascent method (line 18). The policy gradient is calculated as follows:

[0103]

[0104] According to formula (7), the illegal action π is output. i The probability of is 0, therefore the corresponding gradient value is also 0. This verifies that the proposed masking mechanism can effectively promote the DRL agent to learn effective actions and avoid illegal actions. Next, in order to accurately calculate the Q value corresponding to each state, the TSAC method uses a Critic network to fit Q(s) t Specifically, the Actor network is first used to output the state s. t+1 Below (Line 19) Next, calculate the Critic network loss (Line 20); then, minimize the Critic network loss using gradient descent and update it (Line 21).

[0105] The TSAC method updates the task scheduling policy through an iterative loop. During each iteration (line 23), T training samples are sampled based on the current task scheduling policy (lines 6-13), and then the parameters of the Actor and Critic networks are continuously updated using a gradient update method (lines 16-22). Finally, the TSAC method is trained until it converges and outputs the optimal task scheduling policy.

[0106]

[0107]

[0108] Method Evaluation

[0109] The experiment was conducted on a workstation equipped with an Intel(R) Xeon(R) Silver 4208 CPU with a clock frequency of 2.10 GHz and 32 GB of memory. A real-world dataset from a Google cluster was used to verify the effectiveness of the proposed TSAC method. This dataset records the resource usage of different tasks on 125,000 servers in a Google cluster over a month. In the simulation, 10 servers were randomly selected as MEC nodes, and the resource requirements, execution time, and maximum response time of tasks on the MEC nodes were randomly sampled from this dataset. In the experiment, the number of tasks arriving at the MEC nodes in each time slot followed a Poisson distribution (Poisson(τ)), where τ was set to 1.6 by default, and the total number of time slots was 1000. Based on Python 3.9, the TSAC method uses the open-source machine learning framework PyTorch and the open-source reinforcement learning environment framework Gym to build the neural network, and trains the neural network using the Adam optimizer. The learning rate for both the Actor and Critic networks was 0.0003, and the parameter used to control the shear interval was 0.2.

[0110] Based on the above experimental setup and dataset, this invention compares the proposed TSAC method with the following five benchmark methods:

[0111] (1) RANDOM: The scheduler randomly selects a task from the task queue each time and places it in the MEC node for execution;

[0112] (2) FCFS: The scheduler selects the earliest arriving task from the task queue each time and places it in the MEC node for execution;

[0113] (3) GREEDY: The scheduler selects the task with the highest immediate reward from the task queue and places it in the MEC node for execution each time;

[0114] (4) DQN: A reinforcement learning method based on DQN is used for task scheduling;

[0115] (5) Actor-Critic: A reinforcement learning method based on asynchronous Actor-Critic is used for task scheduling.

[0116] First, this invention investigates the convergence of the TSAC method under different learning rate (LR) and clip range (CL) parameters. To make the comparison results more obvious, a system load τ = 1.8 was selected. Figure 4 As shown, when the LR value is too large, the method converges quickly, but the convergence curve does not oscillate. This may be because the Adam optimizer has an adaptive learning rate adjustment mechanism. Compared to the LR parameter, the TSAC method is more sensitive to the CL parameter. The smaller the CR value, the stronger the constraint on policy updates, and the more stable the convergence, but the slower the convergence speed.

[0117] Next, the convergence of different methods was compared. For example... Figure 5As shown, FCFS, as a single-step decision-making method, does not change its performance with the increase of training epochs because it lacks a learning process. The performance of FCFS is far inferior to the other four methods because it only considers scheduling the earliest arriving task in the queue to the MEC server, leading to a large number of tasks failing due to exceeding their maximum tolerable latency. The GREEDY method is deterministic, always selecting the optimal task for scheduling, thus its reward is higher than the RANDOM method. The three DRL-based methods (TSAC, DQN, and Actor-Critic) all outperform the GREEDY method, indicating that DRL-based methods can effectively handle task scheduling problems in MEC environments. DQN, as an offline learning method, samples a large amount of data before training the network, improving efficiency in the early stages of training. However, as the number of training epochs increases, DQN is prone to repeatedly sampling similar data, leading to overfitting and a decrease in performance in the later stages of training. In contrast, Actor-Critic, as an online learning method, requires resampling data for each training session, resulting in a smaller sample size in the early stages and slower training speed. However, the performance of the Actor-Critic method gradually increases with the number of training rounds. Compared to DQN and the Actor-Critic method, the proposed TSAC method exhibits more stable convergence and achieves higher rewards during training. This is because the TSAC method can effectively constrain policy updates during training, avoiding fluctuations or policy mutations caused by excessive differences between old and new policy updates, thus improving the stability of the training process. Simultaneously, through the designed masking mechanism, the TSAC method can avoid actions that violate system constraints during training, enabling it to efficiently approximate the optimal task scheduling policy and thus obtain higher rewards.

[0118] Next, this invention evaluates the impact of different system loads on the performance of various methods, wherein the system load follows a Poisson distribution. For example... Figure 6As shown, the TSAC method achieves superior performance in terms of task failure rate and average relative latency. With increasing system load, the task failure rate of all methods gradually increases. This is because the number of tasks arriving at the MEC node per unit time increases with the load, causing many tasks to fail due to not meeting their maximum tolerable latency or failing to enter the task waiting queue. With increasing system load, the rate of increase in average relative latency for all methods gradually slows down. This is because the resource utilization of the MEC node tends to saturate, and receiving more tasks per unit time can no longer further improve resource utilization. On the contrary, this situation leads to more task failures. Compared to the other five methods, the TSAC method achieves a lower average relative latency, thanks to its effective optimization of the task scheduling strategy, which improves task completion rate while minimizing task waiting and execution time.

[0119] Finally, ablation experiments were conducted on the TSAC method to evaluate the effectiveness of the designed masking mechanism in improving task scheduling performance. Specifically, when the DRL agent outputs an action that violates the constraints, the following three operations will be performed:

[0120] (1) The environment does not execute the action that violates the constraint, but only provides an immediate reward of -0.1 as a slight punishment for the behavior. (i.e., reward = -0.1);

[0121] (2) The environment does not execute the action that violates the constraint, but only provides an immediate reward of -1 as a greater punishment for the behavior.

[0122] (i.e., reward = -1);

[0123] (3) The environment treats actions that violate constraints as waiting actions (i.e., action = 0);

[0124] In addition, to ensure that the DRL agent learns the constraints within a limited number of attempts, a maximum number of actions is specified, that is, the agent can output a maximum of 3000 actions.

[0125] like Figure 7As shown, experimental results indicate that the designed masking mechanism has a significant impact on the performance of the TSAC method. It should be noted that the curve with the highest total reward (i.e., reward = -0.1) does not represent the optimal performance. This is because the agent consistently outputs actions that violate constraints, totaling 3000 times, with each instance receiving an immediate reward of -0.1 from the environment, resulting in a total reward of -300. However, in actual execution, the environment does not execute actions that violate constraints, meaning the environment will remain in its initial state and cannot effectively schedule tasks. When the immediate reward value is set to -1, the DRL agent tends to output valid actions. However, since valid actions are sparse in most cases, the DRL agent struggles to explore valid actions, leading to the worst convergence speed and performance. When the environment treats actions that violate constraints as waiting signals, the DRL agent can handle the task scheduling problem to some extent, but its convergence speed and performance are inferior to the TSAC method using the masking mechanism. Those skilled in the art should understand that the embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0130] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of task scheduling methods based on Actor-Critic deep reinforcement learning in mobile edge computing based on the inspiration of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A task scheduling method based on Actor-Critic deep reinforcement learning in mobile edge computing, characterized in that: For the task scheduling model in the MEC environment, we define the state space, action space and reward function, formalize the optimization problem into a Markov decision process, and adopt an adaptive task scheduling method based on proximal policy optimization. Based on the Actor-Critic DRL framework, the constraint policy update magnitude is combined with a masking mechanism to schedule user tasks and prevent agents from making actions that violate system constraints. The scheduler is viewed as a DRL agent, and the MEC system as the environment; in each time slot, the DRL agent performs tasks according to the task scheduling policy. In the current MEC system state Execute task scheduling actions ; Subsequently, the environment provides feedback and reward signals. And proceed to the next state. ; In the aforementioned adaptive task scheduling method based on near-end strategy optimization: First, initialize the Actor network parameters. Critic network parameters Number of algorithm iterations Number of time slots and number of training rounds In each iteration round, obtain the initial state and initialize the experience replay pool; in each time slot The state is obtained through a masking mechanism. The set of valid actions, that is, the set of actions that satisfy the resource constraints; then, the state... Input the Actor network and calculate the logical value for each action; then, based on the set of valid actions, use... and The function sets the probability of illegal actions to 0 and outputs the probability distribution of valid actions; masking mechanism. The function is defined as: in, For a constant, when Enough hours The probability of the i-th illegal action output is 0; after obtaining the probability distribution of valid actions, random samples are taken from them and the corresponding actions are executed in the environment; subsequently, the immediate reward is calculated according to the formula of the reward function. And obtain the next state. Simultaneously, the state transition process of the current time slot is stored in the experience replay pool. Using the data as training samples, the temporal difference residual generated in each time slot is calculated and used for subsequent optimization of the Critic network. Its definition is: in, For instant rewards, As a discount factor, For strategy In state The expected reward that can be obtained at that time; Fitting using an Actor network and The mapping relationship between them is used to explore the optimal Actor network, so that the appropriate action can be selected in each state to maximize the cumulative reward; to achieve this goal, the old strategy is first temporarily stored in... Next, in each training epoch, the loss function of the Actor network is calculated, defined as: in, Functions are used for constraints This avoids fluctuations or sudden changes in the learning process caused by large differences between the old and new strategies, resulting in a smoother training process. This is used to control the shearing interval, the shearing interval being... ; Next, the Actor network parameters are updated using the gradient ascent method. The policy gradient is calculated as follows: In order to accurately calculate the corresponding state Values, fitted using a Critic network First, output the state using the Actor network. Below Next, the Critic network loss is calculated; then, the Critic network loss is minimized by gradient descent and then updated. The task scheduling strategy is updated iteratively; during each iteration, samples are taken based on the current task scheduling strategy. The system uses training samples and then uses gradient update methods to continuously update the parameters of the Actor network and the Critic network; finally, it trains until convergence and outputs the optimal task scheduling strategy.

2. The task scheduling method based on Actor-Critic deep reinforcement learning in mobile edge computing according to claim 1, characterized in that: In the task scheduling model under the MEC environment, the user's smart device connects to the base station and offloads tasks to the MEC node. Tasks enter the task queue one by one according to the offloading time, and the server equipped in the MEC node provides computing services. The servers configured on a MEC node are denoted as a set. The amount of computing and memory resources provided by each server is denoted as The task unloaded from the smart device to the MEC node is defined as a task queue, denoted as: Where m is the number of tasks; each task is defined as: ,in, Indicates the time when a task arrives in the task queue. Indicates the resources required for the task. Indicates the maximum tolerable delay for the task. This indicates the task execution time; the scheduler allocates tasks to the appropriate servers based on the resource requests of different tasks in the current MEC system and the resource usage of the servers; at the same time, each server records its resource usage at each time step; The task scheduling process is as follows: When a user-unloaded task arrives at the MEC node, the system attempts to add the task to the task queue to await execution. Due to limited MEC resources, the task queue length is also limited. If the task queue is full, no new tasks can be added. In this case, the task will be deemed to have failed to execute. If the task's waiting time exceeds its maximum tolerable delay, it will also lead to task execution failure. At any time, if the scheduler will The task is assigned to the MEC server, which will then allocate available resources to meet its resource requirements; At that moment, when Once execution is complete, the MEC node will send the execution result back to the user and record the response time of the task. Furthermore, the scheduler needs to determine whether each task in the queue meets the scheduling conditions. From the tasks to be scheduled, tasks are selected and scheduled to servers in the MEC nodes. Therefore, the computational complexity is O(n). To reduce computational complexity and enable the scheduler to efficiently formulate task scheduling strategies, the server is treated as a resource pool, and resource fragmentation between servers during task scheduling is ignored. At any given time, the scheduler allows multiple scheduling operations, selecting one task from the list of tasks to be scheduled each time, until a waiting signal is sent to the server.

3. The task scheduling method based on Actor-Critic deep reinforcement learning in mobile edge computing according to claim 2, characterized in that: The optimization objective of the task scheduling model in the MEC environment is to reduce task response time and increase task completion rate. On the one hand, the average relative waiting time of completed tasks is used to indirectly evaluate the task response time, which is defined as: in, This indicates that the task sequence has been completed; On the other hand, the task failure rate is used to indirectly assess the task completion rate, which is defined as: The optimization objective obtained is: in, and These represent the weights of different optimization metrics.

4. The task scheduling method based on Actor-Critic deep reinforcement learning in mobile edge computing according to claim 3, characterized in that: The state space, action space, and reward function are defined as follows: State space: State space Represents the set of all possible states observed by the DRL agent; in time slots , This includes the current task queue status and available resource status in the system, defined as follows: in, and These represent the available computing and memory resources in the resource pool, respectively. This indicates a task that is waiting to be executed in the task queue; Action space: in time slots ,action This refers to selecting a task from the task queue for scheduling, termination, or issuing a waiting signal, and is defined as follows: Among them, when When the DRL agent terminates execution time exceeding the maximum tolerable latency task; In the first case, the DRL agent does not perform task scheduling; unscheduled tasks wait in the queue. In other cases, the DRL agent selects a task from the task queue and allocates resources to it. The environment calculates and provides an immediate reward for the current action based on the reward function. When performing a scheduling operation, the scheduler allocates a virtual machine with the corresponding resources from the resource pool based on the resources required by the task. When performing a waiting operation, if there are completed tasks, the resources occupied by the virtual machine will be released. When transitioning to the next system state, the available resources in the resource pool or the number of tasks in the queue may increase. If the available resources in the system are insufficient to complete a task with minimum resource requirements, a waiting operation can only be performed until a task is completed and resources are released. Reward function: when in time slot After an action is performed, the environment provides a corresponding immediate reward; based on the optimization objective, the reward function is defined as: in, This represents the number of times the task failed to execute. and These represent the weights of task failure rate and relative task response time, respectively; to avoid slow training and tortuous gradient descent caused by numerical differences, [the following is used:] and Normalization is used to improve the convergence speed of the method; it should be noted that in order to maximize the cumulative reward, the DRL agent needs to comprehensively consider the immediate reward and the future reward to select the appropriate action.

Images