A reinforcement learning-based cloud-edge computing task scheduling method

Abstract

This invention relates to a reinforcement learning-based cloud-edge computing task scheduling method in the field of combinatorial optimization in computer science. It establishes server queues and task queues, builds a system model based on the cloud-edge computing task scheduling optimization objective, constructs an LSTM network model architecture based on an attention mechanism, and outputs the probability distribution of resource allocation according to the attention weights of different tasks. Then, an improved A3C-IW reinforcement learning algorithm is used with an asynchronous policy gradient method to obtain task scheduling decisions. A reward function R is defined for each step of the reinforcement learning process, and the optimal computing task scheduling strategy is obtained for different reward-based task scheduling strategies. This invention effectively balances the resource utilization and service quality of cloud-edge systems by using reinforcement learning, minimizing response time and energy consumption, and is better adapted to computing task scheduling in a cloud-edge collaborative environment.

Description

Technical Field

[0001] This invention relates to the field of combinatorial optimization technology in computer science, and in particular to a cloud-edge computing task scheduling method based on reinforcement learning. Background Technology

[0002] With the development of wireless communication and IoT technologies, smart manufacturing has placed higher demands on performance indicators such as latency, energy consumption, and throughput in the production process. While traditional cloud computing possesses powerful computing, storage, and network resources, it still experiences uncontrollable service latency when faced with surging data traffic, making centralized processing models insufficient to meet users' high-performance requirements. Edge computing places resources closer to the terminal, providing computing, storage, and communication capabilities at the network edge. However, edge devices typically have limited storage capacity and weaker processors, resulting in low latency but resource constraints. Cloud-edge collaboration, by combining the advantages of both cloud and edge computing paradigms, can effectively improve resource service capabilities and quality, enabling rational allocation of cloud and edge computing resources and possessing broad development prospects.

[0003] In the context of the Internet of Things (IoT), performing cloud-edge computing resource scheduling can optimize resource utilization, reduce latency, and save energy. However, resource scheduling in a cloud-edge environment is challenging due to factors such as the different states, types, and sizes of tasks, the heterogeneity and geographical dispersion of cloud-edge computing resources, and the diverse performance and stability requirements of different applications.

[0004] Existing research on cloud-edge computing resource scheduling employs heuristic algorithms, simulating natural phenomena or biological behaviors to find approximate optimal solutions. However, the solution process can be affected by initial parameters, easily leading to only local optima. Some studies have explored the application of machine learning algorithms, but these may require large amounts of labeled data, struggle to handle dynamically changing environments, and may have limitations, such as a lack of simultaneous optimization of energy consumption and response time. Others utilize reinforcement learning algorithms such as Deep Q-Learning (DQL), but DQL typically requires significant time for interactive learning of effective policies. While these methods are widely adopted, they still may not guarantee the required quality of service, especially in dynamic cloud-edge computing environments where task response time is critical. Addressing the problems of long average response time, high energy consumption, and low resource utilization in existing task scheduling methods remains a pressing issue.

[0005] To solve the problems of long average response time, high energy consumption, and low resource utilization in existing task scheduling, we designed a cloud-edge computing task scheduling method based on reinforcement learning. By deploying the reinforcement learning model in a distributed manner, the cloud server and the edge server cooperate to perform task scheduling and generate a globally optimal resource scheduling strategy. The core of this strategy is to minimize the response time and energy consumption as much as possible while ensuring that the quality of service (QoS) is met, and to improve the utilization rate of cloud-edge resources. Summary of the Invention

[0006] To overcome the deficiencies in the background art, the present invention discloses a cloud-edge computing task scheduling method based on reinforcement learning.

[0007] To achieve the above invention objectives, the present invention first establishes a server queue and a task queue, builds a system model based on the cloud-edge computing task scheduling optimization objective, constructs an LSTM (Long Short-Term Memory) network model architecture based on the attention mechanism, outputs the probability distribution of resource allocation according to the attention weights of different tasks, and then adopts an improved A3C-IW (Asynchronous Advantage Actor-Critic with Importance Weights) algorithm model. Using the asynchronous policy gradient method, a task scheduling decision is obtained. Define the reward return function R for each step of reinforcement learning, and obtain the optimal computing task scheduling strategy for different reward return task scheduling strategies. The specific technical solutions are as follows:

[0008] A cloud-edge computing task based on reinforcement learning includes the following steps:

[0009] Step 1: Assign n independent computing tasks to m server computing resources for processing, where m < n; among them, the task set T = {T1, T2, T3......T n}, and the parameters of the task T i are: T i =(T i s , T i m , T i bw );

[0010] In the formula, T i s is the task size, T i m is the expected memory required to process the task, and T i bw is the expected bandwidth required to process the task;

[0011] The server set V = {V1, V2, V3, ..., V} m}, server parameter V j for:

[0012] In the formula, Indicates the server's processing capacity. This indicates the available memory on the server. Indicates the available bandwidth of the server;

[0013] The execution time of a task is divided into scheduling intervals of equal duration, and the scheduling intervals are numbered according to the order in which they occur: X = {X1, X2, ...};

[0014] Step 2: Establish an LSTM neural network model based on the attention mechanism. Introduce the attention mechanism at the input of the LSTM neural network model, map the parameters of different servers and tasks into attention vectors, and adjust the writing ratio of different parameters to memory cells by weighted superposition of attention vectors.

[0015] Step 3: Establish an improved reinforcement learning A3C-IW algorithm model. Importance weights are added to the A3C algorithm model to obtain the improved A3C-IW algorithm model. The A3C-IW algorithm model includes a main network and several threads with local networks. The threads interact independently with the environment to obtain empirical data. The main network receives data uploaded by each local network, takes the current state as the network input, and outputs the policy function and value function of the current state.

[0016] The importance weight is introduced into the gradient update by a weight factor and multiplied with the policy gradient, which is used to update the policy network parameters.

[0017] Step 4: Based on the output of the LSTM neural network model, the A3C-IW algorithm model is applied for policy learning and optimization. The task assignment probability distribution p′ is weighted and summed with the probability distribution generated by the Actor network to obtain the final scheduling policy.

[0018] Preferably, attention weights are calculated using an attention mechanism and then weighted with the input features to form weighted features, which are then used as input to the LSTM. for:

[0019] In the formula, x represents the input feature, located as Represents the feature vector of the server. The feature vectors represent the task, and W and b are the learnable weight matrix and bias vector, respectively.

[0020] Then attention weights Weighted by the input feature x;

[0021] In the formula, ⊙ represents element multiplication;

[0022] Finally, the weighted feature weighted_x is used as the input to the LSTM neural network model;

[0023] The output of the LSTM neural network model is: y = LSTM(weighted_x); y is then converted into a probability distribution: p′ = softmax(y).

[0024] In the formula, p′ represents each allocation strategy S i,j The probability of.

[0025] Preferably, the A3C-IW algorithm model has the following two operations:

[0026] W1: Directly assign the parameters of the main network to the local network;

[0027] W2: Update the parameters of the main network using gradients from each thread;

[0028] In this process, after each thread interacts with the environment and receives a certain amount of data, the gradient of the neural network loss function in the thread is calculated, and the main network is updated.

[0029] Preferably, the A3C-IW algorithm model outputs the state value function V through the Critic network in the main network and the corresponding policy function π through the Actor network in the main network, respectively, based on the input state S.

[0030] Preferably, the benefit reward function R = -Loss for each step of reinforcement learning is defined as a measure of model parameter updates;

[0031] Where Loss represents the sum of response time and energy consumption for each scheduling strategy;

[0032] Define Loss i Loss i =ζ·MCT i-1 +ψ·EC i-1 ;(ζ,ψ≥0)∧(ζ+ψ=1).

[0033] ζ and ψ are weighting factors, representing the importance of maximum completion time and energy consumption in the overall performance evaluation.

[0034] Preferably, the maximum completion time calculation formula is defined as: MCT i =S iup +S i work

[0035] In the formula, S i up This is the time required to upload the task, S i work It is the task execution time;

[0036] in,

[0037] In the formula, T i s For task size, T i bw The bandwidth required to handle the task is denoted by d, where d represents the interference encountered during transmission. Indicates the server's processing capacity; f i ∈[0,f i max ] indicates the frequency of the server CPU, f i max It is the maximum CPU frequency that the server can achieve;

[0038] The constraints are as follows:

[0039] Preferably, the energy consumption calculation formula is defined as follows: P i =κ·(f i ) 2

[0040] In the formula, P i It represents the power consumption of the server in processing computing tasks, where κ is a power coefficient, a constant that depends on the CPU model.

[0041] The constraints are as follows:

[0042] Preferably, the method for updating the Actor network is as follows: in each policy function process generated by the Actor network, a Markov decision process is modeled: (S,A,P,R,γ);

[0043] Where S is the state space, and A is the action space, represented as A = {a i,j}, a i,j This indicates that task i will be scheduled to server j for computation, where P is the state transition probability, R is the reward function, and γ is the discount factor.

[0044] Where γ = 1 means that the future reward has the same value as the current reward, and γ < 1 means that the future reward will be discounted over time.

[0045] In action space A and state space S, given the current state s t The intelligent agent performs action a t a t The environment will provide a reward R based on random sampling according to the scheduling policy π(a|s). t and new state s t+1 .

[0046] Preferably, the method for updating the Critic network is as follows: The parameters are updated by minimizing the loss function, and the temporal difference objective (TD) is calculated as the estimate of the state value function made by the target value network at time t+1.

[0047] In the formula, v(s) t+1 ω) is the state-value network's estimate of the state-value function at time t+1, v(s) t ;ω) and Both are estimates of the state-value function, because Based on the actual observed reward R t Therefore, it is fixed. ω is a parameter of the value network. Update ω so that v(s) t ;ω) is closer

[0048] Define the loss function:

[0049] set up The gradient of the loss function is:

[0050] The Time Difference (TD) error, using the temporal difference method, represents the difference between the predicted and actual values ​​of a value network. The TD error is defined as...

[0051] Perform one round of gradient descent to update ω:

[0052] In the formula, It is the learning rate.

[0053] The preferred method for updating the policy network is as follows: Based on the Bellman formula, the action-value function is obtained:

[0054]

[0055] According to the policy gradient theorem, by analyzing the policy gradient... By performing the Monte Carlo approximation, we obtain an unbiased estimate of the gradient of the stochastic policy:

[0056] In the formula, Q π (s t ,a t ) is the action value function, v(s) t ω) is the state value function. It is the gradient of the policy;

[0057] The agent performs action a t Then, the environment gives a new state s. t+1 and reward R t ; using s t+1 and R t Applying the Monte Carlo approximation to the above expectation, we obtain:

[0058] Based on the TD target and TD error, Written as:

[0059] In the formula, It is an approximation of g;

[0060] Using exponential moving average smoothing technique to analyze the dominance function A(s) t ,a t )=Q(s t ,a t )-V(s t The function is then smoothed and normalized to fit within the range [0,1], resulting in the processed dominance function.

[0061] The advantage function is used as an importance weight, multiplied by the policy gradient, and used to update the policy network parameters. Update the policy network parameter θ:

[0062] In the formula, β is the learning rate.

[0063] By employing the technical solution described above, the present invention has the following beneficial effects:

[0064] 1. By deploying reinforcement learning models in a distributed manner, cloud servers and edge servers collaborate to schedule tasks and generate a globally optimal resource scheduling strategy. The core of this strategy is to minimize response time and energy consumption while ensuring service quality and improving the utilization rate of cloud and edge resources.

[0065] 2. Server queues and task queues were established. A system model was built based on the goal of optimizing cloud-edge computing task scheduling. An LSTM network model architecture based on an attention mechanism was constructed. According to the attention weights of different tasks, the probability distribution of resource allocation was output. Then, an improved A3C-IW algorithm model was adopted, using an asynchronous policy gradient method to obtain task scheduling decisions. The reward function R for each step of reinforcement learning was defined. For task scheduling strategies with different reward rewards, the optimal computing task scheduling strategy was obtained. This invention effectively balances the resource utilization and service quality of cloud-edge systems by using reinforcement learning, minimizing response time and energy consumption, and can better adapt to computing task scheduling in a cloud-edge collaborative environment. Attached Figure Description

[0066] Figure 1 This is a flowchart of the present invention;

[0067] Figure 2 This is a system framework diagram of the present invention;

[0068] Figure 3 This is a structural framework diagram of the A3C-IW algorithm model in this invention;

[0069] Figure 4 This is a diagram of the Actor-Critic network structure in this invention. Detailed Implementation

[0070] The present invention will be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that if terms such as "upper", "lower", "front", "rear", "left", "right" indicate orientation or positional relationship, they are only corresponding to the drawings of this application for the purpose of describing the present invention. It should be understood that if terms such as "end", "side", "end portion", "side part", "lateral", "longitudinal", etc. indicate orientation or positional relationship, they are only corresponding to the length and width of the corresponding component. That is, "end" indicates the head and tail area in the length direction of the corresponding component, and "side part" indicates the head and tail area in the width direction of the corresponding component. The present invention is disclosed for the purpose of describing the present invention and does not indicate or imply that the device or element referred to must have a specific orientation.

[0071] A reinforcement learning-based cloud-edge computing task scheduling method is proposed. For specific task scenarios, we design the entire intelligent manufacturing system into multiple independent and autonomous schedulers. Each scheduler dynamically allocates tasks to edge and cloud servers. Each scheduler contains several cloud servers, edge servers, and terminal devices.

[0072] like Figure 2As shown, task information is first uploaded from the production terminal to the cloud database, and simultaneously synchronized to the edge server. Task information includes task attributes and requirements. The database stores all information, including server resource information and task information. The cloud formulates a resource allocation strategy based on the global resource status and task requirements, while the edge server formulates a local resource allocation strategy based on its local server status and task requirements. Task information, server status, and scheduling decisions are periodically synchronized between the cloud and the edge to ensure data consistency. Communication between the cloud and the edge is achieved via a wireless network.

[0073] First, the problem is described and the relevant mathematical symbols are explained, as shown below:

[0074] X i Defined as the i-th scheduling interval;

[0075] m is defined as the number of servers;

[0076] n is defined as the number of computation tasks;

[0077] t i ,t i+1 Defined as the start and end times of the i-th scheduling interval;

[0078] T i Defined as the i-th task in the task set;

[0079] T i s Defined as task T i Size;

[0080] T i m Defined as processing task T i Expected memory requirements;

[0081] T i bw Defined as processing task T i Expected bandwidth required;

[0082] V j Defined as the j-th server in the server set;

[0083] Defined as the server's processing capacity;

[0084] Defined as server V j Available memory;

[0085] Defined as server V j Available bandwidth;

[0086] A feature vector defined to include server parameters and task parameters;

[0087] S i,j Defined as scheduling the i-th task to the j-th server;

[0088] Defined as the optimal scheduling strategy.

[0089] Including the following steps:

[0090] Step 1: Assign n independent computing tasks to m server computing resources for processing, where m < n; among them, the task set T = {T1, T2, T3......T n}, T i represents the i-th task, and the parameters of the task T i are: T i =(T i s , T i m , T i bw );

[0091] In the formula, T i s is the task size, T i m is the expected memory required to process the task, and T i bw is the expected bandwidth required to process the task;

[0092] The server set V = {V1, V2, V3......V m}, V j represents the j-th server, and the parameters of the server V j are:

[0093] In the formula, represents the processing capacity of the server, that is, the number of cores of the server's central processing unit CPU (Central Processing Unit, CPU). Multiple cores indicate that the server can process multiple tasks in parallel, represents the available memory of the server, represents the available bandwidth of the server;

[0094] In the considered model, time is discrete. Divide the execution time of the task into scheduling intervals of equal duration and number the scheduling intervals according to the order of occurrence: X = {X1, X2,...}; that is, the i-th scheduling interval is shown as X i , which starts from time t iIt begins and continues until the beginning of the next interval, i.e., t. i+1 .

[0095] Step 2: Establish an LSTM (Long Short-Term Memory Network) neural network model based on the attention mechanism. Introduce the attention mechanism at the input of the LSTM neural network model, map the parameters of different servers and tasks into attention vectors, and adjust the writing ratio of different parameters to memory cells by weighted superposition of attention vectors.

[0096] As needed, the input to the LSTM neural network model is x, which consists of server parameters, including the number of CPU cores, available memory size, and available bandwidth. All these server parameters are represented in a feature vector as follows: Task T i Including parameters such as task size, required memory, and required bandwidth, the feature vector of the task is represented as... Therefore, x i Defined as As input to the model, parameters from different servers and tasks are mapped to attention vectors, and the weighted summation of these attention vectors adjusts the write ratio of different parameters to memory cells.

[0097] Furthermore, an attention mechanism is introduced at the input of the LSTM to calculate attention weights, which are then weighted with the input features to form weighted features, which are used as the input to the LSTM. The calculation of these attention weights... The formula is as follows:

[0098] In the formula, x represents the input feature, and W and b are the learnable weight matrix and bias vector, respectively.

[0099] Then attention weights Weighted by the input feature x:

[0100] Where ⊙ represents element-wise multiplication, and finally the weighted feature weighted_x is used as the input to the LSTM model.

[0101] The output of the LSTM model is represented as: y = LSTM(weighted_x)

[0102] Transform y into a probability distribution: p′=softmax(y)

[0103] p′ is a matrix representing each allocation strategy S i,j The probability is the probability that the i-th task is assigned to the j-th task.

[0104] Specifically, the primary function of the LSTM neural network model is to address the vanishing or exploding gradient problems encountered by traditional RNNs (Recurrent Neural Networks) when processing long sequences of data. When processing sequences, LSTM models apply the same weights to every element, which can lead to an inability to distinguish which parts are more important to the current task. The introduction of the attention mechanism allows LSTM models to process sequence data more flexibly, especially when dealing with time steps of varying importance. By introducing the attention mechanism, the LSTM model can dynamically focus on the parts most important to the current task, dynamically allocating attention weights, thereby improving the model's performance and accuracy, and enabling it to better handle complex sequence data in real-world applications.

[0105] Step 3: Establish an improved reinforcement learning A3C-IW algorithm model. Importance weights are added to the A3C algorithm model to obtain the improved A3C-IW (A3C with Importance Weights) algorithm model, as shown below. Figure 3 As shown, the A3C-IW algorithm model includes a main network and several threads with local networks. The threads interact independently with the environment to obtain empirical data, meaning that the threads do not interfere with each other and run independently. The main network receives data uploaded by each local network, takes the current state as the network input, and outputs the policy function and value function of the current state.

[0106] The importance weight is introduced into the gradient update by a weight factor and multiplied with the policy gradient to update the policy network parameters, thereby reducing the non-stationarity of policy updates, improving sample efficiency, and improving the convergence of the algorithm.

[0107] The A3C-IW algorithm model has two operations: one is to directly assign the parameters of the main network to the local networks, and the other is to update the parameters of the main network using the gradients from each thread. After each thread interacts with the environment and receives a certain amount of data, it calculates the gradient of the loss function of its own neural network. However, these gradients do not update the neural network within their own thread, but instead update the main network. In other words, n threads independently use the accumulated gradients to update the network model parameters of the main network. Every certain period of time, each thread updates the parameters of its own neural network to the parameters of the main network, thereby guiding subsequent interactions with the environment.

[0108] Furthermore, such as Figure 4As shown, the A3C-IW algorithm model outputs the state value function V through the Critic network in the main network and the corresponding policy function π through the Actor network in the main network, respectively, based on the input state S.

[0109] Step 4: Based on the output of the LSTM model, the A3C-IW algorithm is applied to learn and optimize the policy. The task assignment probability distribution p′ is weighted and summed with the probability distribution generated by the Actor network to obtain the final scheduling policy.

[0110] In other words, the Critic network (value network) is used to calculate the state value function after taking an action in the current state, and the action with the highest state value is selected.

[0111] Overall, such as Figure 1 As shown, firstly, feature vectors containing server and task information are obtained separately. These are then weighted with attention weights obtained through an attention mechanism and used as input to the LSTM model. The LSTM model outputs a probability distribution containing all scheduling strategies. Based on the LSTM model output, a reinforcement learning algorithm is applied to obtain the optimal scheduling strategy.

[0112] Furthermore, to maximize the expectation that the model is optimal in order to reduce the loss in each interval and thus reduce the cumulative loss, we define the benefit reward function R = -Loss for each step of reinforcement learning as a measure of model parameter updates.

[0113] Where Loss represents the sum of response time and energy consumption for each scheduling strategy;

[0114] Define Loss i Loss i =ζ·MCT i-1 +ψ·EC i-1 ;(ζ,ψ≥0)∧(ζ+ψ=1);

[0115] ζ and ψ are weighting factors, representing the importance of maximum completion time and energy consumption in the overall performance evaluation.

[0116] Below are the definitions of completion time and energy consumption.

[0117] Define the formula for calculating the Maximum Completion Time (MCT): MCT i =S i up +S i work

[0118] In the formula, S iup This is the time required to upload the task, S i work It is the task execution time;

[0119] in,

[0120] In the formula, T i s For task size, T i bw The bandwidth required to handle the task is denoted by d, where d represents the interference encountered during transmission. f represents the server's processing power, specifically the number of CPU cores. i ∈[0,f i max ] indicates the frequency of the server CPU, f i max It is the maximum CPU frequency that the server can achieve;

[0121] Define the formula for calculating Energy Consumption (EC): P i =κ·(f i ) 2

[0122] In the formula, P i It represents the power consumption of the server in processing computing tasks, where κ is a power coefficient, a constant that depends on the CPU model.

[0123] This can be described as follows:

[0124] The constraints are: This indicates that the memory used for the task being processed cannot exceed the server's memory. This means that the bandwidth required to process the task cannot exceed the server bandwidth; This means that the CPU frequency allocated to the server for processing tasks cannot exceed the maximum CPU frequency. This means that when the server schedules a task to an edge server for processing, the power consumption of that edge server for processing the computing task cannot exceed the maximum power.

[0125] Furthermore, the task scheduling algorithm based on the A3C-IW algorithm model includes a global Actor-Critic network and several threads. Each thread is a complete AC network, containing an Actor network, a Critic network, and a local environment, which can run in parallel and interact with their local environment to learn. The Actor model is used to generate scheduling decisions, while the Critic model is used to evaluate the value function of the state. The updating of the neural network model mainly involves updating the policy network and the value network. That is, the policy network and the value network are trained by updating their parameters.

[0126] For each scheduling decision process generated by the Actor network, it is modeled as a Markov decision process: (S, A, P, R, γ); S is the state space, containing all possible states. In cloud-edge server scheduling, the state includes the current state of the server and the server where the task is located; A is the action space, containing all possible actions. In the scheduling process, the action includes the task scheduling decision, represented as: A = {a i,j}

[0127] In the formula, a i,j This indicates that task i will be scheduled to server j for computation; P is the state transition probability, which is the probability that the system state will transition to a new state after taking a specific action. These probabilities depend on the current state and the action taken; R is the reward function, which is the loss value obtained after taking an action in the current state, R = -Loss; γ is the discount factor, used to calculate the cumulative reward of future rewards, γ = 1 means that the future reward is the same as the current reward, while γ < 1 means that the future reward will be discounted over time.

[0128] In action space A and state space S, given the current state s t The intelligent agent performs action a t a t It is obtained by random sampling based on the scheduling policy π(a|s), and the environment will provide a reward R. t and new state s t+1 .

[0129] For the value network, the parameters are updated by minimizing the loss function, and the temporal difference (TD) objective is calculated as the estimate of the state value function made by the target value network at time t+1.

[0130] In the formula, v(s) t+1 ω) is the state-value network's estimate of the state-value function at time t+1, v(s) t ;ω) and Both are estimates of the state-value function, because The reward R is partly based on actual observations. t Therefore, it is fixed. ω is a parameter of the value network. Update ω so that v(s) t ;ω) is closer

[0131] Define the loss function:

[0132] set up The gradient of the loss function is:

[0133] The TD error, expressed using the time-difference method, represents the difference between the predicted and actual values ​​of a value network. The TD error is defined as... Perform one round of gradient descent to update ω ：

[0134] In the formula, It is the learning rate.

[0135] Such an expectation model update can improve the prediction of the value network v(s) t ;ω) is closer to the target value

[0136] For the policy network, the policy gradient is used to update the parameters: according to the Bellman formula, the action-value function is obtained:

[0137] According to the policy gradient theorem, by analyzing the policy gradient... By performing the Monte Carlo approximation, we obtain an unbiased estimate of the gradient of the stochastic policy:

[0138]

[0139] In the formula, Q π (s t ,a t ) is the action value function, v(s) t ω) is the state value function. It is the gradient of the policy.

[0140] When the agent performs action a t Then, the environment gives a new state s. t+1 and reward R t ; using s t+1 and R t Applying the Monte Carlo approximation to the above expectation, we obtain:

[0141] Based on the previously defined TD target and TD error, we can... Written as:

[0142] It is an approximation of g, and therefore also an approximation of the policy gradient.

[0143] The advantage function measures the additional expected reward of taking a particular action compared to taking a random strategy. It helps us evaluate the merits of an action and guides the learning process.

[0144] Dominance function calculation formula: A(s) t ,a t )=Q(s t ,a t )-V(s t )

[0145] Since the advantage function depends on multiple random factors and has a large variance, directly multiplying it with the gradient can lead to unstable parameter updates. Therefore, it is necessary to use the exponential moving average method to smooth the advantage function and then normalize it to make its range between [0,1], thereby avoiding gradient explosion or gradient vanishing problems, and obtaining the processed advantage function.

[0146] The resulting advantage function, used as importance weights, is multiplied by the policy gradient and used to update the policy network parameters. Update the policy network parameter θ:

[0147] Where β is the learning rate, using the advantage function can reduce the variance of policy gradient estimation, improve learning efficiency, and accelerate the convergence of loss.

[0148] It should be noted that the A3C-IW algorithm model is an asynchronous reinforcement learning algorithm that combines the Actor-Critic method. Compared with Actor-Critic, it adds an advantage function to calculate the gradient of the policy network, thereby guiding the network update. It utilizes the asynchronous policy gradient method to adapt to dynamically changing environments.

[0149] Adding a target network to the A3C-IW algorithm model to calculate the target value helps in better estimating the value function. The target network is used to calculate the TD target; let's denote the target network as... The structure is the same as that of a value network, but the parameters are different.

[0150] The training process of the A3C-IW algorithm model is summarized below:

[0151] Initializing the network: Value Network V ω(s), policy network π θ (s), target network

[0152] Initialize network parameters: θ, ω, Discount factor γ, learning rate β

[0153] Input: State space S, scheduling interval N, time step = 0, maximum time step max

[0154] Output: The optimal scheduling decision for the task.

[0155] Start loop:

[0156] Step 1: Observe the current state s t Make decisions based on the policy network: a t ~π(·|s t ;θ), and perform action a t ;

[0157] Step 2: Observe the reward r from the environment t and new state s t+1 Calculate the advantage function

[0158] If step % N! = 0;

[0159] Step 3: Based on the policy information output by the LSTM network model, use the value network to calculate the state value function of the current state;

[0160] Step 4: Generate task scheduling policy S by combining the policy network. i,j ;

[0161] Step 5: Select the optimal strategy based on the optimization target Loss.

[0162] else:

[0163] Step 6: Let the value network give s t Evaluate:

[0164] Step 7: Let the target network give s t+1 Evaluate:

[0165] Step 8: Calculate the TD target and TD error:

[0166] Step 9: Update the value network:

[0167] Step 10. Update the policy network:

[0168] Step 11: step+=1

[0169] if step == step max

[0170] Step 12: End the loop;

[0171] Step 13: Algorithm ends.

[0172] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.

Claims

1. A method for cloud-edge computing task scheduling based on reinforcement learning, characterized in that: The method comprises the following steps: Step one: distribute n independent computing tasks to m server computing resources for processing, and m < n; wherein the task set T = {T1, T2, T3... Tn}, the parameters of the task T n i i i s i m i bw} are: T i = (T i s ,T i m ,T i bw ). where T i s is the task size, T i m is the expected memory required to process the task, T i bw is the expected bandwidth required to process the task; A set of servers V = {V1, V2, V3... V m}, the parameters V j of the servers are: wherein represents the processing power of the server, represents the available memory of the server, represents the available bandwidth of the server; Divide the execution time of the task into scheduling intervals with equal duration, and number the scheduling intervals according to the occurrence sequence: X={X1, X2,...}; Step two: establish an LSTM neural network model based on an attention mechanism, introduce the attention mechanism at the input end of the LSTM neural network model, map the parameters of different servers and tasks into attention vectors, and adjust the writing proportion of different parameters to the memory unit through weighted superposition of the attention vectors; Step three: establish an improved reinforcement learning A3C-IW algorithm model, add importance weights to the A3C algorithm model to obtain the improved A3C-IW algorithm model, the A3C-IW algorithm model comprises a main network and a plurality of threads with local networks, the threads independently interact with the environment to obtain experience data, and the main network receives data uploaded by each local network, takes the current state as the input of the network, and outputs the policy function and the value function of the current state; The A3C-IW algorithm model outputs the state value function V through the Critic network in the main network and outputs the corresponding policy function π through the Actor network in the main network; The method for updating the Actor network is that, in the process of generating each policy function by the Actor network, a Markov decision process is modeled: (S, A, P, R, γ); where S is the state space, A is the action space, denoted as A = {a i,j}, a i,j represents scheduling task i to server j for computation, P is the state transition probability, R is the reward function, and γ is the discount factor; Wherein, γ=1 indicates that the future reward and the current reward value are the same, and γ<1 indicates that the future reward is discounted over time; In the action space A and the state space S, given the current state s t , the agent performs an action a t , a t According to the scheduling strategy π(a|s), a random sample is obtained, and the environment will give the reward R t and the new state s t+1 ; The method for updating the Critic network is that, the parameters are updated by minimizing the loss function, and the TD target of the time series difference method is calculated as the estimation of the state value function by the target value network at t+1 time: where v(s t+1 ; ω) is the estimate made by the state-value network at time t + 1 of the state-value function, v(s t ; ω) and are estimates of the state-value function, since is based on the true observed reward R t , so that ω is the parameter of the value network, and ω is updated so that v(s t ; ω) is closer to Define the loss function: Let The gradient of the loss function is: The TD error represents the difference between the predicted value of the value network and the actual value using the time-difference method, and the TD error is defined as Make a gradient descent update to ω: In the formula, is a learning rate; Wherein, the importance weight is introduced by multiplying a weight factor in the gradient update, and is multiplied by the policy gradient to update the policy network parameters; Step four: apply A3C-IW algorithm model on the basis of LSTM neural network model output, perform policy learning and optimization, and obtain final scheduling strategy by weighting and combining task allocation probability distribution p' and probability distribution generated by the Actor network 2.The cloud-edge computing task scheduling method based on reinforcement learning according to claim 1, wherein: The attention mechanism is used to calculate attention weights, and the input features are weighted to form weighted features as the input of the LSTM, and the attention weights are: where x represents input features, and a feature vector representing the server, a feature vector representing the task, and W and b are learnable weight matrix and bias vector; The attention weights are then applied to the input features x to weight the input features x; In the formula, ⊙ represents element multiplication; Finally, the weighted feature weighted_x is taken as the input of the LSTM neural network model; Wherein, the output of the LSTM neural network model is: y=LSTM(weighted_x); Convert y into a probability distribution: p'=softmax(y) In the formula, p' represents the probability of each distribution policy S i,j . 3.The cloud-edge computing task scheduling method based on reinforcement learning of claim 1, wherein: The A3C-IW algorithm model has the following two operations: W1: directly assign the parameters of the main network to the local network; W2: update the parameters of the main network using the gradient in each thread; Wherein, after each thread interacts with the environment to a certain amount of data, the gradient of the neural network loss function in the thread is calculated, and the main network is updated. 4.The cloud-edge computing task scheduling method based on reinforcement learning of claim 1, wherein: Define the benefit return function R=-Loss of each step of reinforcement learning as a measurement of model parameter update; Wherein, Loss represents the sum of the response time and the energy consumption of each scheduling strategy. Definition of Loss i : Loss i = ζ · MCT i-1 + ψ · EC i-1 ; (ζ, ψ ≥ 0) ∧ (ζ + ψ = 1) ; Wherein, ζ and ψ are weight factors, representing the importance of the maximum completion time and the energy consumption in the overall performance evaluation. Define the maximum completion time calculation formula: MCT i = S i up + S i work In the formula, S i up is the time required to upload the task, S i work is the task run time; wherein where T i s is the task size, T i bw is the expected bandwidth required to process the task, d is the interference of the task during transmission, denotes the processing power of the server; f i ∈ [0, f i max denotes the frequency of the server CPU, f i max is the maximum CPU frequency that the server can reach; wherein the constraints are: Definition of energy consumption calculation formula: P i = κ · (f i ) 2 In the formula, P i It represents the power consumption of the server in processing computing tasks, where κ is a power coefficient, a constant that depends on the CPU model. wherein the constraints are:

5. The method of claim 1, wherein: The updating method of the strategy network is as follows: according to the Bellman formula, the action value function is obtained: According to the policy gradient theorem, an unbiased estimate of the stochastic policy gradient is obtained by performing a Monte Carlo approximation of the policy gradient ​ where Q π (s t , a t ) is the action value function, v(s t ; ω) is the state value function, is the gradient of the policy. The agent performs action a t The environment then gives a new state s t+1 And a reward R t ; Using s t+1 And R t Monte Carlo approximation of the above expectation gives: According to the TD target and the TD error, the TD target and the TD error are written as: ​ wherein is an approximation of g; Using exponential moving average smoothing technique to analyze the dominance function A(s) t ,a t )=Q(s t ,a t )-V(s t The function is then smoothed and normalized to fit within the range [0,1], resulting in the processed dominance function. The advantage function is taken as the importance weight, multiplied with the policy gradient, used to update the policy network parameters, with updating the policy network parameters θ: In the formula, β is the learning rate.

Images