Serverless edge task offloading method based on experience sharing deep reinforcement learning

Abstract

The present invention discloses a serverless edge task offloading method based on experience-sharing deep reinforcement learning. The method comprises: modeling a task offloading performance optimization problem, and generating a serverless edge task offloading method by deep reinforcement learning based on experience sharing. The present invention first models the task offloading optimization problem in each serverless edge computing environment as a Markov decision process, and adopts the deep reinforcement learning of the improved actor-critic framework to solve it. Then, each environment shares experience data through a playback cache located in a cloud center to improve data diversity and sample efficiency and reduce exploration cost. Finally, a strategy search method based on population guidance is introduced to accelerate the convergence of a deep reinforcement learning (DRL) agent and has a lower exploration cost.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and more specifically, relates to a serverless edge task offloading method based on experience sharing deep reinforcement learning. Background Art

[0002] With the increase of intelligent IoT applications and their data in serverless edge computing, resource-poor IoT (Internet of Things) devices find it difficult to guarantee the performance of delay-sensitive tasks. Due to the low bandwidth and high communication delay between IoT devices and cloud servers, the traditional cloud offloading method of offloading tasks to the cloud cannot meet the performance requirements of delay-sensitive IoT applications. Edge computing is one of the feasible solutions to the above problems. By using task offloading technology, IoT devices can offload computing tasks to the edge for execution, shorten task response delays and avoid network congestion, and meet multiple requirements such as security, privacy, and application intelligence. In order to optimize edge computing performance and facilitate the development and deployment of event-driven edge intelligent IoT applications, serverless computing has been extended from cloud computing to edge computing, resulting in serverless edge computing. Some cloud service providers have also provided an edge FaaS platform (EdgeFaaS, referred to as EFaaS; FaaS is Function as a Service) for function execution in edge servers in edge IoT environments, which has attracted widespread applications in various fields such as IoT services, stream processing, and AI reasoning and prediction.

[0003] Due to the limited resources of the serverless edge computing system, when all the task requests of IoT devices are offloaded to EFaaS, the workload of EFaaS may be too high, resulting in task accumulation and resource contention, resulting in increased task delays and even task failures. Therefore, some task requests need to be executed on IoT devices and some offloaded to EFaaS for execution, so as to ensure the efficiency of edge system resource utilization and overall system performance. However, generating efficient offloading strategies for task requests of IoT devices with the goal of optimizing performance is still one of the most challenging problems in edge computing. To solve the above problems, many traditional heuristic optimization research works based on genetic algorithms or game theory algorithms have achieved some results, but these optimization algorithms require a large number of iterations and only consider one-time optimization, which is difficult to adapt to the dynamically changing serverless edge computing environment. In addition, in edge computing, existing research on task offloading optimization based on deep reinforcement learning (DRL) still has challenges. First, a high-performance DRL agent training requires a large amount of diverse data. Most of the existing studies obtain data through interactive trial and error in a single environment. This method has the disadvantages of insufficient data diversity and takes a lot of time to obtain enough data. However, the training of DRL agents has very high trial-and-error and exploration costs, which cannot be borne by a single environment. Summary of the invention

[0004] In view of the above defects or improvement needs of the prior art, the present invention provides a serverless edge task offloading method based on experience-sharing deep reinforcement learning, and proposes a distributed task offloading method based on experience-sharing deep reinforcement learning (ES-DRL). First, the task offloading optimization problem in each serverless edge computing environment is modeled as a Markov decision process, and is solved by deep reinforcement learning using an improved Actor-Critic framework. Then, each environment shares experience data through a replay cache located in the cloud center to improve data diversity, sample efficiency, and reduce exploration costs. Finally, in order to accelerate the convergence of DRL agents, a strategy search method based on population guidance is introduced.

[0005] To achieve the above object, the present invention provides a serverless edge task offloading method based on experience sharing deep reinforcement learning, comprising the following steps:

[0006] (1) ES-DRL, a distributed task offloading method based on experience sharing deep reinforcement learning, uses deep reinforcement learning of the improved Actor-Critic framework to solve the task offloading optimization problem: in the stateful and stateless combined execution model scenario of serverless edge computing, the multi-task offloading problem is transformed into a joint optimization problem, and the performance optimization problem of the task offloading decision process is modeled as a Markov decision process (MDP); taking advantage of the advantages of DRL, the DRL algorithm of the improved Actor-Critic framework is used to solve the problem, which includes adding policy noise to the output action of the target Actor strategy to smooth the expected long-term return Q value, avoiding over-estimation of the Q value by learning two Q value functions in the Critic network, and delaying policy updates to obtain Q value function estimates with smaller variance;

[0007] (2) Optimize offloading performance using ES-DRL, a distributed task offloading method based on experience-sharing deep reinforcement learning: To improve sample diversity and sample efficiency, ES-DRL, a distributed task offloading method based on experience-sharing deep reinforcement learning, adopts an experience-sharing distributed learning architecture with multiple serverless computing environments. DRL agents are deployed in each environment for task offloading decisions and environment interactions. The agents in each environment upload their experience data to the cloud replay cache for sharing, and randomly extract batches of experience data from it for learning. A population-guided strategy search method is designed. By sharing the best individual information of the population, the population evolutionary learning is guided, and the individual network is optimized to improve the convergence speed of the DRL agent and reduce the exploration cost.

[0008] Compared with the prior art, the present invention models the task offloading performance optimization problem as an MDP process in serverless edge computing, improves the DRL algorithm based on the Actor-Critic framework, and applies it to the solution of MDP. Then, a distributed learning architecture with experience sharing is proposed to improve sample diversity and sample efficiency and reduce exploration costs. A population-guided strategy search method is introduced to accelerate the convergence of DRL agents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic diagram of the overall architecture of ES-DRL in a serverless edge task offloading method based on experience sharing deep reinforcement learning in an embodiment of the present invention;

[0010] Figure 2 It is an Actor-Critic framework diagram of an intelligent agent in ES-DRL in a serverless edge task offloading method based on experience sharing deep reinforcement learning in an embodiment of the present invention;

[0011] Figure 3It is a schematic diagram of distributed learning and population-guided strategy search in a serverless edge task offloading method based on experience-sharing deep reinforcement learning in an embodiment of the present invention;

[0012] Figure 4 It is a diagram of the execution process of ES-DRL in a serverless edge task offloading method based on experience sharing deep reinforcement learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0014] In the serverless edge computing model that combines serverless computing and edge computing, resource-constrained edge servers find it difficult to support all task offloading requests within a unit time. Once overloaded, performance degradation will occur. Deep reinforcement learning combines deep learning and reinforcement learning, solving the problem that traditional reinforcement learning cannot be applied to high-dimensional state space and action space.

[0015] Figure 1 Schematic diagram of the overall architecture of ES-DRL in a serverless edge task offloading method based on experience sharing deep reinforcement learning in an embodiment of the present invention; Figure 1 As shown, the present invention provides a serverless edge task offloading method based on experience sharing deep reinforcement learning, comprising:

[0016] (1) ES-DRL, a distributed task offloading method based on experience sharing deep reinforcement learning, uses deep reinforcement learning of the improved Actor-Critic framework to solve the task offloading optimization problem: in the stateful and stateless combined execution model scenario of serverless edge computing, the multi-task offloading problem is transformed into a joint optimization problem, and the performance optimization problem of the task offloading decision process is modeled as a Markov decision process MDP; taking advantage of the advantages of DRL, the DRL algorithm of the improved Actor-Critic framework is used to solve the problem, which includes adding policy noise to the output action of the target Actor strategy to smooth the expected long-term return Q value, avoiding over-estimation of the Q value by learning two Q value functions in the Critic network, and delaying policy updates to obtain Q value function estimates with smaller variance;

[0017] (2) Optimize offloading performance using ES-DRL, a distributed task offloading method based on experience-sharing deep reinforcement learning: To improve sample diversity and sample efficiency, ES-DRL, a distributed task offloading method based on experience-sharing deep reinforcement learning, adopts an experience-sharing distributed learning architecture with multiple serverless computing environments. DRL agents are deployed in each environment for task offloading decisions and environment interactions. The agents in each environment upload their experience data to the cloud replay cache for sharing, and randomly extract batches of experience data from it for learning. A population-guided strategy search method is designed to guide population evolutionary learning by sharing the best individual information of the population, and optimize the individual network to improve the convergence speed of the DRL agent and reduce the exploration cost.

[0018] Figure 2 is an Actor-Critic framework diagram of an intelligent agent in ES-DRL in a serverless edge task offloading method based on experience sharing deep reinforcement learning in an embodiment of the present invention; Figure 2 As shown, the DRL algorithm of the improved Actor-Critic framework is used to solve step (1), which specifically includes:

[0019] (1.1) Modeling of task offloading optimization problem based on MDP: In real environment, it is necessary to make offloading decisions for task requests of multiple IoT devices at the same time. Therefore, the multi-task offloading problem is transformed into a joint optimization problem to minimize the system delay cost; the delay cost is defined as the sum of the processing delay costs of all tasks as shown in formula (1):

[0020]

[0021] Among them, a i ={0,1}, 0 means task o i Processed in IoT local device, 1 represents task o i Offload to EFaaS for processing; L l represents the delay of task offloading to IoT devices for processing, L e represents the delay of task offloading to EFaaS processing; in edge computing, in order to achieve long-term performance optimization in the study of DRL-based task offloading methods, the task offloading optimization problem is usually modeled as an MDP, which discretizes time into multiple time periods, called time steps. In an MDP, a policy π(A t |S t ) in the state Take an action Interact with the environment, and the environment returns a reward R t and the next state S t+1, and define the MDP state, action, and reward of the serverless edge computing (SEC) environment system as follows:

[0022] 1) State: Define the system state S = {F, G, H, U, q e},in Represents a collection of task information. represents the state information set of IoT devices, H represents the set of channel gain and background noise variance, Represents the computing capacity set q of EFaaS function instances e Indicates task o i The queue delay in the EFaaS task execution queue, q i Indicates task o i The queue delay in the local task execution queue, u i For IoT devices i Or process tasks in EFaaS i The computational power of the function instance;

[0023] 2) Action: Based on the system status The task offloading decisions made are each a binary choice, and the action space is defined as a i ∈{0,1}, where 0 means the task is executed on the IoT device and 1 means the task is offloaded to EFaaS for execution;

[0024] 3) Reward: To minimize the delay cost of the SEC system, the reward is defined as the negative value of the system cost, i.e. -L sys , reflecting taking action A at time step t t system performance when

[0025] (1.2) Using the DRL algorithm of the improved Actor-Critic framework to solve the joint optimization problem of task offloading: ES-DRL mainly includes two parts, such as Figure 1As shown, a part is deployed in the edge layer, which includes three components: execution service, agent, and remote service; Execution Service is mainly responsible for communicating with IoT devices and distributing tasks offloaded to EFaaS to corresponding function (container) instances, corresponding to ① and ⑤ in the figure; Agent (DRL agent) is a deep reinforcement learning network, which is used to approximate reinforcement learning strategies. The main function is to make offloading strategies for tasks generated by IoT devices based on the status information of the serverless edge computing environment, including task information, IoT device status information, EFaaS status information, and network status information, corresponding to ② in the figure. Remote Service is mainly responsible for communicating with the main controller of the cloud center, including uploading and pulling data to train the local Agent network, corresponding to ③ and ④ in the figure;

[0026] The other part is deployed in the main controller of the cloud layer, which is mainly responsible for data storage and updating of relevant parameters, and pushes data to each edge environment for DRL agent learning. When the local Agent network in EFaaS is optimized a certain number of times, the relevant information will be pushed to the main controller, which will update the corresponding parameters and return to the local network, so that each local network can learn the optimal offloading strategy. In the ES-DRL architecture, each DRL agent is deployed to EFaaS in different environments and supports task offloading. ES-DRL can adapt to distributed serverless edge computing environments.

[0027] ES-DRL adopts the DRL algorithm based on deep deterministic policy gradient. The network architecture of the DRL agent is as follows: Figure 2 As shown in Figure 1, it consists of two parts: the Actor network and the Critic network. The Actor network includes an online Actor network and a target Actor network, and the Critic network includes two online Critic networks and two target Critic networks. The Actor network is responsible for the system state S at time step t. t As input and output task offloading action A t , and the Critic network is responsible for evaluating the quality of the actions selected by the Actor. System state S t Contains information about IoT devices, EFaaS, networks, and tasks. Based on the current strategy π φ and system status S t , Actor network output unloading strategy A t Contains offloading decisions for multiple tasks.

[0028] In the Actor-Critic framework, to encourage DRL agent exploration, the Actor outputs the task offloading action A t Then add exploration noise, which uses Gaussian noise; the noise action A after adding exploration noise t Acting on the SEC environment, the environment will feedback the immediate reward R to the DRL agent t and the next system state S t+1 , where the immediate reward R t Equal to the negative system delay cost value, that is, -L sys ; After one interaction with the environment, an experience data tuple (S t ,A t ,R t ,S t+1 ) and used for subsequent network parameter updates; smoothly integrate the target strategy into the Actor-Critic framework; add strategy noise to the output action of the target Actor strategy and smoothly calculate the Q value to avoid overfitting, as shown in formula (2):

[0029]

[0030] The Critic network uses state-action pairs (S t ,A t ) as input, and outputs the corresponding expected long-term return, i.e., Q value, expressed as formula (3):

[0031] Q π (S t ,A t )=E[R t |S t ,A t ] (3)

[0032] In the edge computing environment, the complete trajectory is basically unreachable. A one-step time difference (TD) is used to approximate the target Q value and calculate it based on the Bellman equation. The overestimation of the Q value is avoided by learning two Q value functions. DQN (Deep Q-Network) uses the maximum Q value as the target Q value. When the deep learning neural network is used to approximate the reinforcement learning strategy, the max operation in the Q network algorithm has the problem of overestimation of the Q value. When ES-DRL updates the Critic network, it uses the minimum Q value as the target Q value, as shown in formula (4):

[0033]

[0034] The mean square error loss is used as the loss function of the Critic network, and the network parameters are updated by minimizing the loss function, as calculated in formula (5):

[0035]

[0036] Where B is the batch size of experience data used for DRL agent training, θ j represents the parameters of the online Critic network; the Actor network maximizes the expectation of the total discounted reward and uses the Q function to approximate the total discounted return, so the objective function of the Actor network is formula (6):

[0037]

[0038] Where φ represents the online actor network parameters; the actor policy network parameters are updated based on the policy gradient of formula (6); the parameter update process of the actor network and the critic network requires multiple rounds of training; the DRL algorithm of ES-DRL uses experience replay technology to reuse historical data to improve sample efficiency and empower the learning ability of the DRL agent, making up for the low sample efficiency of the online policy algorithm; in each round of training, a small batch of experience data is randomly extracted to update the network parameters;

[0039] In order to make the online Actor network reach a convergence state, it is necessary to update the gradient in multiple steps. The target network is used to provide a more stable update target. The target network is updated by exponential smoothing method, as shown in equations (7) and (8), to improve learning stability.

[0040] θ j ′←τθ j +(1-τ)θ j ′,τ<<1,j=1,2 (7)

[0041] φ′←τφ+(1-τ)φ′,τ<<1,j=1,2 (8)

[0042] Where τ is the soft update factor, θ′ j represents the target Critic network parameters, φ ′ is the target actor network parameter; and ES-DRL adopts delayed policy update, which reduces the update frequency of the policy network, that is, the actor policy network is updated only after the critic network is updated v times. The basic steps of the DRL algorithm of the improved actor-critic framework are as follows:

[0043] ① The state St is passed to the agent, and the agent outputs the unloading action At;

[0044] ② The environment gives the agent immediate reward R t , and feedback the next system state St+1, and upload the experience data to the shared playback cache located in the cloud center master controller middle;

[0045] ③ Update the online critic network parameters θ j ;

[0046] ④ After the network Critic network is updated v times, update the Actor network parameters φ and φ′ and update the target Critic network parameters θ j ′.

[0047] The solution to the task offloading optimization problem using deep reinforcement learning with the improved Actor-Critic framework in step (2) specifically includes:

[0048] (2.1) In the serverless edge computing environment, there are still some problems that hinder the application of DRL intelligent algorithms. First, deep reinforcement learning intelligent algorithms require a large amount of diverse experience data to ensure learning performance. A single DRL agent interacts with the local environment to obtain data, resulting in insufficient data diversity, low sample efficiency, and slow convergence. In addition, in a real edge computing environment, the cost of DRL trial and error is associated with real monetary costs. The trial and error and exploration costs of obtaining experience data in a single environment are too high.

[0049] Therefore, the ES-DRL method adopts an experience-sharing distributed learning architecture, where multiple SEC environments share experience data. Figure 3 is a schematic diagram of distributed learning and population-guided strategy search in a serverless edge task offloading method based on experience-sharing deep reinforcement learning in an embodiment of the present invention; Figure 3 As shown, each SEC environment Env p Deploy a DRL agent, each agent has an Actor network and a Critic network, p∈{1,2,…,P}, p represents the pth agent; at time step t, agent g p First, based on the current state in each environment Generate task offloading actions The agent will Immediate rewards for acting on the SEC environment and receiving feedback from the environment and a state in the environment The agent will experience data Upload to the shared playback cache of the cloud center master controller Finally, each agent is stored in Randomly extract batches of experience data of size B to train the local agent;

[0050] (2.2) Strategy search method based on population guidance: Figure 3As shown in the figure, in order to prevent the agent's search area from collapsing to a point, avoiding falling into local optimality and slow convergence, ES-DRL constructs an enhanced loss function for Actor network update, integrates the best agent information, guides population evolutionary learning, and expands the overall search area of ​​multiple agents; multiple agents under the SEC environment Env are formed into a population, and M rounds are called a generation of the population; the basic steps of the strategy search method based on population guidance are as follows:

[0051] ① Each agent explores in the local environment and uploads the experience data to the shared playback cache located in the cloud center master controller middle;

[0052] ② Each intelligent agent starts from Randomly extract batch data to optimize the local agent network;

[0053] ③After a generation, the agent in each environment uploads the most recent E r Average reward per round, distance information, and the best individual Actor strategy network parameter φ b ;

[0054] ④ In the cloud center controller, select the best individual according to the population fitness and update the parameter β;

[0055] ⑤ Set β and φ b Push it to the environment of other individuals in the population except the best individual to guide the intelligent individuals to carry out evolutionary learning for the next generation.

[0056] use to represent the population, where P represents the size of the population and p∈P represents the index of the individual. represents the network parameters of a DRL agent, and k represents the generation index of the population evolution. In the ES-DRL method, the Actor network and Critic network of each individual h have the same initialization parameters. The fitness of the population is defined as the most recent E of each DRL agent in the training process of each generation. r After the kth generation of evolution, the DRL agent with the largest fitness is selected from the population G. As the best individual, and use b to represent the index of this individual. In the k+1 generation, for the pth individual Keep the Critic network parameter update method unchanged, the Actor network parameter φ b The loss function is set as follows:

[0057]

[0058] Among them, L(φp ) is the original loss function of the kth generation of the pth agent. {.} is an indicator function. If p≠b, the value is 1, otherwise it is 0, which means that the loss function of the Actor strategy of all agents in the population except the best individual is the enhanced version. β is an adaptive weight factor, whose value is greater than 0. There are two Actor strategies π p and π b The distance is calculated by using the mean square error, as shown in formula (18):

[0059]

[0060] Therefore, the original equation (6) for Actor network update can be rewritten as equation (19).

[0061]

[0062] Represents the agent's Actor strategy at the end of the previous generation of training, represents the Actor strategy of the best agent selected at the end of the previous generation of training, represents the current agent Actor strategy. The adaptive weight factor β is updated after each generation of the population and used in the next generation of population evolution. The calculation is shown in formula (20), where The calculation of is shown in formula (21), which is used to estimate P-1 and The average distance of The calculation of is shown in formula (22), which is used to estimate and The average distance between ρ and d indicates the speed of change of each agent. min is a predefined value.

[0063]

[0064] As the intelligent agents in the population evolve and update, the weight factor tends to be stable. exist In the process of population evolution, all individuals are near the best individual The search is performed within the radius of dsearch, so that the individuals converge more quickly. When trapped in a local optimum, will be very small, thus reducing the search range. min Escape from local optimum.

[0065] Figure 41 is an execution process diagram of ES-DRL in a serverless edge task offloading method based on experience sharing deep reinforcement learning in an embodiment of the present invention. The implementation of the distributed task offloading method ES-DRL based on experience sharing deep reinforcement learning is as follows: Figure 4 As shown. First, the parameters of the Actor network and Critic network and the adaptive weight parameter β of each agent in the population are initialized. During the population evolution process, each agent makes a task offloading strategy based on the current state of the environment. And the offloading strategy is applied to the environment, and the agent receives the reward feedback from the environment and the changed state of the environment. Then, the experience data is uploaded to the replay cache of the cloud center master controller. After a certain amount of data is collected, batch experience data is extracted from the replay cache for the agent's network learning. After each generation, the distance information, Actor strategy parameters, and average reward are uploaded to the master controller of the cloud center. The master controller of the cloud center selects the best agent individual, updates the parameters, and pushes the best individual information to guide the learning of other agents.

Claims

1. A serverless edge task offloading method based on experience sharing deep reinforcement learning, characterized in that: The steps include: (1) Distributed task offloading method based on experience sharing deep reinforcement learning ES-DRL uses deep reinforcement learning of the improved Actor-Critic framework to solve the task offloading optimization problem: in the stateful and stateless execution model scenario of serverless edge computing, the multi-task offloading problem is transformed into a joint optimization problem, and the performance optimization problem of the task offloading decision process is modeled as a Markov decision process MDP; taking advantage of the advantages of deep reinforcement learning DRL, the DRL algorithm of the improved Actor-Critic framework is used to solve the problem by adding policy noise to the output action of the target Actor strategy to smooth the expected long-term return Q value, avoiding over-estimation of the Q value by learning two Q value functions in the Critic network, and delaying policy updates to obtain Q value function estimates with small variance; In the step (1), in the stateful and stateless combined execution model scenario of serverless edge computing, the multi-task offloading problem is converted into a joint optimization problem, and the performance optimization problem of the task offloading decision process is modeled as a Markov decision process, specifically including: Modeling of task offloading optimization problem based on MDP: The task offloading problem is transformed into a joint optimization problem; the delay cost is defined as the sum of the processing delay costs of all tasks. Refer to formula (1): Among them, a i ={0,1}, 0 means task o i In the Internet of Things (IoT) local device processing, 1 represents task o i Offload to the EFaaS processing in the edge server; L l represents the delay of task offloading to IoT devices for processing, L e represents the delay of task offloading to EFaaS processing; time is discretized into multiple time periods, called time steps In an MDP, a policy π(A t |S t ) in the state Take an action Interact with the environment, and the environment returns a reward R t and the next state S t+1 , and define the MDP state, action and reward of the SEC environment system as follows: 1) State: Define the system state S = {F, G, H, U, q e },in Represents a collection of task information. represents the state information set of IoT devices, H represents the set of channel gain and background noise variance, represents the computing capability set of EFaaS function instances, q e Indicates task o i The queue delay in the EFaaS task execution queue, q i Indicates task o i The queue delay in the local task execution queue, u i For IoT devices i Or process tasks in EFaaS i The computational power of the function instance; 2) Action: Based on the system status The task offloading decisions made are each a binary choice, and the action space is defined as a i ∈{0,1}, where 0 means the task is executed on the IoT device and 1 means the task is offloaded to EFaaS for execution; 3) Reward: To minimize the delay cost of the SEC system, the reward is defined as the negative value of the system cost, i.e. -L sys , reflecting taking action A at time step t t System performance when In the step (1), the advantages of DRL are utilized to solve the problem by adding policy noise to the output action of the target Actor policy to smooth the expected long-term return Q value, avoiding overestimation of the Q value by learning two Q value functions in the Critic network, and delaying policy updates to obtain a Q value function estimate with a small variance using the DRL algorithm of the improved Actor-Critic framework, which specifically includes: The DRL algorithm based on the improved Actor-Critic framework is used to solve the joint optimization problem of task offloading: In the Actor-Critic framework, to encourage DRL agent exploration, the Actor outputs the task offloading action A. t Then add exploration noise, which uses Gaussian noise; the noise action A after adding exploration noise t Acting on the SEC environment, the environment will feedback the immediate reward R to the DRL agent t and the next system state S t+1 , where the immediate reward R t Equal to the negative system delay cost value, that is, -L sys ; After one interaction with the environment, an experience data tuple (S t ,A t ,R t ,S t+1 ) and used for subsequent network parameter updates; smoothly integrate the target strategy into the Actor-Critic framework; add strategy noise to the output action of the target Actor strategy and smoothly calculate the Q value to avoid overfitting, refer to formula (2): The Critic network uses state-action pairs (S t ,A t ) as input, and outputs the corresponding expected long-term return, i.e., Q value, expressed as reference formula (3): Q π (S t ,A t ) = E[R t |S t ,A t ] (3) In the edge computing environment, the complete trajectory is unreachable. One-step time difference square TD is used to approximate the target Q value and calculate it based on the Bellman equation. The overestimation of the Q value is avoided by learning two Q value functions. DQN (Deep Q-Network) uses the maximum Q value as the target Q value. When the deep learning neural network is used to approximate the reinforcement learning strategy, the max operation in the Q network algorithm has the problem of overestimation of the Q value. When ES-DRL updates the Critic network, it uses the minimum Q value as the target Q value. Refer to formula (4): The mean square error loss is used as the loss function of the Critic network, and the network parameters are updated by minimizing the loss function. The calculation reference formula (5) is: Where B is the batch size of experience data used for DRL agent training, θ j represents the parameters of the online Critic network; the Actor network maximizes the expectation of the total discounted reward and uses the Q function to approximate the total discounted return, so the objective function of the Actor network is formula (6): Where φ represents the online actor network parameters; the actor policy network parameters are updated based on the policy gradient of formula (6); the parameter update process of the actor network and the critic network requires multiple rounds of training; the DRL algorithm of ES-DRL uses experience replay technology to reuse historical data to improve sample efficiency and empower the learning ability of the DRL agent, making up for the low sample efficiency of the online policy algorithm; in each round of training, a small batch of experience data is randomly extracted to update the network parameters; (2) Optimizing offloading performance using ES-DRL, a distributed task offloading method based on experience-sharing deep reinforcement learning: To improve sample diversity and sample efficiency, ES-DRL, a distributed task offloading method based on experience-sharing deep reinforcement learning, adopts an experience-sharing distributed learning architecture with multiple serverless computing environments. DRL agents are deployed in each environment for task offloading decisions and environment interactions. The agents in each environment upload experience data to the cloud replay cache for sharing, and randomly extract batches of experience data from it for learning and designing a population-guided strategy search method. By sharing the best individual information of the population, the population evolutionary learning is guided, and the individual network is optimized to improve the convergence speed of the DRL agent and reduce the exploration cost. In step (2), in order to improve sample diversity and sample efficiency, the distributed task offloading method ES-DRL based on experience sharing deep reinforcement learning adopts an experience sharing distributed learning architecture, which has multiple serverless computing environments. DRL agents are deployed in each environment for task offloading decision-making and environment interaction, specifically including: The ES-DRL method adopts a distributed learning architecture with experience sharing, where multiple SEC environments share experience data; each SEC environment Env p Deploy a DRL agent, each agent has an Actor network and a Critic network, p∈{1,2,…,P}, p represents the pth agent; at time step t, agent g p First, based on the current state in their respective environments Generate task offloading actions The agent will Immediate rewards for acting on the SEC environment and receiving feedback from the environment and a state in the environment The agent will experience data Upload to the shared playback cache of the cloud center master controller Finally, each agent is stored in Randomly extract batches of experience data of size B to train the local agent; In the step (2), the strategy search method based on population guidance specifically includes: ES-DRL constructs an enhanced loss function for Actor network updates, integrates the best agent information, guides population evolutionary learning, and expands the overall search area of ​​multiple agents. Multiple agents under the SEC environment Env are grouped into a population, and M rounds are called a generation of the population. The basic steps of the population-guided strategy search method are as follows: ① Each agent explores in the local environment and uploads the experience data to the shared playback cache located in the cloud center master controller middle; ② Each intelligent agent starts from Randomly extract batch data to optimize the local agent network; ③After a generation, the agent in each environment uploads the average reward, distance information, and the best individual Actor strategy network parameter φ of the last E rounds b ; ④ In the cloud center controller, select the best individual according to the population fitness and update the parameter β; ⑤ Set β and φ b Push it to the environment of other individuals in the population except the best individual to guide the intelligent individuals to carry out evolutionary learning for the next generation.

2. The serverless edge task offloading method based on experience sharing deep reinforcement learning as claimed in claim 1, characterized in that: In order to make the online Actor network reach a convergence state, it is necessary to update the gradient in multiple steps. The target network is used to provide a stable update target. The target network is updated by exponential smoothing method, as shown in formula (7) and formula (8), to improve learning stability. θ′ j ←tth j +(1-τ)θ′ j ,τ<<1,j=1,2 (7) φ′←τφ+(1-τ)φ′,τ<<1,j=1,2 (8) Where τ is the soft update factor, θ′ j represents the target Critic network parameters, φ′ is the target Actor network parameters; and ES-DRL adopts delayed policy update, which reduces the update frequency of the policy network, that is, the Actor policy network is updated only after the Critic network is updated v times; the basic steps of the DRL algorithm of the improved Actor-Critic framework are as follows: ① The state St is passed to the agent, and the agent outputs the unloading action At; ② The environment gives the agent immediate reward R t , and feedback the next system state S t+1 , and upload the experience data to the shared playback cache located in the cloud center master controller middle; ③ Update the online critic network parameters θ j ; ④ After the network Critic network is updated v times, update the Actor network parameters φ and φ′ and update the target Critic network parameters θ′ j .

Images