Micro-service fault-tolerant scheduling method based on reinforcement learning in computing power network

Abstract

The invention discloses a micro-service fault-tolerant scheduling method based on reinforcement learning in a computing power network environment. Comprising the steps of analyzing task fault-tolerant scheduling constraints based on a PB (Primary-Backup) model and designing a heuristic scheduling algorithm FAWS for solving the problem of network host and link equipment faults existing in a network environment, so that the fault tolerance of a system is improved, and tasks are ensured to be completed on time. In order to improve the performance and adaptability of the model, an offline-to-online task fault-tolerant scheduling framework based on reinforcement learning is designed. For the cold start problem in the model training process, a discrete soft action-critic reinforcement learning model is trained offline by using a task running log of the FAWS. In order to solve the problem of inconsistent data distribution among servers in an edge environment, when a DRL model is deployed to an edge server, the DRL model is finely tuned online through a method of jointly using GAE and PPO, so that the decision-making ability of the model is further improved, and an optimal scheduling decision is obtained.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a fault-tolerant scheduling algorithm for a microservice workflow based on reinforcement learning in an edge computing environment with edge server and link device failures, in the face of complex task requests and heterogeneous edge server resources. Background Art

[0002] Computing First Network refers to connecting computing resources scattered in different locations through a network to form a unified and coordinated computing resource pool to provide powerful computing power and flexible resource scheduling. The core goal of the computing network is to efficiently integrate and utilize distributed computing resources to meet the needs of large-scale computing tasks. In recent years, with the popularization of the Internet of Things (IoT) and the rapid development of communication technology, new computing and communication technologies have promoted the emergence of more and more innovative mobile applications and services, such as augmented reality, virtual reality, autonomous driving, and mobile healthcare. The computing network has gradually pushed computing power to the edge. These mobile applications and smart devices have dramatically increased their demand for computing and storage resources traditionally provided by cloud servers, and the amount of data they generate has seen an unprecedented growth. At the same time, they are also faced with the challenge of operating efficiently under strict latency and capacity constraints. The traditional centralized cloud computing architecture cannot fully solve these problems. Multi-access Edge Computing (MEC) in the computing network is a promising computing paradigm to solve such problems. The basic principle of MEC is to extend cloud computing capabilities to network edge hosts close to users, which can significantly alleviate network congestion and reduce service delays because edge computing nodes are close to users and have sufficient computing power. In addition, through collaboration between edge nodes, resource utilization can be improved and user quality of experience (QoE) can be improved.

[0003] Most of the basic operating units (such as edge servers and transmission link devices) in multi-access edge computing scenarios are exposed to the natural environment. Due to the dispersed physical locations between these nodes, it is difficult for edge computing clusters to provide the same level of maintenance and management as cloud computing centers. These components are prone to transient failures caused by electromagnetic interference or cosmic radiation. Such failures are short-lived and generally do not damage the hardware. However, when transient failures or software crashes occur, they will bring degraded service experience to users. Most existing work on task offloading under the edge computing paradigm usually considers minimizing the latency and energy consumption of task execution. They ignore the security issues existing in the various operating units (such as edge servers and transmission link devices) in the edge network system, especially the research on fault perception of link devices (such as switches and repeaters). In addition, most traditional heuristic, meta-heuristic and model-based task offloading methods are not suitable for edge environments where resource availability is constantly changing. When the environment changes, these mathematical models may need to be updated accordingly. Algorithms based on reinforcement learning can solve the above problems to a certain extent, however, this method of online learning at the edge suffers from the cold start problem and the inconsistent distribution of edge data, and has poor adaptability to unexpected disturbances or unseen situations (i.e., new environments), such as changes in applications, number of tasks, or data rates. Therefore, their sample efficiency is very low and they require complete retraining to learn updated strategies for new environments, so they are very time-consuming.

[0004] In summary, a safe and effective fault-tolerant mechanism is urgently needed to deal with the problem of instantaneous device failures encountered in task offloading in computing network scenarios to ensure the user's service experience. In addition, it is necessary to design a task scheduling framework based on reinforcement learning to adapt to the ever-changing edge environment and solve the cold start problem and distribution transfer problem during training. Summary of the invention

[0005] In view of the problems existing in the above-mentioned background technology, the present invention aims to minimize the task calculation delay, while taking into account the failures of network hosts and devices, and designs a microservice fault-aware safety scheduling method based on reinforcement learning in multi-access edge computing. Then, on this basis, a reinforcement learning task offloading framework from offline to online considering fault tolerance is designed, which is suitable for edge computing systems with multiple edge servers to solve the problem of cold start of model training and the problem of data distribution transfer at the edge. Specifically, the following steps are included:

[0006] Step 1: Most large computationally intensive tasks can be modeled as directed acyclic graphs (DAGs). Considering the failure of edge servers and transmission links in the system, constraints on fault-tolerant scheduling of tasks are obtained. Combined with the PB (Primary-Backup) model, the objective function of task offloading is established according to the task calculation delay, and a fault-tolerant FAWS heuristic scheduling algorithm is designed to obtain the offloading strategy of user tasks and record the log information of each task scheduling. The specific steps are as follows:

[0007] 1) A microservice-based mobile application can be modeled as a directed acyclic graph DAG, assuming that an edge server s is reached at time slot t i The DAG of s follows the Bernoulli distribution. Consider that the DAG constructed by the mobile application arriving at the edge server s in time slot t is W = (R, E), R = {n1,n2,...n i ,...,n n} represents the task set, E={e i,j |e i,j =(n i ,n j )} represents the dependency relationship between tasks, where DP(n i ) represents task n i The set of direct predecessor tasks, the corresponding DS(n i ) represents task n i The direct successor task set of n. Define the task set whose predecessor task set is empty as the entry task set n entry , the task set whose successor task set is empty is the exit task set n exit Assume that the task scheduling plan A = {a1, a2, ..., a n}, where a i Represents task n i The failure in the system can be modeled as a Poisson distribution, and the PB model is used to solve the problem of server and host failure. The PB model implements fault tolerance by assigning a primary copy and a backup copy to each task, and scheduling the primary and backup copies to edge servers in different subnets. i The master copy is The backup copy is For ease of understanding, use Represents any replica task.

[0008] 2) Use UT i Y (t) indicates that the task will be Upload from task initiation server a0 to server a i The upload delay is calculated as follows:

[0009]

[0010] Among them, the parameters represents the task in time slot t TP(a0) represents the data transmission rate of edge server a0. represents a0, a i Obviously, if the task is scheduled to run on the edge server that initiated the task, then UT i Y (t) = 0. Using ET i Y (t) represents the task in time slot t On edge server a i The calculation delay on is calculated as follows:

[0011]

[0012] Among them, the parameters represents the t-time slot task The amount of data for the task being executed. i ) represents edge server a i The processing capacity is expressed in CPU cycles per second; ρ represents the number of CPU cycles required to process 1 bit. Indicates that the task will be The data of the operation results is transmitted to its successor task The transmission delay caused by this is calculated as follows:

[0013]

[0014] Among them, parameter a i For the task The edge server where the j For the task The edge server, td ij Indicates the amount of data transferred between tasks. TP(a i ) means a i data transfer rate. Indicates a i , a j If the task Its subsequent tasks Run or task on the same edge server For export tasks, the transmission delay is 0. Representation Task The start time of execution in time slot t. Calculated by the following formula:

[0015]

[0016] The parameter AT is the arrival time of the DAG task, and the arrival time of the first DAG task is defined as 0. Ava() is the idle time of the edge server. Representation Task The time when the execution is completed in time slot t is calculated as follows:

[0017]

[0018] 3) Definition The edge server where the primary copy is located. The edge server where the backup copy is located. The subnet where the edge server to which the primary replica is uninstalled is located. The subnet where the edge server where the backup copy is unloaded is located. The dependency relationship of the real workflow tasks arriving at time slice t is analyzed, and the task scheduling location constraint based on the PB model is expressed as follows:

[0019] C1:

[0020] C2:

[0021] Constraint C1 represents the currently scheduled task n i Backup copy of Cannot be compared with the primary copy In the same subnet; constraint C2 means task n i Backup copy of Cannot be used with n k |n k ∈DP(n i )(Direct Predecessor Task) In the same subnet.

[0022] According to the analysis of the above steps, task n i Master copy Execution can only start after receiving the output data of the previous task (that is, under normal execution). Therefore, according to formula (4), the startup time of the master copy is expressed as follows:

[0023] C3:

[0024] Similarly, we need to consider task n i Master copy Backup copy of the previous task In order to minimize the delay, the constraint between the two is expressed as follows:

[0025] C4:

[0026] In the case of formula (9), Implementation and To avoid the failure of subsequent task execution, set any backup copy according to formula (4) The start-up time is expressed as follows:

[0027] C5:

[0028] Based on the analysis in step 1, by considering the factors of edge server and network device failure in the edge environment, the goal is to minimize the total computing delay of the DAG task, so task n i The final completion time is calculated as follows:

[0029]

[0030] Therefore, the multi-task scheduling optimization problem in the edge service system is established:

[0031]

[0032] stC1,C2,C3,C4,C5

[0033] 4) First, the heuristic algorithm FAWS for DAG task scheduling is implemented according to the greedy idea. Since there are dependencies between tasks, it is necessary to convert DAG into a sequence to determine the unloading order of each task. First, the tasks are sorted in ascending order according to the rank value of each task. The rank value is calculated as follows:

[0034]

[0035] Among them, rank() represents the rank value of a task, a i Represents any edge node, and N represents the number of edge nodes. By calculating the rank value of each task in the DAG, the tasks are input in descending order of rank value to obtain the DAG scheduling priority queue Q. In this way, the unloading order of the tasks is obtained. Then, under the fault tolerance constraint, with the goal of minimizing the overall completion time of task execution, according to the order of the scheduling priority queue Q, the edge server with the earliest completion time is selected as the unloading target for the primary and backup copies of each task according to the greedy idea.

[0036] Step 2: Based on the task scheduling log information obtained by running the fault-tolerant scheduling algorithm, integrate the transitions data that can be used for reinforcement learning training. Use the transitions data to train a discrete softactor-critic (SAC) model offline. Deploy the trained model to each edge node for online decision-making. The specific steps are as follows:

[0037] 1) First, we need to analyze the task scheduling log information. Let the data set sampled from the buffer be B, recorded as Update the model.

[0038] 2) Use transitions data to train a discrete SAC model offline. In the present invention, two Q networks are designed to reduce the estimation bias of the value function. Let φ1 and φ2 be the parameters of the two Q networks, are the target network parameters of φ1 and φ2 respectively. θ is the actor network parameter. To achieve online fine-tuning, an additional offline critic network is trained with the parameter ω. The soft value and target value are calculated as follows:

[0039]

[0040] Where ɑ is the entropy regularization parameter, which is used to control the importance of entropy, and γ is the discount factor. Due to the distribution shift between the heuristic method strategy for collecting data and the offline learning strategy, the CQL regularizer is used to optimize the Q network. The q network parameters are updated by minimizing the following equation using gradient descent, λ c is the CQL weight. Calculated as follows:

[0041]

[0042] In order to facilitate subsequent online fine-tuning, an additional critic network is trained. The critic network parameters are updated by minimizing the MSE using gradient descent, and the actor network parameters are updated by maximizing the MSE using gradient descent. The calculation is as follows:

[0043]

[0044] 3) Finally, the trained model is deployed to each edge server;

[0045] Step 3: The edge server regularly collects real-time data of task offloading, combines the PPO-Clip algorithm to fine-tune the DRL model online, and finally uses the fine-tuned model to make real-time offloading decisions for tasks to obtain the optimal choice. The specific steps are as follows:

[0046] 1) During offline training, an additional critic network is trained. By retaining the actor-critic network trained offline and migrating it to the online fine-tuning architecture, the dataset C is sampled in batches from the online data buffer pool. GAE is used to balance the bias of value estimation and the variance of returns, and the overestimation problem is solved by designing the advantage function. The update of GAE is calculated as follows:

[0047]

[0048] Similar to the offline training update process, the critic network parameters are updated by minimizing the MSE through gradient ascent, calculated as follows:

[0049]

[0050] 2) The PPO-Clip algorithm is used to update the actor network. The PPO objective function consists of two parts: the clipping objective function and the entropy regularization term. By limiting the speed of network parameter update, the actor network parameters are updated by maximizing the objective function using gradient ascent, which is calculated as follows:

[0051]

[0052] 3) Send the fine-tuned model to each edge node to make real-time decisions on the primary and backup copies of the task;

[0053] Compared with the prior art, the advantages of this method are:

[0054] 1. Considering the failure problem of basic operating units (especially network equipment) in the edge environment, the characteristics of fault-tolerant dependent task scheduling based on the PB model are analyzed, and the tasks are offloaded to appropriate edge servers to ensure that the tasks are completed on time and improve the fault tolerance of the system.

[0055] 2. Designed an offline-to-online task scheduling framework based on reinforcement learning, which makes offloading decisions for the primary replica and the backup replica respectively. On the basis of achieving fault-tolerant task scheduling, it solves the data distribution problem of model training and improves the robustness of the model to environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the method of the present invention;

[0057] Figure 2 is a system architecture diagram of the present invention;

[0058] Figure 3 is a flow chart of offline training of SAC agent of the present invention;

[0059] Figure 4is the flow chart of online fine-tuning agent of the present invention; DETAILED DESCRIPTION

[0060] The following is attached to the instruction manual Figure 1 The technical solution of the present invention is further described in detail.

[0061] like Figure 1 As shown, a flow chart of a microservice fault-aware safe scheduling method based on reinforcement learning is shown, and the method includes the following steps:

[0062] 4) In step 1, if Figure 2 As shown in the system architecture diagram, there are N edge servers in the system, and N edge servers provide computing services for all mobile users. The link devices in the system (such as switches and routers) divide the system into multiple partitions or subnets. Each switch connects one or more edge nodes to form a subnet. Each edge server in the system is deployed together through an access point (AP). The edge server connects a certain number of user devices, and user tasks access the edge system from the AP. Mobile applications based on microservices can be modeled as a directed acyclic graph DAG. Let DAG be W = (R, E), R = {n1, n2, ... n i ,...,n n} represents the task set, E={e i,j |e i,j =(n i ,n j )} represents the dependency relationship between tasks, where DP(n i ) represents task n i The set of direct predecessor tasks, the corresponding DS(n i ) represents task n i The direct successor task set of n. Define the task set whose predecessor task set is empty as the entry task set n entry , the task set whose successor task set is empty is the exit task set n exit Assume that the task scheduling plan A = {a1, a2, ..., a n}. i Represents task n i The failure in the system can be modeled as a Poisson distribution, and the PB model is used to solve the problem of server and host failure. The PB model implements fault tolerance by assigning a primary copy and a backup copy to each task, and scheduling the primary and backup copies to edge servers in different subnets. i The master copy is The backup copy is For ease of understanding, use Indicates any copy task. Use UT i Y(t) indicates that the task will be Upload from task initiation server a0 to server a i The upload delay is calculated as follows:

[0063]

[0064] Among them, the parameters represents the task in time slot t TP(a0) represents the data transmission rate of edge server a0. represents a0, a i Obviously, if the task is scheduled to run on the edge server that initiated the task, then UT i Y (t) = 0. Using ET i Y (t) indicates task On edge server a i The calculation delay on is calculated as follows:

[0065]

[0066] Among them, the parameters represents the t-time slot task The amount of data for the task being executed. i ) represents edge server a i The processing capacity is expressed in CPU cycles per second; ρ represents the number of CPU cycles required to process 1 bit. Indicates that the task will be The data of the operation results is transmitted to its successor task The transmission delay caused by this is calculated as follows:

[0067]

[0068] Among them, parameter a i For the task The edge server where the j For the task The edge server where the td ij Indicates the amount of data transferred between tasks. TP(a i ) means a i data transfer rate. Indicates a i , a j If the task Its subsequent tasks Run or tasks on the same edge server For export tasks, the transmission delay is 0. Representation Task The start time of execution in time slot t. Calculated by the following formula:

[0069]

[0070] The parameter AT is the arrival time of the DAG task, and the arrival time of the first DAG task is defined as 0. Ava() is the idle time of the edge server. Representation Task The time when the execution is completed in time slot t is calculated as follows:

[0071]

[0072] definition The edge server where the primary copy is located. The edge server where the backup copy is located. The subnet where the edge server to which the primary replica is uninstalled is located. The subnet where the edge server where the backup copy is unloaded is located. The dependency relationship of the real workflow tasks arriving at time slice t is analyzed, and the task scheduling location constraint based on the PB model is expressed as follows:

[0073] C1:

[0074] C2:

[0075] Constraint C1 represents the currently scheduled task n i Backup copy of Cannot be compared with the primary copy In the same subnet; constraint C2 means task n i Backup copy of Cannot be used with n k |n k ∈DP(n i )(Direct Predecessor Task) In the same subnet.

[0076] According to the previous analysis, design task n i Backup copy of Only in the primary copy The execution can only be started after the execution is completed. This is because the probability of frequent system failures is not high, so the probability of needing to start the backup copy to execute the task is not high. Task n i Master copy Execution can only start after receiving the output data of the previous task (that is, under normal execution). Therefore, according to formula (4), the startup time of the master copy is expressed as follows:

[0077] C3:

[0078] Similarly, we need to consider task n i Master copy Backup copy of the previous task In order to minimize the delay, the constraint between the two is expressed as follows:

[0079] C4:

[0080] In the case of formula (9), Implementation and To avoid the failure of subsequent task execution, set any backup copy according to formula (4) The start-up time is expressed as follows:

[0081] C5:

[0082] Based on the analysis in step 1, by considering the factors of edge server and network device failure in the edge environment, the goal is to minimize the total computing delay of the DAG task, so task n i The final completion time is calculated as follows:

[0083]

[0084] Therefore, the multi-task scheduling optimization problem in the edge service system is established:

[0085]

[0086] stC1,C2,C3,C4,C5

[0087] First, the heuristic algorithm FAWS for DAG task scheduling is implemented according to the greedy idea. Since there are dependencies between tasks, it is necessary to convert DAG into a sequence to determine the order of unloading each task. First, the tasks are sorted in ascending order according to the rank value of each task. The rank value is calculated as follows:

[0088]

[0089] Among them, rank() represents the rank value of a task, a i Represents any edge node, and N represents the number of edge nodes. By calculating the rank value of each task in the DAG, the tasks are input in descending order of the rank value to obtain the DAG scheduling priority queue Q. In this way, the unloading order of the tasks is obtained. Then, under the fault tolerance constraint, with the goal of minimizing the overall completion time of task execution, according to the order of the scheduling priority queue Q, the edge node with the earliest completion time is selected as the unloading target for the primary and backup copies of each task according to the greedy idea. If the task meets the fault tolerance constraints, the next step can be performed, otherwise the task will reselect the unloading target. In addition to meeting the fault tolerance avoidance principle, the task must also meet the resource constraints of the edge node. Finally, by capturing the environmental information before and after task scheduling and combining the unloading decision of the DAG task, the elements required for offline training reinforcement learning, such as state, action, next_state, reward, etc. are constructed. And the standardized data is configured as transition and stored in the log Buffer;

[0090] In step 2, data is sampled in batches from the buffer and used to train a discrete soft actor-critic reinforcement learning model. This can effectively solve the problem of inconsistent data distribution in edge DRL model training and improve the stability and sample efficiency of the model. The algorithm flow of offline training of discrete SAC with heuristic data is as follows: Figure 3 As shown, firstly, data is taken out from the buffer in batches, and the relevant data is standardized and normalized. Suppose the data set sampled from the buffer is B, recorded as 1≤i≤|Β| The model is updated. Then two Q networks are used to avoid overestimation. First, the soft value is calculated, then the target Q value is calculated, and finally the Q network is updated. The softvalue and targetvalue are calculated as follows:

[0091]

[0092] Where ɑ is the entropy regularization parameter, which is used to control the importance of entropy, and γ is the discount factor. Due to the distribution shift between the heuristic method strategy for collecting data and the offline learning strategy, the CQL regularizer is used to optimize the Q network. CQL regularizes the Q value function to ensure its conservative estimation to avoid excessive overestimation during the value function learning process, thereby reducing instability and suboptimality of the strategy. The q network parameters are updated by minimizing the following equation using gradient descent, λ c is the CQL weight. Calculated as follows:

[0093]

[0094] In order to facilitate subsequent online fine-tuning, an additional critic network is trained. The critic network parameters are updated by minimizing the MSE using gradient descent, and the actor network parameters are updated by maximizing the MSE using gradient descent. The calculation is as follows:

[0095]

[0096] Finally, the trained model is deployed to each edge server;

[0097] In step 3, each edge server will accumulate a large amount of task operation data during execution. It is necessary to regularly collect these real-time data and fine-tune the DRL model to update the scheduling model so that the model can better adapt to the local data distribution characteristics. The algorithm flow of online training DRL model is as follows: Figure 4 As shown in the figure, during offline training, an additional critic network is trained. The actor-critic network trained offline is retained and migrated to the online fine-tuning architecture. Then the dataset C is sampled in batches from the online data buffer pool. Since there is a distribution transfer problem between offline data and online data during the online fine-tuning model, and the Q network and critic network are pre-trained based on offline data, this will lead to an overestimation of observations not seen in the online stage. GAE is used to balance the bias of value estimation and the variance of returns, and the overestimation problem is solved by designing the advantage function. The update of GAE is calculated as follows:

[0098]

[0099] GAE balances the variance and bias by smoothing the advantage estimates of different step sizes, so that the advantage function can be better estimated and the critic can be made more robust. Similar to the offline training update process, the critic network parameters are updated by minimizing the MSE through gradient ascent. Calculated as follows:

[0100]

[0101] The PPO-Clip algorithm is used to update the actor network. The PPO objective function consists of two parts: the clipping objective function and the entropy regularization term. By limiting the amplitude of the policy update, the policy is prevented from changing too much at each update, ensuring the stability of the algorithm and helping to converge to a better policy. By limiting the speed of network parameter updates, the actor network parameters are updated by maximizing the objective function through gradient ascent. Calculated as follows:

[0102]

[0103] The fine-tuned model is sent to each edge node to make real-time decisions on the primary and backup copies of the task. The real-time decision-making process is as follows: If the scheduled task is the primary copy, the actor network returns (action, prob, val) based on the current input state to obtain the unloading plan for the task scheduling. If the scheduled task is the backup copy, the actor network excludes actions that do not meet the constraints based on the current input state and the fault tolerance requirements, and then selects the optimal action as the unloading plan. The data returned by the scheduling task is combined with the environmental information to form a new transition, which is then input into the buffer for experience playback sampling.

Claims

1. A fault-tolerant scheduling method for microservices based on reinforcement learning in a computing network environment, characterized by: firstly, analyzing the fault-tolerant scheduling constraints based on task dependencies, and designing a heuristic scheduling algorithm FAWS with the goal of minimizing the total delay of task execution. Then, a reinforcement learning-based offline-to-online task scheduling framework is designed to make offloading decisions for the primary replica and the backup replica respectively. The specific steps are as follows: Step 1: Most large computing-intensive tasks can be modeled as directed acyclic graphs (DAGs). Considering the failure of edge servers and transmission links in the system, constraints on task fault-tolerant scheduling are obtained. Combined with the PB (Primary-Backup) model, the objective function of computing task offloading is established according to edge computing delay, and a fault-tolerant FAWS heuristic scheduling algorithm is designed to obtain the offloading strategy of user tasks, and log the scheduling information of each task. 1) In step 1, define DAG as W = (R, E), R = {n1,n2,...n i ,...,n n } represents the task set, E={e i,j |e i,j =(n i ,n j )} represents the dependency relationship between tasks, where Let DP(n i ) represents task n i The set of direct predecessor tasks, the corresponding DS(n i ) represents task n i The direct successor task set of n. Define the task set whose predecessor task set is empty as the entry task set n entry , the task set whose successor task set is empty is the exit task set n exit Assume that the task scheduling plan A = {a1, a2, ..., a n }. i Represents task n i The failure in the system can be modeled as a Poisson distribution, and the PB model is used to solve the problem of server and host failure. The PB model implements fault tolerance by assigning a primary copy and a backup copy to each task, and scheduling the primary and backup copies to edge servers in different subnets. i The master copy is The backup copy is For ease of understanding, use Represents any copy task. Indicates that the task will be Upload from task initiating server a0 to server a i The upload delay is calculated as follows: Among them, the parameters represents the task in time slot t TP(a0) represents the data transmission rate of edge server a0. represents a0, a i Obviously, if the task is scheduled to run on the edge server that initiated the task, then UT i Y (t) = 0. Indicates the task On edge server a i The calculation delay on is calculated as follows: Among them, the parameters represents the t-time slot task The amount of data for the task being executed. i ) represents edge server a i The processing capacity is expressed in CPU cycles per second; ρ represents the number of CPU cycles required to process 1 bit. Indicates that the task will be The data of the operation results is transmitted to its successor task The transmission delay caused by this is calculated as follows: Among them, parameter a i For the task The edge server where the j For the task The edge server, td ij Indicates the amount of data transferred between tasks. TP(a i ) means a i data transfer rate. Indicates a i , a j If the task Its subsequent tasks Run or task on the same edge server For export tasks, the transmission delay is 0. Indicates the task The start time of execution in time slot t. Calculated by the following formula: The parameter AT is the arrival time of the DAG task, and the arrival time of the first DAG task is defined as 0. Ava() is the idle time of the edge server. Indicates the task The time when the execution is completed in time slot t is calculated as follows: Step 2: Based on the task scheduling log information obtained by running the fault-tolerant scheduling algorithm, integrate the transitions data that can be used for reinforcement learning training. Use the transitions data to train a discrete softactor-critic (SAC) model offline. Deploy the trained model to each edge node for online decision-making. Step 3: The edge server regularly collects real-time data of task offloading, combines the PPO-Clip algorithm to fine-tune the DRL model online, and finally uses the fine-tuned model to make real-time offloading decisions for tasks.

2. According to the reinforcement learning-based microservice fault-tolerant scheduling method of claim 1, it is characterized by: definition The edge server where the primary copy is located. The edge server where the backup copy is located. The subnet where the edge server to which the primary replica is uninstalled is located. The subnet where the edge server where the backup copy is unloaded is located. The dependency relationship of the real workflow tasks arriving at time slice t is analyzed, and the task scheduling location constraint based on the PB model is expressed as follows: Constraint C1 represents the currently scheduled task n i Backup copy of Cannot be compared with the primary copy In the same subnet; constraint C2 means task n i Backup copy of Cannot be used with n k |n k ∈DP(n i )(Direct Predecessor Task) In the same subnet.

3. According to the reinforcement learning-based microservice fault-tolerant scheduling method of claim 1, it is characterized by: In step 1, based on the analysis of task dependencies, design task n i Backup copy of Only in the primary copy The execution can only be started after the execution is completed. This is because the probability of frequent system failures is not high, so the probability of needing to start the backup copy to execute the task is not high. Task n i Master copy Execution can only start after receiving the output data of the previous task (that is, under normal execution). Therefore, according to formula (4), the startup time of the master copy is expressed as follows: Similarly, we need to consider task n i Master copy Backup copy of the previous task In order to minimize the delay, the constraint between the two is expressed as follows: In the case of formula (9), Implementation and To avoid the failure of subsequent task execution, set any backup copy according to formula (4) The start-up time is expressed as follows:

4. According to the reinforcement learning-based microservice fault-tolerant scheduling method of claim 1, it is characterized by: In step 2, a discrete SAC model is trained offline using transitions data. In the present invention, two Q networks are designed to reduce the estimation bias of the value function. Let φ1 and φ2 be the parameters of the two Q networks, are the target network parameters of φ1 and φ2 respectively. θ is the actor network parameter. To achieve online fine-tuning, an additional offline critic network is trained, with the parameter ω. The softvalue and targetvalue are calculated as follows: Where ɑ is the entropy regularization parameter, which is used to control the importance of entropy, and γ is the discount factor. Due to the distribution shift between the heuristic method strategy for collecting data and the offline learning strategy, the CQL regularizer is used to optimize the Q network. The q network parameters are updated by minimizing the following equation using gradient descent. Calculated as follows: In order to facilitate subsequent online fine-tuning, an additional critic network is trained. The critic network parameters are updated by minimizing the MSE using gradient descent, and the actor network parameters are updated by maximizing the MSE using gradient descent. The calculation is as follows:

5. According to the reinforcement learning-based microservice fault-tolerant scheduling method of claim 1, it is characterized by: In step 3, during offline training, an additional critic network is trained. By retaining the actor-critic network trained offline and migrating it to the online fine-tuning architecture, the dataset C is sampled in batches from the online data buffer pool. GAE is used to balance the bias of value estimation and the variance of returns, and the overestimation problem is solved by designing the advantage function. The update of GAE is calculated as follows: Similar to the offline training update process, the critic network parameters are updated by minimizing the MSE through gradient ascent, calculated as follows: The PPO-Clip algorithm is used to update the actor network. The PPO objective function consists of two parts: the clipping objective function and the entropy regularization term. By limiting the speed of network parameter update, the actor network parameters are updated by maximizing the objective function using gradient ascent, which is calculated as follows:

Images