Elastic resource scheduling method and device for micro-service and electronic equipment

Abstract

The invention relates to the field of micro-services, and provides an elastic resource scheduling method and device for micro-services and electronic equipment.The method comprises the steps that a to-be-scheduled micro-service system is determined, and the micro-service system comprises a server and a micro-service instance deployed on the server; constructing a long-term scheduling optimization model by taking the minimization of the long-term average request time delay of the micro-service system as a target; converting the long-term scheduling optimization model into a time slot scheduling optimization model in each time slot based on a Lyapunov optimization method; and obtaining an initial solution of the optimization model based on a near-end optimization strategy, then optimizing the initial solution for a dynamically changing request through a pre-trained deep reinforcement learning model, and performing elastic resource scheduling of the micro-service based on an optimization result. According to the elastic resource scheduling method and device for the micro-service and the electronic equipment provided by the invention, scaling and migration of the micro-service instance are performed aiming at the dynamic request arrival rate, and multi-objective optimization of micro-service scaling and migration cost and user request time delay is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of microservices, and in particular to a microservice elastic resource scheduling method and device and electronic equipment. BACKGROUND

[0002] Microservice architecture decomposes monolithic network services into a series of lightweight services that can be deployed and managed independently, greatly improving the scalability and flexibility of Internet applications. With the maturity of cloud computing and containerization technology, the capabilities of microservices in dynamic deployment, rapid startup, and elastic scaling have been significantly enhanced, and have gradually become the mainstream architecture choice for cloud data center software systems.

[0003] Microservice architecture exhibits unique advantages in the design and management of modern complex distributed systems. User requests received by a microservice system often need to be processed by multiple microservices in coordination, and microservices are highly decoupled and work in coordination through remote calls, so complex request calls and routing selection are required. How to accurately and efficiently select the appropriate routing path for user requests, reduce request response time, and balance the request load between microservices is the key to improving system service performance and resource utilization.

[0004] In actual operation, microservice systems face dynamic changes in request load. In order to cope with uneven distribution and rapid growth of request load, elastic resource scheduling is required, i.e., migrating microservice instances from one resource node to another, or dynamically increasing and decreasing the number of microservice instances. Through appropriate methods for elastic resource scheduling of microservices, long-term satisfaction of user service quality requirements (QoS) while reducing overhead is an urgent problem to be solved. SUMMARY

[0005] The present application provides a microservice elastic resource scheduling method, device and electronic equipment to solve the defects of the prior art that the microservice system service performance and resource utilization are poor in the face of dynamic changes in request load.

[0006] The present application provides a microservice elastic resource scheduling method, comprising: determining a microservice system to be scheduled, the microservice system comprising a server and microservice instances deployed on the server; constructing a long-term scheduling optimization model with the objective of minimizing the long-term average request latency of the microservice system, the constraint conditions of the long-term scheduling optimization model comprising long-term total overhead constraints of the microservice system, memory and central processing unit (CPU) resource constraints of each server; The long-term scheduling optimization model is converted into a time slot scheduling optimization model in each time slot based on a Lyapunov optimization method, and a constraint condition of the time slot scheduling optimization model includes memory and central processing unit (CPU) resource constraints of each server. An initial solution of the optimization model is obtained based on a proximal optimization strategy, and then the initial solution is optimized for dynamic changes in requests by using a pre-trained deep reinforcement learning model, and elastic resource scheduling of the microservice is performed based on an optimization result.

[0007] According to the microservice deployment and elastic resource scheduling method provided by the application, the initial solution is optimized for dynamic changes in requests by using a pre-trained deep reinforcement learning model, and the method comprises the following steps of: Obtaining a current state of the microservice system, wherein the current state includes remaining memory and CPU information of each server, microservice information already deployed on each server, and historical information of microservice requests; Extracting state features of the current state based on a deep neural network in the deep reinforcement learning model; Optimizing the initial solution for dynamic changes in requests based on a policy network in the deep reinforcement learning model and the state features to obtain an optimization result, wherein the optimization result includes a target server, a target microservice instance, and a number change of the target microservice instance.

[0008] According to the microservice deployment and elastic resource scheduling method provided by the application, a reward function of the deep reinforcement learning model is determined based on a backlog of a virtual queue at the end of a current time slot, a total cost in the current time slot, and an average request delay of the microservice system in the current time slot.

[0009] According to the microservice deployment and elastic resource scheduling method provided by the application, the long-term scheduling optimization model is converted into a time slot scheduling optimization model in each time slot based on a Lyapunov optimization method, and a constraint condition of the time slot scheduling optimization model includes memory and central processing unit (CPU) resource constraints of each server. Determining an upper boundary of Lyapunov drift based on a number of microservice instances in each time slot, a number of servers, a number of changes of microservice instances, a number of changes of servers, and a difference between a total cost in each time slot and a cost budget; Determining an upper boundary of a Lyapunov drift plus penalty function in each time slot based on the upper boundary of the Lyapunov drift, a backlog of a virtual queue at the end of each time slot, and an average request delay of the microservice system in each time slot; Determining an optimization objective of the time slot scheduling optimization model in each time slot based on the upper boundary of the Lyapunov drift plus penalty function in each time slot.

[0010] According to the microservice deployment and elastic resource scheduling method provided by the application, the total overhead in each time slot is determined based on the following steps: The deployment overhead in each time slot is determined based on the total number of microservice instances in each time slot, the resources required for running a single microservice instance, the total number of servers in each time slot, and the resources required for running a single server. The scaling overhead in each time slot is determined based on the number of microservice instances that need to be changed in each time slot, the resource overhead of deploying or deleting a single microservice instance, the number of servers that need to be changed in each time slot, and the resource overhead of turning on or off a single server. The total overhead in each time slot is determined based on the deployment overhead and the scaling overhead in each time slot.

[0011] According to the microservice deployment and elastic resource scheduling method provided by the application, the long-term average request latency of the microservice system is determined based on the following steps: The average waiting latency of any request stream on any routing path is determined based on the average number of requests waiting at each microservice instance and the request arrival rate of each microservice instance. The average processing latency of any request stream on any routing path is determined based on the ability of each microservice instance to process requests. The average communication latency of any request stream on any routing path is determined based on whether the predecessor microservice instance of each microservice instance on the routing path is on the same server as it. The long-term average request latency of the microservice system is determined based on the importance weight of each request stream, the probability weight of each routing path, the average waiting latency, the average processing latency, and the average communication latency.

[0012] According to the microservice deployment and elastic resource scheduling method provided by the application, before determining the microservice system to be scheduled, the method further includes performing initial deployment of microservices by using a first-fit decreasing algorithm, and the initial deployment specifically includes: The microservices are sorted according to their resource requirements, and microservices with larger resource requirements are preferentially deployed; The number of instances required for each microservice is determined based on the total arrival rate of each microservice in all request streams. All instances required for each microservice are deployed to servers based on the maximum number of microservice instances that each server can carry.

[0013] According to the microservice deployment and elastic resource scheduling method provided by the application, the training steps of the deep reinforcement learning model include: The candidate action is obtained based on the state feature of the current time step and an initial policy network, the candidate action is sampled to obtain an execution action, and a reward of the current time step is calculated; The value function of the current time step is obtained based on the state feature of the current time step and an initial value network; The advantage function is determined based on the reward of the current time step, and the return of the current time step is determined based on the advantage function and the value function; The policy loss, the value loss and the entropy loss are respectively determined based on the advantage function, the value function and the return of the current time step, and the total loss is determined based on the policy loss, the value loss and the entropy loss; The initial policy network and the initial value network are iterated based on the total loss, and the deep reinforcement learning model is obtained.

[0014] The application also provides a micro-service elastic resource scheduling device, comprising: A micro-service system determination unit is configured to determine a micro-service system to be scheduled, wherein the micro-service system comprises a server and micro-service instances deployed on the server. A long-term model construction unit is configured to construct a long-term scheduling optimization model with the objective of minimizing the long-term average request delay of the micro-service system, wherein the constraint conditions of the long-term scheduling optimization model comprise a long-term total cost constraint of the micro-service system and a memory and central processing unit (CPU) resource constraint of each server. A time slot model construction unit is configured to convert the long-term scheduling optimization model into a time slot scheduling optimization model in each time slot based on a Lyapunov optimization method, wherein the constraint conditions of the time slot scheduling optimization model comprise the memory and CPU resource constraint of each server. An optimization scheduling unit is configured to obtain an initial solution of the optimization model based on a proximal optimization strategy, and then optimize the initial solution for dynamic changes in requests by using a pre-trained deep reinforcement learning model, and perform elastic resource scheduling of the micro-service based on the optimization result.

[0015] The application also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the micro-service elastic resource scheduling method of any of the above when executing the computer program.

[0016] The application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the micro-service elastic resource scheduling method of any of the above.

[0017] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the elastic resource scheduling method of any of the above microservices.

[0018] The microservice elastic resource scheduling method, device and electronic equipment provided by the application minimize the long-term average request delay of the microservice system to build a long-term scheduling optimization model, ensure that the system meets the user's demand for low delay in the global range, improve the user experience, and realize the Pareto optimality of performance and cost through the constraint condition; the long-term scheduling optimization model is converted into a time slot scheduling optimization model in each time slot based on the Lyapunov optimization method, the long-term problem is decomposed into a local optimization problem in each time slot, and the system remains stable under dynamic load; the initial solution of the optimization model is obtained based on the proximal optimization strategy, and then the initial solution is optimized for dynamic changes in requests through the pre-trained deep reinforcement learning model, and the elastic resource scheduling of the microservice is performed based on the optimization result. Multi-objective optimization of microservice scaling migration cost and user request delay is realized, and high performance, high stability and adaptability of microservice scheduling are realized. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 It is a flowchart of the microservice elastic resource scheduling method provided by the application.

[0021] Figure 2 It is a scene diagram of the microservice elastic resource scheduling method provided by the application.

[0022] Figure 3 It is a training structure diagram of the deep reinforcement learning model provided by the application.

[0023] Figure 4 It is a structure diagram of the microservice elastic resource scheduling device provided by the application.

[0024] Figure 5 It is a structure diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0026] In order to accurately and efficiently select a suitable routing path for a user request, reduce the request response delay, and balance the request load between microservices, an embodiment of the present application proposes a microservice elastic resource scheduling method. In the method, first, a long-term scheduling optimization model is constructed with the minimum long-term average request delay of the microservice system as the target, to ensure that the system meets the user's demand for low delay in the global range, improve the user experience, and achieve the Pareto optimality of performance and cost through the constraint condition; then, the long-term scheduling optimization model is converted into a time slot scheduling optimization model in each time slot based on the Lyapunov optimization method, to decompose the long-term problem into a local optimization problem in each time slot, and ensure that the system remains stable under dynamic load; an initial solution of the optimization model is obtained based on the proximal optimization strategy, and then the initial solution is optimized for dynamic changes in requests through a pre-trained deep reinforcement learning model, and the elastic resource scheduling of the microservice is performed based on the optimization result.

[0027] The embodiment of the present application realizes multi-objective optimization of microservice scaling migration cost and user request delay, realizes high performance, high stability and adaptability of microservice scheduling, is suitable for elastic scaling and resource optimization in a cloud native environment, and significantly improves system reliability and operation efficiency.

[0028] The embodiment of the present application can be applied to scenarios that need to perform elastic resource scheduling of microservices, such as cloud native microservices, edge computing, hybrid cloud architecture and the like. The execution subject of the method can be an electronic device such as a terminal device, a computer, a server, a server cluster or a specially designed elastic resource scheduling device, or an elastic resource scheduling apparatus provided in the electronic device, which can be realized by software, hardware or a combination of both.

[0029] Figure 1 is a flowchart of the microservice elastic resource scheduling method provided by the present application, as shown in Figure 1 The method comprises the following steps: Step 110, determining a microservice system to be scheduled, the microservice system comprising a server and a microservice instance deployed on the server.

[0030] Specifically, a microservice system is an architectural style that splits an application into multiple independently deployed, loosely coupled service units. Each service unit (microservice) focuses on a single business function and works collaboratively through lightweight communication protocols.

[0031] A server can be a physical machine or a virtual machine that provides computing resources (CPU, memory, storage, etc.). A microservice instance refers to a single copy of a microservice deployed on a server, which can be independently scaled up or down.

[0032] A microservice system to be scheduled refers to a system that needs to dynamically adjust resource allocation (such as scaling, migration) to meet performance requirements (such as low latency, high throughput) and optimize costs.

[0033] Figure 2 is a schematic diagram of the application scenario of the elastic resource scheduling method of the microservice provided by the present application, referring to Figure 2 , the microservice system includes servers S1-S5, microservice instances m1-m5, the server S1 is deployed with microservice instances m1, m1 and m4, the server S2 is deployed with microservice instances m3, m5, the server S3 is deployed with microservice instances m1, m2, the server S4 is deployed with microservice instances m2, m3, and the server S5 is deployed with microservice instances m4, m4 and m5. There are three user request streams, R1, R2 and R3. The microservice instances corresponding to the request stream R1 are m4→m1→m5→m2, the microservice instances corresponding to the request stream R2 are m1→m3→m4→m2, and the microservice instances corresponding to the request stream R3 are m5→m4→m1→m3. The request queue of the microservice instance m3 has R1 and R3, and the request queue of the microservice instance m5 has R1.

[0034] In step 120, a long-term scheduling optimization model is constructed with the goal of minimizing the long-term average request latency of the microservice system. The constraint conditions of the long-term scheduling optimization model include the long-term total cost constraint of the microservice system, the memory and central processing unit (CPU) resource constraints of each server.

[0035] Specifically, the long-term average request latency refers to the average response time of the microservice system processing all requests during a long period of operation. It is an important indicator of system performance and directly affects user experience. Minimizing the long-term average request latency can ensure that the system still provides low-latency services under high load.

[0036] The long-term total cost refers to the total cost of using all resources (such as servers, storage, network bandwidth, etc.) of the microservice system during a long period of operation. In the optimization process, it is necessary to ensure that the long-term total cost does not exceed the preset budget limit.

[0037] At the same time, the memory and CPU resources of each server are limited, and the deployment and running of microservice instances cannot exceed the limits of these resources.

[0038] In some embodiments, the request latency can be obtained by mathematically modeling the problem using queuing theory, and combining a cost function to obtain a long-term optimization problem, thereby obtaining a long-term scheduling optimization model.

[0039] The long-term average request latency of the microservice system is determined based on the following steps: Based on the average number of requests waiting at each microservice instance in any request stream, and the request arrival rate of each microservice instance, the average waiting latency of the request stream on the routing path is determined. Based on the ability of each microservice instance to process requests, the average processing latency of the request stream on the routing path is determined. Based on whether the predecessor microservice instance of each microservice instance on the routing path is on the same server as it, the average communication latency of the request stream on the routing path is determined. Based on the importance weight of each request stream, the probability weight of each routing path, the average waiting latency, the average processing latency, and the average communication latency, the long-term average request latency of the microservice system is determined.

[0040] Specifically, the model parameters and definitions are as follows:

[0041] Multiple instances of the same microservice can be deployed on each server to improve the server's processing capacity for requests of a certain microservice. Each instance has its own core, so instances are independent of each other. When there are multiple instances of the same microservice on the same server, they can be modeled as a queuing system that contains all instances of the microservice. The service intensity of the queuing system is defined as which is specifically represented as:

[0042] represents the ability of one microservice instance to process requests, represents the request arrival rate of microservice m, represents the number of microservice instances deployed on the server to process microservice m. For a server, when the queuing system reaches a steady state, the average number of requests waiting at microservice is represented as whose expression is:

[0043] where the parameter The expression of the average waiting time is:

[0044] There are three sources of request response delay, namely, waiting delay, processing delay and communication delay. Taking a routing path as an example, specific analysis is carried out: (1) Waiting delay: the time spent by the request waiting for the microservice instance processing, denoted by The average waiting delay of the request microservice is denoted by The calculation formula is:

[0045] The average waiting delay of the request flow in a routing path is

[0046] In the formula, denotes the set of microservices required by the request flow

[0047] (2) Processing delay: the time spent by the microservice processing the request, denoted by The processing delay of the microservice is denoted by The calculation formula is:

[0048] The microservice sequence passed by the request flow in any routing path is the same, and the processing capacity of the server core is also completely the same, so the processing delay of the request flow is irrelevant to the routing path. The average processing delay of any routing path of the request flow is:

[0049] (3) Communication delay: the time spent by the request transferring within the microservice system, denoted by The communication delay of the preceding microservice of the microservice is denoted by The calculation formula is:

[0050] wherein, is a binary decision variable, indicating whether the preceding microservice of the microservice is on the same server, if yes, the value is 0, otherwise the value is 1. The average communication delay of the request flow in a routing path is:

[0051] According to the above analysis, the request flow In a routing path The average request response delay , expressed as:

[0052] The request flow The average request response delay of all routing paths:

[0053] In the formula, The probability of the routing path .

[0054] Finally, according to the importance of each request flow A weighted coefficient , after weighted average, the average request response delay of the entire microservice system :

[0055] The time-averaged average request delay of the microservice system, i.e. the long-term average request delay Can be expressed as:

[0056] In the formula, t is the time variable, T is the termination time, The average function.

[0057] At the same time, in order to ensure the stability of the system performance, balance the relationship between resource consumption and user response speed, it is necessary to define the resource consumption function. That is, the constraint condition of the long-term scheduling optimization model includes the long-term total cost constraint of the microservice system, the memory and central processing unit CPU resource constraint of each server.

[0058] For the long-term total cost of the microservice system, the total cost in each time slot can be modeled first.

[0059] In some embodiments, the total cost in each time slot is determined based on the following steps: Based on the total number of microservice instances in each time slot, the resources required for a single microservice instance to run, the total number of servers in each time slot, and the resources required for a single server to run, the deployment cost in each time slot is determined. Based on the number of microservice instances that need to be changed in each time slot, the resource cost of deploying or deleting a single microservice instance, the number of servers that need to be changed in each time slot, and the resource cost of turning on or off a single server, the scaling cost in each time slot is determined. The total overhead in each time slot is determined based on the deployment overhead and the scaling overhead in each time slot.

[0060] Specifically, the resource consumption of the system running is positively correlated with the number of microservice instance deployments in the time slot , and the total overhead in the time slot can be defined as:

[0061] wherein is the resource required for a single microservice instance running, is the resource required for a single server running. is the total number of microservice instances in the time slot , and is the total number of servers in the time slot .

[0062] The resource overhead caused by instance scaling mainly comes from the deployment and deletion of microservices and the startup and shutdown of servers, which is proportional to the change in the number of microservices and servers, and thus the scaling overhead in the time slot can be defined as:

[0063] wherein is the resource overhead of deploying or deleting a single microservice instance, is the resource overhead of turning on or off a single server, is the number of microservice instances that need to be changed in the time slot , and is the number of servers that need to be changed in the time slot .

[0064] The total overhead in the time slot can be defined as: wherein

[0065] and are non-negative weighting coefficients that weigh the running overhead and the resource scheduling overhead, respectively. The time-averaged long-term overhead budget given by the service provider is denoted as

[0066] , and thus the total overhead needs to satisfy the following constraint condition, i.e., the long-term total overhead constraint can be expressed as:

[0067] ​​​In addition, the long-term scheduling optimization model needs to meet the memory and CPU resource constraints of each server.

[0068] In summary, the long-term optimization problem of elastic resource scheduling under dynamic changes in system load can be represented as problem P1, that is, the long-term scheduling optimization model can be represented as: That is, minimizing the average request latency of the microservice system. The constraints of this long-term scheduling optimization model include:

[0069]

[0070]

[0071]

[0072] In the formula, Indicates server microservices deployed on Quantity, Indicates the deployment of microservices Required CPU resources Indicates server CPU resources available Indicates the deployment of microservices Required memory resources Indicates server The available memory resources.

[0073] Step 130: Based on the Lyapunov optimization method, the long-term scheduling optimization model is transformed into a time-slot scheduling optimization model within each time slot. The constraints of the time-slot scheduling optimization model include the memory and CPU resource constraints of each server.

[0074] Specifically, a day is divided into 288 time slots, each lasting 5 minutes. The long-term optimization problem of the system is transformed into a problem within each time slot using a Lyapunov drift-penalty function. The time slot scheduling optimization model independently optimizes local decisions within each time slot, while long-term stability is ensured through Lyapunov drift constraints. Lyapunov drift measures the change in system state (such as a virtual queue) and is used to constrain system stability. The drift-penalty function adds penalty terms (such as cost or delay) to the drift constraints, achieving a trade-off between stability and performance.

[0075] In some embodiments, step 130 specifically includes: Step 131: Determine the upper boundary of Lyapunov drift based on the number of microservice instances, the number of servers, the number of microservice instance changes, the number of server changes, and the difference between the total overhead and the overhead budget in each time slot. Step 132, based on the upper bound of Lyapunov drift, the backlog of virtual queue at the end of each time slot, the average request delay of microservice system in each time slot, determine the upper bound of Lyapunov drift plus penalty function in each time slot; Step 133, based on the upper bound of Lyapunov drift plus penalty function in each time slot, determine the time slot scheduling optimization model in each time slot.

[0076] Specifically, the problem P1 is a time-averaged optimization problem, and the Lyapunov optimization theory can convert the time-averaged optimization problem into an optimization problem in each time slot by constructing a virtual queue. , represents the backlog of the virtual queue at the end of the time slot The update equation of the virtual queue in each time slot is:

[0077] Wherein, The initial virtual queue backlog at time ; The meaning of

[0078] Lemma 1: If the virtual queue satisfies the above stability condition, the time-averaged constraint condition of the total cost is also satisfied.

[0079] Based on the Lyapunov optimization method, the Lyapunov function is defined as:

[0080] Further, the Lyapunov drift is defined as:

[0081] Lemma 2: There is an upper bound for Lyapunov drift in each time slot, and the upper bound of Lyapunov drift can be determined based on the number of microservice instances, the number of servers, the number of microservice instance changes and the number of server changes in each time slot, and the difference between the total cost and the cost budget in each time slot, which is expressed by the formula:

[0082]

[0083] Wherein, is a constant, , , , are the maximum values of the number of microservices, the number of servers, the number of microservice changes, and the number of server changes in the time slot t, respectively, and can be regarded as known constants. , , , , are corresponding weight factors.

[0084] In order to maintain the stability of the virtual queue while minimizing the average response delay, the optimization objective can also be decomposed into each time slot, and the minimum values of the Lyapunov drift and the average request response delay of the microservice graph are solved at the same time, and the two are balanced through non-negative coefficients . Thus, the "drift plus penalty" function in each time slot can be obtained, which is expressed by the formula as follows:

[0085] The drift plus penalty function has an upper boundary:

[0086] According to the Lyapunov optimization theory, the original problem can be converted into minimizing the upper boundary of the given "drift plus penalty" function, that is:

[0087] After sorting, the following can be obtained:

[0088] wherein the first term is a constant, the second term is a constant, and the third term is the total cost, is the average response delay of the microservice system after the elastic resource scheduling, and both are determined by the elastic resource scheduling scheme. Based on the above analysis, the original problem P1 can be converted into a problem P2 in each time slot through the "drift plus penalty" principle, and according to the framework of the minimum condition expectation, it can be converted into a solution of the minimum unconditional expectation. That is, the time slot scheduling optimization model in each time slot can be expressed as . The constraint conditions of the time slot scheduling optimization model include:

[0089]

[0090]

[0091] ​​​

[0092] In step 140, an initial solution of the optimization model is obtained based on a proximal optimization strategy, and then the initial solution is optimized for dynamic changes in requests by a pre-trained deep reinforcement learning model, and elastic resource scheduling of the microservice is performed based on the optimization result.

[0093] Specifically, the goal of the proximal optimization strategy is to quickly generate an initial solution of resource allocation that meets basic constraints and has high quality, providing a starting point for subsequent DRL optimization. An initial resource allocation scheme that meets the constraint conditions can be generated by a heuristic algorithm suitable for global search, i.e., an initial solution.

[0094] Deep reinforcement learning is a machine learning method that combines deep learning and reinforcement learning. In deep reinforcement learning, an agent interacts with the environment and learns the optimal behavior policy based on the reward signals fed back by the environment. A deep reinforcement learning model is usually composed of a neural network, which is used to approximate the policy function (i.e., the probability distribution of selecting an action given a state) and the value function (i.e., the estimate of future rewards). A trained deep reinforcement learning model can adjust resource allocation based on dynamic environments to optimize long-term goals.

[0095] Elastic resource scheduling refers to dynamically adjusting the computing resources (such as CPU, memory, storage, etc.) allocated to each microservice according to the real-time state and load changes of the microservice system, to meet the performance requirements of the system and improve resource utilization. When the system load increases, the number of instances of microservices or the allocation of more resources is automatically increased; when the system load decreases, the number of instances is reduced or the excess resources are released, thereby realizing the elastic scaling of resources.

[0096] In the embodiments of the present application, a proximal optimization provides a fast initial solution, combined with the dynamic adjustment capability of DRL, to achieve efficient and adaptive microservice resource scheduling.

[0097] In some embodiments, step 140 specifically includes: In step 141, the current state of the microservice system is obtained, including the remaining memory and CPU information of each server, the microservice information already deployed on each server, and the historical information of microservice requests; In step 142, the state features of the current state are extracted based on the deep neural network in the deep reinforcement learning model; In step 143, the initial solution is optimized for dynamic changes in requests based on the policy network in the deep reinforcement learning model and the state features, to obtain an optimization result, which includes the target server, the target microservice instance, and the number of changes in the target microservice instance.

[0098] Specifically, the deep reinforcement learning model is an Actor-Critic structure, and the main structure includes a deep neural network (DNN), a policy network (actor) and a value network (critic). The DNN adopts a multi-layer Tanh activated fully connected network to process the input environment state features. The policy network actor determines the action to be executed through a parallel three-headed output, and the three output heads are used to output the server selection s, the microservice type selection m and the number of increase or decrease. The original action space is changed from the original product form to the sum form through the three-headed output, which greatly reduces the action space, greatly improves the running efficiency of the algorithm under the premise of ensuring stable convergence, and conforms to the actual situation.

[0099] The environment state and action of this algorithm are designed as follows: State: The state perceived by the reinforcement learning model mainly includes the current data center network state (including the remaining memory and CPU information of each server), the deployed microservice information, and the historical information of the microservice chain request received by the data center. The state space State can be specifically represented as:

[0100] Among them, represents the remaining memory information of each server in the microservice system, represents the remaining CPU information of each server in the microservice system, represents the deployed microservice information of each server in the microservice system, which specifically includes the type and corresponding number of microservices deployed on the server, represents the historical information of the microservice chain request received by the data center, which is used to predict the arrival of the next time slot microservice request, so as to make adjustments in advance.

[0101] Action: The action space is designed as follows:

[0102] Among them, represents the server selected by the policy, that is, which server to operate, represents the selected microservice type, represents the change in the number of microservices. Therefore, action represents the increase or decrease of the number of microservices on server by .

[0103] In this embodiment, the current state of the microservice system is first obtained, the current state is input into the deep neural network in the deep reinforcement learning model, and the state features are extracted. Then the state features are input into the policy network in the deep reinforcement learning model, and the optimized execution action is output by the policy network, that is, the optimization result of the current time slot is obtained. Here, the target server refers to the server selected by the policy, the target microservice instance refers to the selected microservice instance, and the change in the number of target microservice instances refers to the increase or decrease of the target microservice instances on the target server.

[0104] Based on the above embodiment, the reward function of the deep reinforcement learning model is determined based on the backlog of the virtual queue at the end of the current time slot, the total cost in the current time slot, and the average request delay of the microservice system in the current time slot.

[0105] Specifically, the embodiment uses a reward + risk-averse reward function to design the policy, so that the agent focuses more on the optimization of the target, finds the potential optimal solution, and avoids making wrong behaviors through risk aversion. The reward is designed as follows according to the result of Lyapunov optimization:

[0106] At the same time, when the agent makes the following actions, the execution of this action is prohibited: a. When the agent wants to increase a certain microservice instance on a certain server, the memory and CPU limits of the server are exceeded, and the maximum number of this microservice instance is increased on the premise of not exceeding the resource limits of this server.

[0107] b. When the agent wants to reduce a certain microservice instance on a certain server, there is no such microservice instance on the server, and the execution of this action is cancelled.

[0108] Based on the above embodiment, the deep reinforcement learning model can be trained before step 140. The training steps include: Based on the state features of the current time step and the initial policy network, candidate actions are obtained, the candidate actions are sampled to obtain execution actions, and the reward of the current time step is calculated; Based on the state features of the current time step and the initial value network, the value function of the current time step is obtained; Based on the reward of the current time step, the advantage function is determined, and based on the advantage function and the value function, the return of the current time step is determined; Based on the advantage function, the value function, and the return of the current time step, the policy loss, the value loss, and the entropy loss are determined respectively, and based on the policy loss, the value loss, and the entropy loss, the total loss is determined; Based on the total loss, the initial policy network and the initial value network are iterated in parameters to obtain the deep reinforcement learning model.

[0109] Specifically, Figure 3 is a training structure schematic diagram of the deep reinforcement learning model provided by the application, referring to Figure 3 , the agent interacts with the micro-service environment for training, so as to converge on the reward function. The agent obtains the state from the environment, and outputs possible actions, i.e. candidate actions, through the initial policy network. And the execution action is obtained by sampling the candidate action , and the value function is output through the initial value network . After the action is executed, a new state and a reward are obtained.

[0110] After each training cycle, generalized advantage estimation (GAE) is used to calculate the advantage function of each step, which indicates the advantage of the action selected by the agent relative to the remaining selection. The advantage function is solved through the recursive formula .

[0111] Among them, is a discount factor, is a decay factor of GAE, is a TD error, which is calculated as follows:

[0112] and the return can be calculated:

[0113] In this embodiment, in order to balance the exploration and stability of the algorithm, the calculation of the loss mainly includes three parts of the policy loss, the value loss and the entropy loss.

[0114] The policy loss is used to optimize the initial policy network, and is calculated as follows:

[0115] Among them, is an importance sampling ratio, which measures the difference between the current policy and the old policy, and is limited by the clip function to avoid instability caused by too large policy update; is a hyperparameter (usually with a small value, such as 0.1 or 0.2), which controls the amplitude of policy update.

[0116] The entropy loss is used to encourage the exploration of the policy and avoid the model falling into a local optimal solution. The specific calculation is as follows:

[0117] The value loss is used to update the initial value network, and the mean square error (MSE) is used to calculate the difference between the current extreme network estimated value and the actual return, and the specific calculation is as follows:

[0118] The total loss can be obtained as follows, wherein The total loss can be expressed as:

[0119] On this basis, the total loss obtained is used for gradient back propagation and the network parameters are updated to obtain a trained deep reinforcement learning model.

[0120] Based on any of the above embodiments, before determining the microservice system to be scheduled, the method further comprises performing initial deployment of the microservice through the first-fit decreasing algorithm, and the initial deployment specifically comprises: The microservices are sorted according to their resource requirements, and the microservices with larger resource requirements are deployed first; Based on the total arrival rate of various microservices in all request flows, the required instance numbers of various microservices are determined; All instances required by various microservices are deployed to servers based on the maximum number of microservice instances carried by each server.

[0121] Specifically, the initial deployment of the microservice system can be implemented through the first-fit decreasing algorithm. The algorithm steps are as follows: Step 1: Sort the microservices according to their resource requirements. Specifically, the microservices are arranged in descending order according to their CPU and memory requirements. In this way, the microservices with larger resource requirements are deployed first, and the microservices with larger resource requirements occupy more resources, so they should be processed first to better match the resources of the nodes.

[0122] Step 2: Calculate the total arrival rate of various microservices in all request flows. In order to ensure service intensity , the required instance numbers of various microservices are calculated first, and the required instances of the microservice are deployed to the servers.

[0123] Step 3: Iterate through each server and calculate the maximum number of microservice instances that the current server can support. If the current server has sufficient resources, deploy the microservice instance to that server and update the server status. If the current server cannot fully deploy all instances of the microservice, calculate the remaining number of instances after the server is fully deployed, and deploy the remaining instances to other servers. This process will continue until all instances of the microservice are deployed, or there are not enough resources to deploy more instances.

[0124] In summary, this invention applies an open queuing network model to model the microservice system and, based on Liyapunov theory, transforms the long-term stability problem of microservice deployment cost and latency into an optimization problem within a single time slot. The first-adaptive decreasing algorithm is used for the initial deployment, while a novel reinforcement learning algorithm is designed to scale and migrate microservice instances based on dynamic request arrival rates, thereby achieving multi-objective optimization of microservice scaling and migration costs and user request latency.

[0125] The elastic resource scheduling device for microservices provided by the present invention is described below. The elastic resource scheduling device for microservices described below can be referred to in correspondence with the elastic resource scheduling method for microservices described above.

[0126] Based on the above embodiments, Figure 4 This is a schematic diagram of the microservice elastic resource scheduling device provided by the present invention, as shown below. Figure 4 As shown, the device includes: The microservice system determination unit 410 is used to determine the microservice system to be scheduled, wherein the microservice system includes a server and microservice instances deployed on the server; The long-term model building unit 420 is used to build a long-term scheduling optimization model with the goal of minimizing the long-term average request latency of the microservice system. The constraints of the long-term scheduling optimization model include the long-term total overhead constraints of the microservice system, the memory and CPU resource constraints of each server. The time slot model construction unit 430 is used to transform the long-term scheduling optimization model into a time slot scheduling optimization model within each time slot based on the Lyapunov optimization method. The constraints of the time slot scheduling optimization model include memory and CPU resource constraints for each server. The optimization scheduling unit 440 is used to obtain the initial solution of the optimization model based on the near-end optimization strategy, and then optimize the initial solution for dynamically changing requests through a pre-trained deep reinforcement learning model, and perform elastic resource scheduling of microservices based on the optimization results.

[0127] Based on the above embodiments, the optimized scheduling unit is specifically used for: obtaining a current state of the micro-service system, the current state including remaining memory and CPU information of each server, micro-service information already deployed on each server, and historical information of micro-service requests; extracting state features of the current state based on a deep neural network in the deep reinforcement learning model; optimizing the initial solution for dynamically changing requests based on a policy network in the deep reinforcement learning model and the state features to obtain an optimization result, the optimization result including a target server, a target micro-service instance, and a number change of the target micro-service instance.

[0128] Based on the above embodiment, the reward function of the deep reinforcement learning model is determined based on a backlog of a virtual queue at the end of a current time slot, a total cost in the current time slot, and an average request delay of the micro-service system in the current time slot.

[0129] Based on the above embodiment, the time slot model construction unit is specifically configured to: determine an upper bound of Lyapunov drift based on a number of micro-service instances, a number of servers, a number of micro-service instance changes, and a number of server changes in each time slot, and a difference between a total cost and a cost budget in each time slot; determine an upper bound of Lyapunov drift plus a penalty function in each time slot based on the upper bound of Lyapunov drift, a backlog of a virtual queue at the end of each time slot, and an average request delay of the micro-service system in each time slot; determine an optimization objective of a time slot scheduling optimization model in each time slot based on the upper bound of Lyapunov drift plus the penalty function in each time slot.

[0130] Based on the above embodiment, the time slot model construction unit is specifically configured to: determine a deployment cost in each time slot based on a total number of micro-service instances, resources required for running a single micro-service instance, a total number of servers, and resources required for running a single server in each time slot; determine a scaling cost in each time slot based on a number of micro-service instances that need to be changed, a resource cost of deploying or deleting a single micro-service instance, a number of servers that need to be changed in each time slot, and a resource cost of turning on or off a single server; determine a total cost in each time slot based on the deployment cost and the scaling cost in each time slot.

[0131] Based on the above embodiment, the long-term model construction unit is specifically configured to: determine an average waiting latency of the any request stream in any routing path based on an average number of requests waiting at each microservice instance in any request stream and a request arrival rate of each microservice instance; determine an average processing latency of the any request stream in any routing path based on a capability of each microservice instance to process requests; determine an average communication latency of the any request stream in any routing path based on whether a predecessor microservice instance of each microservice instance on a routing path is on the same server as the microservice instance; determine a long-term average request latency of the microservice system based on an importance weight of each request stream, a probability weight of each routing path, the average waiting latency, the average processing latency, and the average communication latency.

[0132] Based on the above embodiments, the apparatus further includes a primary deployment unit, specifically configured to: sort the microservices according to resource requirements, and preferentially deploy microservices with larger resource requirements; determine the required number of instances of each microservice based on a total arrival rate of all request streams; traverse each server, and deploy all required instances of each microservice to the server based on a maximum number of microservice instances that can be carried by the server.

[0133] Based on the above embodiments, the apparatus further includes a model training unit, specifically configured to: obtain a candidate action based on the state feature of the current time step and the initial policy network, sample the candidate action to obtain an execution action, and calculate a reward of the current time step; obtain a value function of the current time step based on the state feature of the current time step and the initial value network; determine an advantage function based on the reward of the current time step, and determine a return of the current time step based on the advantage function and the value function; determine a policy loss, a value loss, and an entropy loss based on the advantage function, the value function, and the return of the current time step, and determine a total loss based on the policy loss, the value loss, and the entropy loss; perform parameter iteration on the initial policy network and the initial value network based on the total loss to obtain the deep reinforcement learning model.

[0134] Figure 5 An example of an entity structure diagram of an electronic device is shown as Figure 5As shown, the electronic device can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logic instruction in the memory 530 to execute the elastic resource scheduling method of the microservice, and the method includes: determining a microservice system to be scheduled, the microservice system including a server and a microservice instance deployed on the server; constructing a long-term scheduling optimization model with the minimization of long-term average request delay of the microservice system as the target, the constraint condition of the long-term scheduling optimization model including long-term total cost constraint of the microservice system, memory and central processing unit (CPU) resource constraint of each server; converting the long-term scheduling optimization model into a time slot scheduling optimization model in each time slot based on the Lyapunov optimization method, the constraint condition of the time slot scheduling optimization model including the memory and CPU resource constraint of each server; based on the current state of the microservice system, optimizing the time slot scheduling optimization model through a pre-trained deep reinforcement learning model based on proximal policy optimization, and performing elastic resource scheduling of the microservice based on the optimization result.

[0135] In addition, the logic instruction in the memory 530 described above can be implemented in the form of a software functional unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0136] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer-readable storage medium, and the computer program can be executed by a processor to enable a computer to perform the method of the microservice elastic resource scheduling method provided by the above-mentioned methods, which comprises: determining a microservice system to be scheduled, the microservice system comprising servers and microservice instances deployed on the servers; constructing a long-term scheduling optimization model with the objective of minimizing the long-term average request latency of the microservice system, the constraint conditions of the long-term scheduling optimization model comprising long-term total cost constraints of the microservice system, memory and central processing unit (CPU) resource constraints of each server; converting the long-term scheduling optimization model into a time slot scheduling optimization model within each time slot based on a Lyapunov optimization method, the constraint conditions of the time slot scheduling optimization model comprising memory and CPU resource constraints of each server; and optimizing the time slot scheduling optimization model based on a pre-trained deep reinforcement learning model based on proximal policy optimization based on the current state of the microservice system, and performing the microservice elastic resource scheduling based on the optimization result.

[0137] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which can be executed by a processor to implement the microservice elastic resource scheduling method provided by the above-mentioned methods, which comprises: determining a microservice system to be scheduled, the microservice system comprising servers and microservice instances deployed on the servers; constructing a long-term scheduling optimization model with the objective of minimizing the long-term average request latency of the microservice system, the constraint conditions of the long-term scheduling optimization model comprising long-term total cost constraints of the microservice system, memory and central processing unit (CPU) resource constraints of each server; converting the long-term scheduling optimization model into a time slot scheduling optimization model within each time slot based on a Lyapunov optimization method, the constraint conditions of the time slot scheduling optimization model comprising memory and CPU resource constraints of each server; and optimizing the time slot scheduling optimization model based on a pre-trained deep reinforcement learning model based on proximal policy optimization based on the current state of the microservice system, and performing the microservice elastic resource scheduling based on the optimization result.

[0138] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.

[0139] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0140] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for elastic resource scheduling of microservices, characterized in that, The application relates to a method for scheduling a micro-service system, and the method comprises the following steps: determining a micro-service system to be scheduled, wherein the micro-service system comprises servers and micro-service instances deployed on the servers; constructing a long-term scheduling optimization model with the objective of minimizing the long-term average request latency of the micro-service system, wherein constraint conditions of the long-term scheduling optimization model comprise a long-term total cost constraint of the micro-service system, memory and central processing unit (CPU) resource constraints of each server; transforming the long-term scheduling optimization model into a time-slot scheduling optimization model in each time slot based on a Lyapunov optimization method, wherein constraint conditions of the time-slot scheduling optimization model comprise the memory and CPU resource constraints of each server; obtaining an initial solution of the optimization model based on a proximal optimization strategy, and then optimizing the initial solution for dynamic changes in requests by using a pre-trained deep reinforcement learning model, and performing elastic resource scheduling of micro-services based on an optimization result.

2. The method of claim 1, wherein, The optimization of the initial solution for dynamic changes in requests by using the pre-trained deep reinforcement learning model comprises the following steps: obtaining a current state of the micro-service system, wherein the current state comprises residual memory and CPU information of each server, micro-service information already deployed on each server and historical information of micro-service requests; extracting state features of the current state based on a deep neural network in the deep reinforcement learning model; optimizing the initial solution for dynamic changes in requests based on a policy network in the deep reinforcement learning model and the state features to obtain an optimization result, wherein the optimization result comprises a target server, a target micro-service instance and a quantity change of the target micro-service instance.

3. The method of claim 2, wherein, A reward function of the deep reinforcement learning model is determined based on a backlog of a virtual queue at the end of a current time slot, a total cost in the current time slot and an average request latency of the micro-service system in the current time slot.

4. The method of claim 1, wherein, The transformation of the long-term scheduling optimization model into the time-slot scheduling optimization model in each time slot based on the Lyapunov optimization method comprises the following steps: determining an upper boundary of Lyapunov drift based on the number of micro-service instances in each time slot, the number of servers, the number of changes of micro-service instances and the number of changes of servers and the difference between a total cost in each time slot and a cost budget; determining an upper boundary of a Lyapunov drift plus penalty function in each time slot based on the upper boundary of the Lyapunov drift, the backlog of the virtual queue at the end of each time slot and the average request latency of the micro-service system in each time slot; determining an optimization objective of the time-slot scheduling optimization model in each time slot based on the upper boundary of the Lyapunov drift plus penalty function in each time slot.

5. The method of claim 4, wherein, The total cost in each time slot is determined based on the following steps: determining a deployment cost in each time slot based on the total number of micro-service instances in each time slot, the resources required for running a single micro-service instance, the total number of servers in each time slot and the resources required for running a single server. determine a scaling overhead in each time slot based on a number of microservice instances that need to be changed in each time slot, a resource overhead of deploying or deleting a single microservice instance, a number of servers that need to be changed in each time slot, and a resource overhead of turning on or off a single server; determine a total overhead in each time slot based on the deployment overhead and the scaling overhead in each time slot.

6. The method of claim 1, wherein, determine a long-term average request latency of the microservice system based on the following steps: determine an average waiting latency of any request stream in any routing path based on an average number of requests waiting at each microservice instance and a request arrival rate of each microservice instance; determine an average processing latency of the any request stream in any routing path based on a processing capacity of each microservice instance; determine an average communication latency of the any request stream in any routing path based on whether a predecessor microservice instance of each microservice instance on a routing path is on the same server as the microservice instance; determine the long-term average request latency of the microservice system based on an importance weight of each request stream, a probability weight of each routing path, the average waiting latency, the average processing latency, and the average communication latency.

7. The method of claim 1 to 6, wherein, Before the determining the microservice system to be scheduled, the method further comprises performing an initial deployment of microservices by a first-fit decreasing algorithm, and the initial deployment specifically comprises: sorting the microservices according to resource requirements of the microservices, and preferentially deploying microservices with larger resource requirements; determining a required number of instances of each microservice based on a total arrival rate of the microservice in all request streams; deploying all instances of each microservice on servers based on a maximum number of microservice instances that each server can carry.

8. The method of claim 1 to 6, wherein, The training steps of the deep reinforcement learning model comprise: obtaining candidate actions based on state features of a current time step and an initial policy network, sampling the candidate actions to obtain executed actions, and calculating a reward of the current time step; obtaining a value function of the current time step based on the state features of the current time step and an initial value network; determining an advantage function based on the reward of the current time step, and determining a return of the current time step based on the advantage function and the value function; determining a policy loss, a value loss, and an entropy loss based on the advantage function, the value function, and the return of the current time step, and determining a total loss based on the policy loss, the value loss, and the entropy loss; performing parameter iteration on the initial policy network and the initial value network based on the total loss to obtain the deep reinforcement learning model.

9. An elastic resource scheduling device for microservices, characterized in that, comprise: a microservice system determination unit configured to determine a microservice system to be scheduled, the microservice system comprising servers and microservice instances deployed on the servers; a long-term model construction unit configured to construct a long-term scheduling optimization model with a goal of minimizing a long-term average request latency of the microservice system, and the constraint conditions of the long-term scheduling optimization model comprising a long-term total overhead constraint of the microservice system and a memory and central processing unit (CPU) resource constraint of each server; a time slot model construction unit, configured to convert the long-term scheduling optimization model into a time slot scheduling optimization model in each time slot based on a Lyapunov optimization method, and a constraint condition of the time slot scheduling optimization model includes memory and central processing unit (CPU) resource constraints of each server; an optimization scheduling unit, configured to obtain an initial solution of the optimization model based on a proximal optimization strategy, and then optimize the initial solution for dynamic changes in requests by using a pre-trained deep reinforcement learning model, and perform elastic resource scheduling of the microservice based on an optimization result.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the elastic resource scheduling method of the microservice according to any one of claims 1 to 8 when executing the computer program.

Images