A cluster automatic scaling method and system based on a proximal policy optimization algorithm

Abstract

The application discloses a cluster automatic scaling method and system based on a proximal policy optimization algorithm. The method comprises the following steps: predicting the throughput of each root micro-service endpoint; estimating the required instance number of all micro-services according to the predicted throughput of each root micro-service endpoint and a micro-service dependency topology; waiting for an event to trigger the cluster automatic scaler to run; when the event is a capacity reduction event, constructing an environment state according to current and future information of a micro-service cluster, using a trained proximal policy optimization intelligent agent to obtain an action, and determining whether the capacity reduction decision of a node that needs to be reduced is to sleep or to be directly released according to the action; and calling an executor to execute the capacity reduction decision. Compared with the prior art, the application can reduce CPU overhead and improve cluster energy efficiency.

Description

Technical Field

[0001] This invention belongs to the field of cloud computing technology, and in particular relates to a cluster autoscaling method and system based on a near-end strategy optimization algorithm. Background Technology

[0002] Microservice architecture divides a traditional monolithic application into a group of small, autonomous microservices responsible for different business functions and coordinating their work through lightweight communication mechanisms. Based on containerization technology, microservice applications can easily scale instances elastically to cope with dynamically changing workload demands. These containerized instances are often deployed on virtual machine nodes in public or private clouds, forming a cloud cluster with a container-virtual machine two-tier architecture. To adapt to the changing resource requirements of microservice scaling, the cluster also needs to adjust the number of virtual machines in a timely manner to ensure normal service operation, reduce resource overhead, and thus improve energy efficiency.

[0003] Currently, cluster autoscaling technologies are mainly divided into two categories. One is reactive autoscaling, which is based on thresholds of cluster resource metrics to trigger scaling up or down; the other is predictive autoscaling, which is based on predictions of workload and other factors to achieve proactive scaling up or down.

[0004] A typical example of reactive autoscaling is "Method, Apparatus and System for Automatic Scaling of Cluster Resources (CN116107687A)". This document scans the current resource utilization based on request commands and determines whether to create or delete cloud servers based on whether the scaling threshold has been reached. The default cluster autoscaler integrated into the container orchestration system Kubernetes (K8s) is also a type of reactive autoscaler. It adjusts the K8s cluster size according to resource requirements. A pod is the smallest deployable unit that can be created and managed in Kubernetes. When a pod cannot be scheduled on an existing node due to insufficient resources, the number of nodes is increased; when node utilization is low and pods on a node can be scheduled to other nodes, the number of nodes is reduced. However, this type of reactive autoscaling may lead to frequent node creation or deletion. The additional CPU overhead caused by virtual machine creation and initialization can easily lead to resource waste and increase service downtime, resulting in performance degradation.

[0005] A typical example of predictive autoscaling is the "Kubernetes cluster autoscaler based on multiple node types (Wang M, Zhang D, Wu BA cluster autoscaler based on multiple node types Kubernetes[C] / / 2020IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC).IEEE,2020,1:575-579.)". Regarding cluster expansion, this paper uses a prediction module to predict that the system load will exceed a set threshold at the predicted time point, thus triggering the expansion process in advance and adding pre-loaded standby nodes to the cluster. Regarding cluster shrinkage, this paper adds a release waiting process before releasing nodes; nodes are only released if the predicted load does not rise above the threshold during the waiting period. Predictive scaling improves system performance under rapid load pressure and avoids frequent cluster scaling, reducing instability. However, this method does not fully utilize the characteristics of virtual machines, only considering the creation and deletion of virtual machines, and the idle nodes during the release waiting period also lead to some resource waste, failing to maximize cluster energy efficiency. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings and challenges of the prior art and provide a cluster autoscaling method and system based on a near-end strategy optimization algorithm to reduce CPU overhead and improve cluster energy efficiency.

[0007] The objective of this invention is achieved by at least one of the following technical solutions.

[0008] A cluster autoscaling method based on a near-end policy optimization algorithm includes:

[0009] Predict the throughput of each root microservice endpoint;

[0010] Estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology.

[0011] Wait for an event to trigger the cluster autoscaler to run;

[0012] When the event is a scaling down event, the environment state is constructed based on the current and future information of the microservice cluster, and the trained near-end strategy is used to optimize the agent to obtain the action. Based on the action, the scaling down decision of the node that needs to be scaled down is determined to be either put into hibernation or released directly.

[0013] The executor is invoked to execute the scaling-down decision.

[0014] The above scheme predicts the throughput of each root microservice endpoint, specifically including:

[0015] For a root microservice endpoint r∈R, where R represents the set of root microservice endpoints, let the current date be day d, and retrieve the historical throughput data for D consecutive days. in This represents the historical throughput data of the root microservice endpoint r on day db, 0 <b<D;

[0016] Predict the throughput data for day d based on historical throughput data over D consecutive days. in W represents the predicted throughput of the root microservice endpoint r at the predicted time point w on day d. The value of W represents the number of predicted time points, which depends on the time step scale T0. W = 24h / T0, where h represents hours.

[0017] The above scheme estimates the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology. Specifically, this includes:

[0018] The throughput of all microservice endpoints in the corresponding call chain is evaluated based on the predicted throughput of each root microservice endpoint and the microservice dependency topology.

[0019] Calculate the execution time required for a microservice endpoint to complete a service request at a given throughput based on the throughput of all microservice endpoints in the corresponding call chain;

[0020] The parallelism of each microservice is calculated based on the execution time and the throughput of all microservice endpoints in the corresponding call chain, and the number of instances required for each microservice is calculated based on the parallelism.

[0021] In the above scheme, the step of evaluating the throughput of all microservice endpoints on the corresponding call chain based on the predicted throughput of each root microservice endpoint and the microservice dependency topology specifically includes:

[0022] Based on the microservice dependency topology, determine the upstream endpoint u of any microservice endpoint l, whose dependency relationship is θ. l,u θ l,u This represents the average number of times microservice endpoint l is called by endpoint u when endpoint u is called once;

[0023] Based on dependency θ l,u Let the throughput of endpoint u at the predicted time point w be... The throughput obtained by microservice endpoint l from endpoint u at the predicted time point w is then... Calculated using the following formula:

[0024]

[0025] Total throughput of microservice endpoint l at the predicted time point w The calculation formula is as follows:

[0026]

[0027] Where U(l) represents all upstream endpoints of microservice endpoint l;

[0028] Based on formulas (1) and (2), the root microservice endpoint serves as the upstream endpoint in the corresponding call chain. According to the predicted throughput of each root microservice endpoint, the total throughput of all microservice endpoints in the call chain at each predicted time point w is recursively calculated starting from the root microservice endpoint.

[0029] In the above scheme, the step of calculating the execution time required for each microservice endpoint to complete a service request under a given throughput based on the throughput of all microservice endpoints in the corresponding call chain specifically includes:

[0030] Use an SVR-based performance model to build a regression model of throughput versus execution time for any microservice endpoint. Then the total throughput of microservice endpoint l at each predicted time point w will be calculated. Given the throughput, the execution time of microservice endpoint l is:

[0031]

[0032] In the above scheme, the step of calculating the parallelism of each microservice based on the execution time and the throughput of all microservice endpoints in the corresponding call chain, and calculating the number of instances required for each microservice based on the parallelism, specifically includes:

[0033] Based on Little's Law, for a given throughput, microservice endpoint l... and corresponding execution time The number of concurrent requests that need to be processed for:

[0034]

[0035] Calculate the parallelism p of each microservice s s,w :

[0036]

[0037] Where, microservices s∈S, S represents all microservices in the microservice application, and the degree of parallelism p s,wL(s) refers to the total number of concurrent requests processed by all microservice endpoints of a microservice s at the predicted time point w.

[0038] The number of instances n required for microservice s at various predicted time points w within a 24-hour period. s,w We obtain it from the following formula:

[0039]

[0040] Where, p smax The maximum concurrency of a microservice s represents the maximum number of concurrent requests that a single instance of a microservice s can handle simultaneously. It is set when the microservice application is deployed.

[0041] In the above scheme, when the event is a scaling-down event, constructing the environment state based on the current and future information of the microservice cluster specifically includes:

[0042] Node status includes four states: running, idle, sleeping, and released. The idle state indicates that all microservice instances on the node are in a suspended state, which is a transitional state.

[0043] The scaling down event refers to all microservice instances on a certain node being in a suspended state. The suspended state of a microservice instance is a state in the microservice activation and suspension mechanism. It means that when a microservice needs to be scaled down, it is suspended by hiding its microservice instance endpoint, so that it cannot receive requests.

[0044] The environment status includes two parts: the current status and the expected status, which correspond to the current and future instance count information of the microservice cluster, respectively.

[0045] The instantaneous state represents the number of instances of each microservice in the cluster that are in the running state and the suspended state at the current time t, expressed as:

[0046]

[0047] in, This indicates the number of instances of microservice s that are currently running. This represents the number of microservice s instances that are in a suspended state; S is the total number of microservices.

[0048] The expected state represents the number of instances required for each microservice at time t+Δt, which is based on the number of instances n required for microservice s at each prediction time point w within the 24 hours of that day. s,w Get, represented as in This represents the number of instances of microservice s required at time t+Δt corresponding to the predicted time point w;

[0049] Combining the immediate state and the expected state, the environmental state at time t can be expressed as:

[0050] In the above scheme, the action is the action of the agent in a given state s. t The output below is represented as a t It is a one-hot vector, when When the value is 1, the node enters a sleep state; otherwise, the node is released directly.

[0051] A reward is given after the agent performs an action based on the objective; the reward is set as follows: after performing action a t Start T r The CPU time used by the node startup process, instance initialization process, and node sleep process within a given time period; assuming that M nodes are started, N microservice instances are created, and O nodes enter sleep state within this time period, then action a t Rewards The calculation formula is:

[0052]

[0053] in, This indicates the total CPU time used by the node startup process. This represents the average CPU utilization of node i during startup. Indicates the startup time of node i; This indicates the total CPU time used by the instance initialization process. This represents the average CPU utilization of microservice instance j during initialization. Indicates the initialization time of microservice instance j; This represents the total CPU time used during node sleep. This represents the average CPU utilization of node m during sleep. This represents the sleep time of node m. The hyperparameter α is used to adjust the optimization direction of the agent. When α = 1, the agent will optimize the strategy to minimize CPU overhead; when α > 1, it will tend to put the node to sleep; when α < 1, it will tend to release the node directly.

[0054] In the above scheme, the step of using a trained proximal policy to optimize the agent to obtain actions specifically includes:

[0055] The agent network parameters are updated using a proximal policy optimization algorithm. This algorithm optimizes the policy based on an Actor-Critic architecture and uses a deep neural network to fit the Actor and Critic networks. The network parameters for the Actor and Critic are defined as θ and φ, respectively, and the agent's policy network is defined as π. θ The agent objective function of the Actor network is then defined as:

[0056]

[0057] in Representing the new strategy π θ and old strategies In state s t Next, execute action a t The probability ratio, clip represents the clipping function; A t Represents state-action pairs (s) t ,a t Advantage estimation, V φ (s t ) is the output of the Critic network, i.e., the state value estimate; ε is a hyperparameter that determines the scope of policy update pruning.

[0058] The loss function of the Critic network is defined as V φ (s t )and The mean square error, i.e.:

[0059]

[0060] In each episode, the agent collects T e The state, action, and reward information of each timestamp are used as the trajectory. The stochastic gradient ascent algorithm is used to update the parameter θ to maximize the direction of the Actor network agent target. k+1 =argmax θ L P (θ); The parameter φ is updated using the stochastic gradient descent algorithm in the direction that minimizes the mean square error, i.e., φ k+1 =argmin φ L VF (φ); If the learning curve of the agent is converged by iteratively updating E episodes, the policy network is applied to the decision of the cluster autoscaler.

[0061] This invention also provides a cluster autoscaling system based on a near-end policy optimization algorithm, comprising:

[0062] The throughput prediction module is used to predict the throughput of each root microservice endpoint.

[0063] The instance count calculation module is used to estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology.

[0064] The trigger module is used to wait for events to trigger the cluster autoscaler to run;

[0065] The scaling-down event decision module is used to construct the environment state based on the current and future information of the microservice cluster when the event is a scaling-down event, and to optimize the agent using the trained near-end strategy to obtain the action. Based on the action, it determines whether the scaling-down decision of the node that needs to be scaled down is to put it into hibernation or to release it directly.

[0066] The execution module is used to call the executor to execute the scaling-down decision.

[0067] Compared with the prior art, the present invention has the following advantages and technical effects:

[0068] 1. The cluster autoscaling method and system based on the near-end policy optimization algorithm proposed in this invention effectively considers the current and future resource requirements of microservices. By training the agent, the optimal policy is obtained, which solves the challenge of which strategy is more energy-efficient, such as hibernating or releasing nodes when the cluster is downsizing, so as to minimize CPU overhead and improve cluster energy efficiency.

[0069] 2. The cluster scaling strategy proposed in this invention makes full use of the characteristics of virtual machines. In addition to creating nodes and directly releasing nodes, it also considers the operation of hibernating nodes. This allows nodes to occupy far less resources than required for normal operation, while avoiding the time and CPU overhead required for loading images and initialization when frequently recreating nodes.

[0070] 3. This invention aims to reduce the CPU overhead of business logic-irrelevant processes in a cluster. It sets the reward of the reinforcement learning agent as the weighted sum of the CPU time of the node startup process, instance initialization process, and node sleep process, thereby enabling the agent to optimize in the direction of reducing cluster CPU overhead and thus improving cluster energy efficiency. Attached Figure Description

[0071] Figure 1 This is a flowchart illustrating a cluster autoscaling method based on a near-end strategy optimization algorithm, which is a specific embodiment of the present invention.

[0072] Figure 2 This is a schematic diagram of node state transitions in a specific embodiment 1 of the present invention.

[0073] Figure 3 This is a schematic diagram of the state transition of a microservice instance in a specific embodiment 1 of the present invention.

[0074] Figure 4 This is a schematic diagram of the learning curve of the proximal policy optimization agent in a specific embodiment 1 of the present invention.

[0075] Figure 5 This is an architecture diagram of a cluster autoscaling system based on a near-end strategy optimization algorithm, which is a specific embodiment 2 of the present invention. Detailed Implementation

[0076] To make the technical solutions and advantages of the present invention clearer, the following detailed description is provided in conjunction with the accompanying drawings, but the implementation and protection of the present invention are not limited thereto.

[0077] Example 1

[0078] like Figure 1 As shown, the specific steps of a cluster autoscaling method based on a near-end policy optimization algorithm in this embodiment include:

[0079] S101. Predict the throughput of each root microservice endpoint;

[0080] S102. Estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology;

[0081] S103. Wait for an event to trigger the cluster autoscaler to run;

[0082] S104. When the event is a scaling-down event, construct the environment state based on the current and future information of the microservice cluster, and use the trained near-end strategy to optimize the agent to obtain the action. Based on the action, determine whether the scaling-down decision of the node that needs to be scaled down is to put it into hibernation or to release it directly.

[0083] S105. Invoke the executor to execute the scaling-down decision.

[0084] In step S101, the root microservice endpoint refers to the API interface of the root microservice. The user-facing root microservice calls downstream microservices, and downstream microservices recursively call even more downstream microservices, forming a call chain. A root microservice has several API interfaces for user calls, and downstream microservices also have several API interfaces for upstream microservice calls. The throughput refers to the number of concurrent requests processed by a microservice endpoint per unit of time. The predicted throughput of each root microservice endpoint refers to the predicted throughput of each root microservice endpoint over a certain period of time in the future.

[0085] In this embodiment, step S101 predicts the throughput of each root microservice endpoint based on the Prophet time-series prediction algorithm. The specific steps include:

[0086] For a root microservice endpoint r∈R, where R represents the set of root microservice endpoints, let the current date be day d, and retrieve the historical throughput data for consecutive days D. in This represents the historical throughput data of the root microservice endpoint r for the 24 hours on day db, 0 <b<D;

[0087] Based on historical throughput data over D consecutive days and the Prophet algorithm, predict the throughput data for day d. in W represents the predicted throughput of the root microservice endpoint r at the predicted time point w on day d. The value of W represents the number of predicted time points, which depends on the time step scale T0. W = 24h / T0, where h represents hours.

[0088] For example, D=5 is set, which means using 5 consecutive days of historical throughput data. As input, and with the time step scale T0 = 5 minutes, then W = 288, yielding the throughput prediction data for day d.

[0089] In the specific implementation process, the specific steps of step S102 include:

[0090] S1021. Evaluate the throughput of all microservice endpoints in the corresponding call chain based on the predicted throughput of each root microservice endpoint and the microservice dependency topology; specifically:

[0091] Based on the microservice dependency topology, determine the upstream endpoint u of any microservice endpoint l, whose dependency relationship is θ. l,u θ l,u This represents the average number of times microservice endpoint l is called by endpoint u when endpoint u is called once; wherein the microservice dependency topology is built and maintained by a distributed tracing tool, describing the dependencies between microservice endpoints.

[0092] Based on dependency θ l,u Let the throughput of endpoint u at the predicted time point w be... The throughput obtained by microservice endpoint l from endpoint u at the predicted time point w is then... Calculated using the following formula:

[0093]

[0094] Total throughput of microservice endpoint l at the predicted time point w The calculation formula is as follows:

[0095]

[0096] Where U(l) represents all upstream endpoints of microservice endpoint l;

[0097] Based on formulas (1) and (2), the root microservice endpoint serves as the upstream endpoint in the corresponding call chain. According to the predicted throughput of each root microservice endpoint, the total throughput of all microservice endpoints in the call chain at each predicted time point w is recursively calculated starting from the root microservice endpoint.

[0098] In practical implementation, a microservice application includes multiple microservices, including a root microservice. The root microservice's API is available for user calls, while endpoints of non-root microservices are available for calls from other microservices. In this call chain, the upstream endpoint is the root microservice endpoint, which calls its downstream endpoints, and these downstream endpoints also call their respective downstream endpoints. Therefore, taking the root microservice endpoint as the upstream endpoint and using it as the starting point, the predicted throughput value of the root microservice endpoint r at the predicted time point w on day d is used. Recursively calculate the total throughput of all microservice endpoints in the entire call chain at each predicted time point w.

[0099] S1022. Calculate the execution time required for a microservice endpoint to complete a service request under a given throughput based on the throughput of all microservice endpoints in the corresponding call chain;

[0100] In practical implementation, an SVR-based performance model is used to estimate the execution time at a given throughput. This performance model represents the relationship between execution time and throughput; that is, given a certain throughput, the execution time can be predicted based on the performance model. The execution time refers to the processing time required for a microservice endpoint to complete a service request at a given throughput.

[0101] Specifically, a regression model of throughput versus execution time is established for any microservice endpoint l using an SVR-based performance model. Then the total throughput of microservice endpoint l at each predicted time point w will be calculated. Given the throughput, the execution time of microservice endpoint l is:

[0102]

[0103] S1023. Calculate the parallelism of each microservice based on the execution time and the throughput of all microservice endpoints on the corresponding call chain, and calculate the number of instances required for each microservice based on the parallelism.

[0104] Specifically, based on Little's Law, the microservice endpoint l, given a throughput and corresponding execution time The number of concurrent requests that need to be processed for:

[0105]

[0106] Calculate the parallelism p of each microservice s s,w :

[0107]

[0108] Where, microservices s∈S, S represents all microservices in the microservice application, and the degree of parallelism p s,w L(s) refers to the total number of concurrent requests processed by all microservice endpoints of a microservice s at the predicted time point w.

[0109] The number of instances n required for microservice s at various predicted time points w within a 24-hour period. s,w We obtain it from the following formula:

[0110]

[0111] Where, p smax The maximum concurrency of microservice s represents the maximum number of concurrent requests that a single instance of microservice s can handle simultaneously. It is set when the microservice application is deployed. The larger this value is, the stronger the application's concurrent processing capability.

[0112] Based on the above steps S101 and S102, the number of instances required for any microservice s at various points in time within a 24-hour period of the day can be predicted, i.e., N. s ={n s,1 ,...,n s,w ,...,n s,W}, where n s,w This represents the number of instances required for microservice s at the predicted time point w.

[0113] In step S103, the waiting event triggers the cluster autoscalor to run. The event includes two types of events: expansion event and shrinkage event, which correspond to the two cases of cluster expansion and shrinkage, respectively.

[0114] The scaling event refers to the need to create a new microservice instance but the cluster has no available nodes, that is, the remaining resources of the current cluster nodes are insufficient to run the microservice instance to be created; the scaling event triggers the cluster autoscalor to perform a scaling operation, that is, to increase the number of cluster nodes, such as incrementing the number of nodes by 1; at this time, the executor can be directly called to perform the scaling operation.

[0115] The scaling-down event refers to all microservice instances on a certain node being in a suspended state; the scaling-down event triggers the cluster autoscalor to perform a scaling-down operation. Therefore, when the event that occurs during the waiting event in step S103 that triggers the cluster autoscalor to run is a scaling-down event, step S104 is executed, and the cluster autoscalor will make a decision through a reinforcement learning agent, deciding whether to convert the node state to a dormant state or release it directly.

[0116] The node state includes four states: running, idle, sleeping, and released; for example... Figure 2 As shown, a node is in the running state when at least one microservice instance is running on it. When all microservice instances on a node are suspended, the node will enter the idle state, which is a transitional state. The cluster autoscalorie will decide whether to enter the dormant state or release it directly. The dormant state means that the node will release some CPU core resources. The release state means that the node will be deleted, that is, release all resources owned by the node. In addition, when at least one microservice instance on a node in the dormant state is activated, the node will return to the running state.

[0117] The suspended state of the microservice instance is one of the states in the microservice instance state transition, such as... Figure 3 As shown, when a microservice instance is created, it first undergoes initialization. After completing operations such as pulling the instance image and application initialization, the instance starts normally and enters the running state. In this state, the instance exposes its endpoints, i.e., API interfaces, for users or other instances to call. When the microservice autoscalor determines that the instance needs to be scaled down, it will enter the suspended state by hiding its microservice instance endpoints, making it unable to receive requests. When the instance needs to be scaled up, the suspended instances are activated first, returning them to the running state. When the node where the instance resides is deleted, all suspended instances on the node will enter the terminated state, meaning the instance will release all its resources.

[0118] It should be noted that in this embodiment, the scaling of microservice instances is accomplished using Kubernetes' Horizontal Pod Autoscaler (HPA); the HPA adjusts the number of microservice instances using the following formula:

[0119]

[0120] Where n current n represents the current number of microservice instances. target Metric represents the target number of microservice instances. current This indicates the current metric value. target This represents the target metric value; in this embodiment, the relevant metric value is CPU utilization.

[0121] In step S104, the environment state is constructed based on the current and future information of the microservice cluster, specifically including the following steps:

[0122] The environment status includes two parts: the current status and the expected status, which correspond to the current and future instance count information of the microservice cluster, respectively.

[0123] The instantaneous state represents the number of instances of each microservice in the cluster that are in the running state and the suspended state at the current time t, expressed as:

[0124]

[0125] in, This indicates the number of instances of microservice s that are currently running. This represents the number of microservice s instances that are in a suspended state; S is the total number of microservices.

[0126] The expected state represents the number of instances required for each microservice at time t+Δt, which is based on the number of instances n required for microservice s at each prediction time point w within the 24 hours of that day. s，w Get, represented as in This represents the number of instances required by microservice s at time t+Δt corresponding to the predicted time point w; if Δt = 20min can be set, then the expected state represents the number of instances required by each microservice 20 minutes after the current time.

[0127] Combining the immediate state and the expected state, the environmental state at time t can be expressed as:

[0128] Furthermore, the action is the action performed by the agent in a given state s. t The output below is represented as a t It is a one-hot vector, when When the value is 1, the node enters a sleep state; otherwise, the node is released directly, such as 'a'. t ={1,0} indicates that the node will enter a sleep state;

[0129] Furthermore, to characterize the quality of an agent's actions, a reward needs to be given after the agent performs the action, based on the objective. The goal of the cluster autoscalorie is to reduce the CPU overhead of business logic-independent processes such as microservice instance initialization, node startup, and node hibernation, in order to maximize cluster energy efficiency. Therefore, the reward is set as the reward from the execution of action a. t Start T rThe CPU time used by the node startup process, instance initialization process, and node sleep process within a given time period; assuming that M nodes are started, N microservice instances are created, and O nodes enter sleep state within this time period, then action a t Rewards The calculation formula is:

[0130]

[0131] in, The total CPU time used by the node startup process can be calculated as follows: This represents the average CPU utilization of node i during startup. Indicates the startup time of node i; This represents the total CPU time used by the instance initialization process, which can be calculated as follows: This represents the average CPU utilization of microservice instance j during initialization. Indicates the initialization time of microservice instance j; The total CPU time used during node sleep can be calculated as follows: This represents the average CPU utilization of node m during sleep. This represents the sleep time of node m. The hyperparameter α is used to adjust the optimization direction of the agent. When α = 1, the agent will optimize the strategy to minimize CPU overhead; when α > 1, it will tend to put nodes to sleep; when α < 1, it will tend to release nodes directly.

[0132] Furthermore, for the training of the agent, a proximal policy optimization algorithm is used to update the agent's network parameters. This algorithm optimizes the policy based on an Actor-Critic architecture and uses a deep neural network to fit the Actor and Critic networks. The network parameters of the Actor and Critic are defined as θ and φ, respectively, and the agent's policy network is defined as π. θ The agent objective function of the Actor network is then defined as:

[0133]

[0134] in Representing the new strategy π θ and old strategies In state s t Next, execute action a t The probability ratio can be calculated as follows: clip represents the clipping function; A tRepresents state-action pairs (s) t a t The advantage estimate can be calculated as follows: V φ (s t ) is the output of the Critic network, i.e., the state value estimate; ε is a hyperparameter that determines the scope of policy update pruning.

[0135] The loss function of the Critic network is defined as V φ (s t )and The mean square error, i.e.:

[0136]

[0137] In each episode, the agent collects T e The state, action, and reward information of each timestamp are used as the trajectory. The stochastic gradient ascent algorithm is used to update the parameter θ to maximize the direction of the Actor network agent target. k+1 =argmax θ L P (θ); The parameter φ is updated using the stochastic gradient descent algorithm in the direction that minimizes the mean square error, i.e., φ k+1 =argmin φ L VF (φ); If the learning curve of the agent is converged through iterative updates of E episodes, the policy network can be applied to the decision-making of the cluster autoscaler. Figure 4 The figure shows the learning curve of the agent. As can be seen from the figure, in this embodiment, a total of 5000 iterations of training were performed, and the cumulative reward value of the agent per episode basically converged after about 1500 iterations.

[0138] In step S105, based on the scaling-down decision of the cluster autoscalor, the executor is invoked to execute the corresponding strategy. Specifically, if the event awaited by the cluster autoscalor in step S103 is a scaling-up event, step S105 is executed directly. The executor executes the cluster autoscalor's decision to add nodes and starts the nodes by calling the virtual machine management platform interface (VMPA). In this embodiment, the VMPA is the OpenStack nova API. However, if the event awaited by the cluster autoscalor in step S103 is a scaling-down event, based on the scaling-down strategy obtained in step S104, the executor parses the actions output by the cluster autoscalor agent. It either uses CPU hot-plugging technology to reclaim virtual CPU cores to put the node to sleep or calls the OpenStack nova API to release the node. For the latter, the executor is also responsible for releasing all suspended microservice instances on that node. After the executor completes the corresponding operations, step S103 is restarted, waiting for the event to trigger the cluster autoscalor to run.

[0139] The method in this embodiment takes into account the operation of dormant nodes in addition to the node state. It also considers the CPU time of processes such as node startup, instance initialization, and node dormancy. It adjusts the trade-off between dormant nodes and released nodes by weighted summation, and uses this as a reward function to minimize CPU overhead and improve cluster energy efficiency as much as possible.

[0140] Example 2

[0141] like Figure 5 As shown, this embodiment also provides a cluster autoscaling system based on a near-end policy optimization algorithm, including:

[0142] Throughput prediction module 201 is used to predict the throughput of each root microservice endpoint;

[0143] The instance count calculation module 202 is used to estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology.

[0144] Trigger module 203 is used to wait for events to trigger the cluster autoscaler to run;

[0145] The scaling-down event decision module 204 is used to construct the environment state based on the current and future information of the microservice cluster when the event is a scaling-down event, and to optimize the agent using the trained near-end strategy to obtain the action, and to determine the scaling-down decision of the node that needs to be scaled down, whether to put it into hibernation or to release it directly.

[0146] The execution module 205 is used to call the executor to execute the scaling-down decision.

[0147] This embodiment of the system additionally considers the operation of dormant nodes when dealing with node states, and effectively takes into account the current and future resource requirements of microservices. By training the agent, it obtains the optimal strategy, solving the challenge of manually determining whether dormant or released nodes are more energy-efficient when scaling down the cluster, thereby minimizing CPU overhead and improving cluster energy efficiency. Obviously, the above embodiments of the present invention are merely examples to clearly illustrate the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A cluster autoscaling method based on a near-end strategy optimization algorithm, characterized in that, include: Predict the throughput of each root microservice endpoint; Estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology. Wait for an event to trigger the cluster autoscaler to run; When the event is a scaling down event, the environment state is constructed based on the current and future information of the microservice cluster, and the trained near-end strategy is used to optimize the agent to obtain the action. Based on the action, the scaling down decision of the node that needs to be scaled down is determined to be either put into hibernation or released directly. The executor is invoked to execute the scaling-down decision; The action is the action of the agent in a given state. The output below is represented as ; It is a one-hot vector, when When the value is 1, the node enters a sleep state; otherwise, the node is released directly. A reward is given after the agent performs an action, based on the objective; the reward is set from the date the action is performed. rise The CPU time used by the node startup process, instance initialization process, and node sleep process within a given time period; assuming there are [number] CPU times used within this time period. One node was started. A microservice instance is created. If a node enters a sleep state, then the action... Rewards The calculation formula is: in, This indicates the total CPU time used by the node startup process. , Represents a node Average CPU utilization during startup Represents a node Startup time; This indicates the total CPU time used by the instance initialization process. , Represents a microservice instance Average CPU utilization during initialization Represents a microservice instance Initialization time; This represents the total CPU time used during the node's sleep process. , Represents a node Average CPU utilization during sleep. Represents a node Sleep time, Hyperparameters Used to adjust the optimization direction of the agent, when The agent will optimize its strategy to minimize CPU overhead; when At that time, it will tend to sleep at the node; when The node will tend to be released directly at that time; The process of optimizing the agent using a trained proximal policy to obtain actions specifically includes: The proximal policy optimization algorithm is used to update the agent network parameters. This algorithm optimizes the policy based on the Actor-Critic architecture and uses a deep neural network to fit the Actor and Critic networks. The network parameters for the Actor and Critic are defined as follows: and Define the agent's policy network as The agent objective function of the Actor network is then defined as: in Indicating a new strategy and old strategies In state Next action The probability ratio, ;clip represents the clipping function; Represents state-action pairs The advantage estimate, ; It is the output of the Critic network, i.e., the state value estimate; It is a hyperparameter that determines the scope of the clipping update in the strategy; The loss function of the Critic network is defined as follows: and The mean square error, i.e.: In each episode, the agent collects... The state, action, and reward information of each timestamp are used as the trajectory, and the stochastic gradient ascent algorithm is used to update the parameters to maximize the direction of the Actor network agent target. ,Right now Update parameters using the stochastic gradient descent algorithm in the direction that minimizes the mean square error. ,Right now Iterative updates Once the learning curve of an agent converges in a given episode, the policy network is applied to the decision-making of the cluster autoscaler.

2. The cluster autoscaling method based on a near-end strategy optimization algorithm according to claim 1, characterized in that, Predicting the throughput of each root microservice endpoint, specifically including: For root microservice endpoints , Let the current date be the set of root microservice endpoints. Daily, obtain continuous Historical data of daily throughput ,in Represents the root microservice endpoint In the Historical daily throughput data ; According to the continuous Daily throughput historical data prediction Daily throughput data ,in Represents the root microservice endpoint In the Predicted time of day The predicted throughput value, The value represents the number of prediction time points, which depends on the time step scale. , , Indicates hours.

3. The cluster autoscaling method based on a near-end strategy optimization algorithm according to claim 2, characterized in that, Estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology, specifically including: The throughput of all microservice endpoints in the corresponding call chain is evaluated based on the predicted throughput of each root microservice endpoint and the microservice dependency topology. Calculate the execution time required for a microservice endpoint to complete a service request at a given throughput based on the throughput of all microservice endpoints in the corresponding call chain; The parallelism of each microservice is calculated based on the execution time and the throughput of all microservice endpoints in the corresponding call chain, and the number of instances required for each microservice is calculated based on the parallelism.

4. The cluster autoscaling method based on a near-end strategy optimization algorithm according to claim 3, characterized in that, The process of evaluating the throughput of all microservice endpoints in the corresponding call chain based on the predicted throughput of each root microservice endpoint and the microservice dependency topology specifically includes: Based on the microservice dependency topology, determine the upstream endpoint u of any microservice endpoint l, and its dependency relationship is as follows: , This represents the average number of times microservice endpoint l is called by endpoint u when endpoint u is called once; Based on dependency relationships Let the throughput of endpoint u at the predicted time point w be... The throughput obtained by microservice endpoint l from microservice endpoint u at the predicted time point w is... Calculated using the following formula: Total throughput of microservice endpoint l at the predicted time point w The calculation formula is as follows: in, Let represent all upstream endpoints of microservice endpoint l; based on formulas (1) and (2), the root microservice endpoint is the upstream endpoint on the corresponding call chain. According to the predicted throughput of each root microservice endpoint, the total throughput of all microservice endpoints on the call chain at each predicted time point w is recursively calculated starting from the root microservice endpoint. .

5. A cluster autoscaling method based on a near-end strategy optimization algorithm according to claim 4, characterized in that, The step of calculating the execution time required for each microservice endpoint to complete a service request under a given throughput based on the throughput of all microservice endpoints in the corresponding call chain specifically includes: Use an SVR-based performance model to build a regression model of throughput versus execution time for any microservice endpoint. Then the total throughput of microservice endpoint l at each predicted time point w will be calculated. Given the throughput, the execution time of microservice endpoint l is:

6. A cluster autoscaling method based on a near-end strategy optimization algorithm according to claim 5, characterized in that, The step of calculating the parallelism of each microservice based on the execution time and the throughput of all microservice endpoints in the corresponding call chain, and calculating the number of instances required for each microservice based on the parallelism, specifically includes: Based on Little's Law, microservice endpoint l, given throughput and corresponding execution time The number of concurrent requests that need to be processed for: Calculate the parallelism ps,w for each microservice s: Among them, microservices , This represents all the microservices in a microservice application, and the degree of parallelism... This refers to the total number of concurrent requests handled by all microservice endpoints of a microservice s at the predicted time point w. This represents all microservice endpoints of microservice s; The number of instances required for microservice s at various predicted time points within a 24-hour period of the day. We obtain it from the following formula: in, The maximum concurrency of a microservice s represents the maximum number of concurrent requests that a single instance of a microservice s can handle simultaneously. It is set when the microservice application is deployed.

7. A cluster autoscaling method based on a near-end strategy optimization algorithm according to claim 6, characterized in that, The construction of the environment state based on the current and future information of the microservice cluster specifically includes: Node status includes four states: running, idle, sleeping, and released. The idle state indicates that all microservice instances on the node are in a suspended state, which is a transitional state. The scaling down event refers to all microservice instances on a certain node being in a suspended state. The suspended state of a microservice instance is a state in the microservice activation and suspension mechanism. It means that when a microservice needs to be scaled down, it is suspended by hiding its microservice instance endpoint, so that it cannot receive requests. The environment status includes two parts: the current status and the expected status, which correspond to the current and future instance count information of the microservice cluster, respectively. The instantaneous state represents the number of instances of each microservice in the cluster that are in the running state and the suspended state at the current time t, expressed as: in, microservices The number of instances in the running state. microservices The number of instances in a suspended state; The total number of microservices; The expected state indicates The number of instances required for each microservice at any given time is predicted, based on the microservice. On that day Each predicted time point Number of instances required Get, represented as ,in , indicating microservices exist The time corresponds to the predicted time point The required number of instances; Combining the current state and the expected state, at time The environmental state can be represented as .

8. A cluster autoscaling system based on a near-end policy optimization algorithm, characterized in that, include: The throughput prediction module is used to predict the throughput of each root microservice endpoint. The instance count calculation module is used to estimate the number of instances required for all microservices based on the predicted throughput of each root microservice endpoint and the microservice dependency topology. The trigger module is used to wait for events to trigger the cluster autoscaler to run; The scaling-down event decision module is used to construct the environment state based on the current and future information of the microservice cluster when the event is a scaling-down event, and to optimize the agent using the trained near-end strategy to obtain the action. Based on the action, it determines whether the scaling-down decision of the node that needs to be scaled down is to put it into hibernation or to release it directly. The execution module is used to call the executor to execute the scaling-down decision; The action is the action of the agent in a given state. The output below is represented as ; It is a one-hot vector, when When the value is 1, the node enters a sleep state; otherwise, the node is released directly. A reward is given after the agent performs an action, based on the objective; the reward is set from the date the action is performed. rise The CPU time used by the node startup process, instance initialization process, and node sleep process within a given time period; assuming there are [number] CPU times used within this time period. One node was started. A microservice instance is created. If a node enters a sleep state, then the action... Rewards The calculation formula is: in, This indicates the total CPU time used by the node startup process. , Represents a node Average CPU utilization during startup Represents a node Startup time; This indicates the total CPU time used by the instance initialization process. , Represents a microservice instance Average CPU utilization during initialization Represents a microservice instance Initialization time; This represents the total CPU time used during the node's sleep process. , Represents a node Average CPU utilization during sleep. Represents a node Sleep time, Hyperparameters Used to adjust the optimization direction of the agent, when The agent will optimize its strategy to minimize CPU overhead; when At that time, it will tend to sleep at the node; when The node will tend to be released directly at that time; The step of optimizing the agent's actions using a trained proximal policy specifically includes: updating the agent's network parameters using a proximal policy optimization algorithm; the proximal policy optimization algorithm optimizes the policy based on an Actor-Critic architecture and uses a deep neural network to fit the Actor and Critic networks; and defining the network parameters of the Actor and Critic as follows: and Define the agent's policy network as The agent objective function of the Actor network is then defined as: in Indicating a new strategy and old strategies In state Next action The probability ratio, ;clip represents the clipping function; Represents state-action pairs The advantage estimate, ; It is the output of the Critic network, i.e., the state value estimate; It is a hyperparameter that determines the scope of pruning in policy updates; the loss function of the Critic network is defined as... and The mean square error, i.e.: In each episode, the agent collects... The state, action, and reward information of each timestamp are used as the trajectory, and the stochastic gradient ascent algorithm is used to update the parameters to maximize the direction of the Actor network agent target. ,Right now Update parameters using the stochastic gradient descent algorithm in the direction that minimizes the mean square error. ,Right now Iterative updates Once the learning curve of an agent converges in a given episode, the policy network is applied to the decision-making of the cluster autoscaler.

Images