Scheduling method and device based on microservice link analysis and reinforcement learning

Abstract

The present invention relates to a scheduling method and device based on microservice link analysis and reinforcement learning. The method and device schedule cloud server clusters based on a deep learning model, and reinforcement learning can use three different scheduling methods: horizontal expansion, vertical expansion, and control. The present invention uses a workload and link analyzer based on a deep learning model to analyze and make decisions on the links of microservices, and selects the critical paths and key nodes with the longest delay. The present invention trains cloud server load data based on the reinforcement learning algorithm of Deep Q-Learning, and uses the algorithm to train a deep learning model that is suitable for different load conditions. The present invention solves the problem of possible mismatch between link length and microservice delay, and can obtain an optimization target for the delay itself. Scheduling resources for cloud servers based on this can effectively alleviate the existing microservice link delay problem.

Description

Technical Field

[0001] The present invention relates to a scheduling method and device based on microservice link analysis and reinforcement learning. Background Art

[0002] The microservice scheduling algorithm for cloud servers has become a key indicator of their load capacity under highly variable loads. An excellent microservice link scheduling algorithm can reduce server resource waste for cloud server providers, significantly improve the user experience of cloud server users, enhance cloud server stability, and more rationalize resource allocation for microservices within the cloud server, thereby enhancing the robustness of microservices across the entire cloud server cluster. Currently, mainstream cloud server microservice link delay scheduling algorithms include resource allocation scheduling algorithms and longest-link-based microservice scheduling algorithms. Among these:

[0003] (1) Resource Allocation Scheduling Algorithm: The resource allocation scheduling algorithm utilizes the preset resources of the cloud server microservices to perform allocation. This algorithm can achieve good results when resources are sufficient. This resource allocation method can minimize resource competition between microservices and improve the stability of the cloud server to a certain extent.

[0004] (2) Microservice scheduling algorithm based on the longest link: The microservice scheduling algorithm based on the longest link uses the link relationship between microservices to allocate resources. For microservices in the cluster, by analyzing their link relationship, the longest link or links can be obtained, and resources are allocated based on this standard. All microservices on the microservice link with the longest link will be given priority in allocating resources, and resources will be allocated in order from the longest link to the shortest link. This resource allocation method can alleviate resource competition between microservices to a certain extent, give priority to providing more resources to longer links, and reduce the overall time delay of microservices.

[0005] However, while many existing forecasting methods, such as regression, can provide simple time series forecasts, they struggle to address the current high-dimensional cloud server workload forecasting problem. While recurrent neural networks and long-short-term memory (LSTM) can relatively accurately predict server loads in the short term, their structure and limitations make them less accurate for workload predictions over a period of time. LSTM also takes a long time to predict, making it less timely.

[0006] At the same time, while many existing mainstream microservice resource scheduling methods can simply schedule microservices when resources are sufficient, they struggle to cope with resource competition in cloud server clusters, which are highly sensitive to latency. According to resource allocation scheduling algorithms, in cloud server clusters, dozens of containers often run simultaneously on a single node, resulting in microservice resources not being allocated according to pre-set resources, leading to resource competition. This scheduling algorithm is not suitable for the current cloud server cluster architecture. According to the microservice scheduling algorithm based on the longest link, if the key node of a microservice is not on the longest link, resources cannot be correctly allocated to the key node, resulting in the inability to promptly resolve the microservice latency problem in the cloud server. Summary of the Invention

[0007] The embodiments of the present invention provide a scheduling method and device based on microservice link analysis and reinforcement learning to at least solve the technical problem of microservice delay in existing cloud servers.

[0008] According to one embodiment of the present invention, a scheduling method based on microservice link analysis and reinforcement learning is provided, comprising the following steps:

[0009] The workload and link analyzer based on the deep learning model analyzes and makes decisions on the links of cloud server microservices, selects the key paths and key nodes with the longest delay, and obtains cloud server load data;

[0010] Based on the Deep Q-Learning reinforcement learning algorithm, we train the cloud server load data and use it to train a deep learning model that adapts to different load conditions.

[0011] Use deep learning models to schedule cloud server clusters, including horizontal scaling, vertical scaling, and control.

[0012] Furthermore, the workload and link analyzer based on the deep learning model analyzes and makes decisions on the links of the cloud server microservices, selects the key paths and key nodes with the longest delay, and obtains cloud server load data including:

[0013] Preprocess the workload and clean the data, and process the raw data obtained from the actual data set to obtain the number of requests;

[0014] After collecting the relationship between the number of requests and the workload of the cloud server, we used a deep learning algorithm to obtain the data on the number of requests required for the entire experiment.

[0015] After generating a request and sending it to the request processor, it sends the request to the target cluster and collects load information from the cloud server cluster.

[0016] The load information is passed to the workload and link analyzer. Through the critical path analysis of the microservices, the link relationship of the microservices in the current state is obtained. Combined with the obtained link delay data, the critical path of the microservices is obtained.

[0017] The key nodes are judged and the results are obtained through the classification method based on the decision tree.

[0018] Furthermore, the workload is preprocessed and data cleansed, and the raw data obtained from the actual data set is processed to obtain the number of requests including:

[0019] First, we remove the columns containing empty data, classify the dataset into time series, and then use the grouping function to calculate the average value of each parameter with the same timestamp;

[0020] Use MinMaxScaler to normalize the data set. The operation of MinMaxScaler is based on the min-max scaling method. The specific formulas are shown in Formula 1 and Formula 2 below.

[0021]

[0022] X scaled =X std *(X max -X min )+X min (2)

[0023] Use MinMaxScaler to transform each feature and scale each feature to a value between 0 and 1. X represents the set of data to be processed. min and X max is the minimum and maximum data in the set, and the final processed data is represented by X scaled express.

[0024] Furthermore, after collecting the relationship between the number of requests and the workload of the cloud server, and obtaining data on the number of requests required for the entire experiment through a deep learning algorithm, the method further includes:

[0025] User behavior is analyzed, and based on the user behavior, the different locations where the user visits the website are recorded. By simulating different requests to visit the website, the user behavior in real life is simulated.

[0026] Furthermore, the load situation includes: the delay of the microservice itself, the resource consumption of the cloud server, the request delay of the microservice, and the request success rate information.

[0027] Furthermore, the key nodes are judged by the decision tree-based classification method and the results obtained include:

[0028] By training dynamically changing link relationships, a decision tree model is obtained to determine whether a node is a key node. The decision tree has three decision results: key node, non-key node, and potential key node. Among the three results, each key path includes at least one key node, and non-key nodes and potential key nodes do not exist in the key path.

[0029] Furthermore, a reinforcement learning algorithm based on Deep Q-Learning is used to train cloud server load data. Deep learning models adapted to different load conditions are trained using the reinforcement learning algorithm based on Deep Q-Learning, including:

[0030] After determining the location of key nodes, scheduling decisions for the cloud server cluster will be made through a trained deep learning model; reinforcement learning uses Deep Q-Learning based on a deep learning neural network.

[0031] Furthermore, the model is trained by calculating the Q-value after a series of actions. The calculation equation is shown in Formula 3:

[0032]

[0033] The Q-value generated by being in state s and performing action a is the reward r(s,a) plus the highest Q-value generated by the next state s'; γ is the discount, which controls the impact of future rewards on the Q-value of the current state;

[0034] Iterate the equation in Formula 3 above to obtain Formula 4, which is the equation in the final convergence case:

[0035] Q*(s,a)=∑ s′ P(s′∣s,a)(R(s,a,s′)+γmax a′ Q*(s′,a′)) (4)

[0036] The part of Equation 4 that stores the past Q-value is obtained as Equation 5:

[0037] γmax a′ Q*(s′,a′) (5).

[0038] Furthermore, horizontal scaling adjusts resources by adding or removing replicas of microservices on cloud servers within the cluster; cloud server clusters reduce the workload of the cloud server cluster by adding replicas of microservices;

[0039] Vertical scaling adjusts the processing capacity of the current cloud server by adjusting the amount of CPU, memory, or network resources allocated to microservices. Cloud server clusters reduce the workload of cloud server clusters by increasing the resources of a single cloud server or multiple cloud servers within the cluster for a single or multiple microservices.

[0040] Control reduces microservice link latency by dynamically turning on or off optional components. When cloud services and clusters transition from extreme conditions to normal working conditions, the turned-off optional components will be turned back on by control.

[0041] According to another embodiment of the present invention, a scheduling device based on microservice link analysis and reinforcement learning is provided, comprising:

[0042] The load generator is used to analyze and make decisions on the links of cloud server microservices based on the workload and link analyzer of the deep learning model, select the critical path and key nodes with the longest delay, and obtain cloud server load data;

[0043] Workload and Link Analyzer, which uses Deep Q-Learning-based reinforcement learning algorithms to train cloud server load data and develop deep learning models that adapt to different load conditions.

[0044] The cluster scheduler is used to schedule cloud server clusters using deep learning models. The scheduling methods include horizontal expansion, vertical expansion, and control.

[0045] A storage medium storing a program file capable of implementing any of the above-mentioned scheduling methods based on microservice link analysis and reinforcement learning.

[0046] A processor is used to run a program, wherein when the program is running, any one of the above-mentioned scheduling methods based on microservice link analysis and reinforcement learning is executed.

[0047] The scheduling method and device based on microservice link analysis and reinforcement learning in the embodiment of the present invention schedules the cloud server cluster based on a deep learning model. Reinforcement learning can use three different scheduling methods: horizontal expansion, vertical expansion and control. The present invention uses a workload and link analyzer based on a deep learning model to analyze and make decisions on the links of microservices, and selects the key paths and key nodes with the longest delay. The present invention trains cloud server load data based on the reinforcement learning algorithm of Deep Q-Learning, and uses the algorithm to train a deep learning model that is suitable for different load conditions. The present invention uses the maximum delay link in the microservice link to determine the position of the key node in the link. Compared with the microservice scheduling algorithm based on the longest link, the present invention solves the problem of possible mismatch between link length and microservice delay, and can obtain an optimization target for the delay itself. Based on this, resource scheduling of cloud servers can effectively alleviate the existing microservice link delay problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 This is a system model diagram of the scheduling method based on microservice link analysis and reinforcement learning of the present invention;

[0050] Figure 2 This is a diagram of three scheduling strategies based on microservice link analysis and reinforcement learning in the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0052] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] Example 1

[0054] According to one embodiment of the present invention, a scheduling method based on microservice link analysis and reinforcement learning is provided, comprising the following steps:

[0055] The workload and link analyzer based on the deep learning model analyzes and makes decisions on the links of cloud server microservices, selects the key paths and key nodes with the longest delay, and obtains cloud server load data;

[0056] Based on the Deep Q-Learning reinforcement learning algorithm, we train the cloud server load data and use it to train a deep learning model that adapts to different load conditions.

[0057] Use deep learning models to schedule cloud server clusters, including horizontal scaling, vertical scaling, and control.

[0058] The scheduling method based on microservice link analysis and reinforcement learning in the embodiment of the present invention schedules the cloud server cluster based on a deep learning model. Reinforcement learning can use three different scheduling methods: horizontal expansion, vertical expansion and control. The present invention uses a workload and link analyzer based on a deep learning model to analyze and make decisions on the links of microservices, and selects the key paths and key nodes with the longest delay. The present invention trains the cloud server load data based on the reinforcement learning algorithm of Deep Q-Learning, and uses the algorithm to train a deep learning model that is suitable for different load conditions. The present invention uses the maximum delay link in the microservice link to determine the position of the key node in the link. Compared with the microservice scheduling algorithm based on the longest link, the present invention solves the problem of possible mismatch between link length and microservice delay, and can obtain an optimization target for the delay itself. Based on this, resource scheduling of the cloud server can effectively alleviate the existing microservice link delay problem.

[0059] The workload and link analyzer based on the deep learning model analyzes and makes decisions on the links of cloud server microservices, selects the key paths and key nodes with the longest delay, and obtains cloud server load data including:

[0060] Preprocess the workload and clean the data, and process the raw data obtained from the actual data set to obtain the number of requests;

[0061] After collecting the relationship between the number of requests and the workload of the cloud server, we used a deep learning algorithm to obtain the data on the number of requests required for the entire experiment.

[0062] After generating a request and sending it to the request processor, it sends the request to the target cluster and collects load information from the cloud server cluster.

[0063] The load information is passed to the workload and link analyzer. Through the critical path analysis of the microservices, the link relationship of the microservices in the current state is obtained. Combined with the obtained link delay data, the critical path of the microservices is obtained.

[0064] The key nodes are judged and the results are obtained through the classification method based on the decision tree.

[0065] Among them, the workload is preprocessed and data cleansed, and the raw data obtained from the actual data set is processed to obtain the number of requests including:

[0066] First, we remove the columns containing empty data, classify the dataset into time series, and then use the grouping function to calculate the average value of each parameter with the same timestamp;

[0067] Use MinMaxScaler to normalize the data set. The operation of MinMaxScaler is based on the min-max scaling method. The specific formulas are shown in Formula 1 and Formula 2 below.

[0068]

[0069] X scaled =X std *(X max -X min )+X min (2)

[0070] Use MinMaxScaler to transform each feature and scale each feature to a value between 0 and 1. X represents the set of data to be processed. min and X max is the minimum and maximum data in the set, and the final processed data is represented by X scaledexpress.

[0071] After collecting the relationship between the number of requests and the workload of the cloud server, and obtaining data on the number of requests required for the entire experiment through a deep learning algorithm, the method further includes:

[0072] User behavior is analyzed, and based on the user behavior, the different locations where the user visits the website are recorded. By simulating different requests to visit the website, the user behavior in real life is simulated.

[0073] The load conditions include: the delay of the microservice itself, the resource consumption of the cloud server, the request delay of the microservice, and the request success rate information.

[0074] Among them, the key nodes are judged by the classification method based on the decision tree and the results obtained include:

[0075] By training dynamically changing link relationships, a decision tree model is obtained to determine whether a node is a key node. The decision tree has three decision results: key node, non-key node, and potential key node. Among the three results, each key path includes at least one key node, and non-key nodes and potential key nodes do not exist in the key path.

[0076] Among them, the reinforcement learning algorithm based on Deep Q-Learning is used to train cloud server load data. The deep learning models trained by the reinforcement learning algorithm based on Deep Q-Learning that adapt to different load conditions include:

[0077] After determining the location of key nodes, scheduling decisions for the cloud server cluster will be made through a trained deep learning model; reinforcement learning uses Deep Q-Learning based on a deep learning neural network.

[0078] The model is trained by calculating the Q-value after a series of actions. The calculation equation is shown in Formula 3:

[0079]

[0080] The Q-value generated by being in state s and performing action a is the reward r(s,a) plus the highest Q-value generated by the next state s'; γ is the discount, which controls the impact of future rewards on the Q-value of the current state;

[0081] Iterate the equation in Formula 3 above to obtain Formula 4, which is the equation in the final convergence case:

[0082] Q*(s,a)=∑ s′ P(s′∣s,a)(R(s,a,s′)+γmax a′ Q*(s′,a′)) (4)

[0083] The part of Equation 4 that stores the past Q-value is obtained as Equation 5:

[0084] γmax a′ Q*(s′,a′) (5).

[0085] Among them, horizontal expansion adjusts resources by adding or removing replicas of microservices on cloud servers within the cluster; cloud server clusters reduce the workload of cloud server clusters by adding replicas of microservices;

[0086] Vertical scaling adjusts the processing capacity of the current cloud server by adjusting the amount of CPU, memory, or network resources allocated to microservices. Cloud server clusters reduce the workload of cloud server clusters by increasing the resources of a single cloud server or multiple cloud servers within the cluster for a single or multiple microservices.

[0087] Control reduces microservice link latency by dynamically turning on or off optional components. When cloud services and clusters transition from extreme conditions to normal working conditions, the turned-off optional components will be turned back on by control.

[0088] The scheduling method based on microservice link analysis and reinforcement learning of the present invention is described in detail below with specific embodiments:

[0089] As an integral component of modern IT technology, user-oriented cloud computing services offer numerous benefits. User-oriented microservices are typically supported by distributed cloud server clusters. When using these services, users are often highly sensitive to server link latency, as server link latency directly impacts the server's service quality, impacting the user experience. Link latency within a microservice cluster is caused by resource competition among microservices within the cluster. To address the issue of high link latency for microservices within cloud servers, this paper proposes an efficient and intelligent scheduling method based on reinforcement learning that performs online adjustments based on varying cloud server loads. This method uses methods including horizontal and vertical scaling to reallocate resources within the server. The purpose of this paper is to utilize a server load scheduling algorithm to address the impact of link latency on microservices within cloud servers. This paper utilizes the maximum-latency link within a microservice link to determine the location of key nodes within the link. Compared to microservice scheduling algorithms based on the longest link, this paper addresses the potential mismatch between link length and microservice latency, generating an optimization target for latency itself. This optimization target allows cloud server resource scheduling to effectively mitigate existing microservice link latency issues.

[0090] To achieve the above objectives, the present invention schedules cloud server clusters based on deep learning models. Reinforcement learning can use three different scheduling methods: horizontal expansion, vertical expansion, and control. The present invention uses a workload and link analyzer based on a deep learning model to analyze and make decisions on the links of microservices, and selects the links (critical paths) and nodes (critical nodes) with the longest delay. The present invention is based on the reinforcement learning algorithm of Deep Q-Learning, trains cloud server load data, and uses the algorithm to train a deep learning model that is suitable for different load conditions. Specifically:

[0091] (1) Data source introduction

[0092] The first set of data sets used in the present invention is the workload data set cluster-trace-v2018 of the cloud server cluster from Alibaba in 2018. This data set contains workload data spanning 8 days, which is generated by 4034 isomorphic servers. Each of the servers has 96 CPU cores, and their memory size is also the same. The present invention selected machine_usage.csv from the data set for analysis. The machine_usage.csv data set includes parameters such as machine number machine_id, timestamp time_stamp, CPU utilization cpu_util_percent of the current machine, and memory usage mem_util_percent of the machine at the current moment. The time interval for recording data is 10 seconds. The present invention removes some reference variables that often have null values, and assigns 0 to the null values of reference variables that occasionally have null values.

[0093] The present invention uses two experimental platforms, one of which is a microservice platform based on SockShop. SockShop simulates the user-facing part of an e-commerce website that sells socks. The purpose is to help users demonstrate and test microservices and cloud-native technologies. SockShop's microservices are designed to have a minimum estimated resource allocation, that is, the quota of each microservice is as small as possible during initialization. DNS is used between microservices to find other related microservices. When scheduling the cloud server for testing, a load balancer or service router can be inserted into the entire microservice framework according to the needs of the present invention. The SockShop version used in the present invention is based on the official version of Kubernetes, which includes a Jaeger-based microservice monitoring component in addition to the basic microservice components.

[0094] Another experimental platform used in this paper is train-ticket, a train ticket booking system based on a microservices architecture, comprising 41 microservices. The programming languages and frameworks used include Java, Python, and Go. train-ticket simulates the front-end and back-end of an online ticketing website, including functions such as ticket purchase, ticket sales, ticket query, ticket refund, and login. It also includes a Jaeger microservice detection component for monitoring system status and a workload balancer for controlling the overall stability of the microservices.

[0095] (2) Concept Definition

[0096] The system model of the present invention (such as Figure 1 The system (as shown in Figure 2) includes the following components: load generator, workload and link analyzer, and cluster scheduler.

[0097] Historical workload: Historical workload refers to the workload collected regularly from a running cloud server cluster by a specific server or built-in program. It mainly includes information such as timestamp, machine number, CPU utilization, and memory usage. In this paper, historical workload comes from the Alibaba dataset.

[0098] Load Generator: In our model, the load generator is comprised of a series of components, including workloads and processors, a Locust-based request generator, and a database storing historical workloads. These components coordinate to extract and process the required workload data and features from the historical database. The load generator then simulates real user usage and converts these into requests that are sent to the server cluster.

[0099] Workload and Link Analyzer: The main task of the workload and link analyzer is to analyze the requests sent from the cloud server and perform link analysis based on the link relationship of the microservices to identify the key nodes and critical paths. After that, the cluster scheduler with the deep learning / deep learning model deployed sends scheduling requests to the microservices of the cloud server cluster.

[0100] Cluster Scheduler: The cluster scheduler primarily operates on cloud server clusters, accepting scheduling requests from workload and link analyzers to schedule microservices within the corresponding cloud servers. This scheduling approach is categorized into three main types: vertical scaling, horizontal scaling, and brownout scheduling.

[0101] Critical Nodes and Critical Paths: A critical path refers to the path in a microservice link that has the highest sum of microservice latencies among all the longest non-repeated links in the entire system. A microservice cluster can have one or more critical paths. After finding the critical path, critical nodes are identified by testing the resource utilization of different microservices. This involves limiting resource consumption under a certain number of requests and observing the stability and latency of the entire system to distinguish between critical and non-critical nodes on the path.

[0102] (3) Solution

[0103] 3.1 Step 1: Preprocessing of source data

[0104] This step includes Figure 1Step 1 in the algorithm involves workload preprocessing and data cleaning, and processing the raw data obtained from the actual dataset. Whether the raw data is from Alibaba or Google cloud workloads, the present invention first removes columns containing empty data. This is because, regardless of whether a zero-padding scheme is used or the data is simply ignored, these redundant items will negatively impact the predicted data. Next, the present invention classifies the dataset according to time series and calculates the average value of each parameter with the same timestamp. This is achieved using a grouping function (Python Groupby function). Next, the present invention normalizes the Alibaba dataset. Normalization is a data processing method that reduces dimensionality. From a model optimization perspective, normalization can not only improve the model's convergence speed but also improve prediction accuracy. There are two forms of normalization: one is to convert numbers into decimals, that is, between 0 and 1. The other is to convert dimensional expressions into non-dimensional expressions, that is, convert them into scalars. The present invention chooses the first normalization method and uses MinMaxScaler to implement this function. MinMaxScaler operates based on the min-max scaling method, and the specific formulas are shown in Formulas 1 and 2 below.

[0105]

[0106] X scaled =X std *(X max -X min )+X min (2)

[0107] The present invention uses MinMaxScaler to transform each feature and scale each feature to a value between 0 and 1, where X represents the set of data to be processed, and X min and X max is the minimum and maximum data in the set, and the final processed data is represented by X scaled express.

[0108] 3.2 Step 2: Convert to the number of requests based on user behavior

[0109] After obtaining the processed Alibaba workload, the present invention simulates requests generated in the cluster to achieve the same effect as the original dataset. After collecting the relationship between the number of requests and the workload of the cloud server, the present invention uses a deep learning algorithm to obtain the data on the number of requests required for the entire experiment.

[0110] After this, the present invention analyzes user behavior and records the different locations of the website visited by the user based on the user's behavior. The different locations of the website visited by the user here refer to the website and its sub-sites based on the microservice homepage, such as the website homepage, website index page, website category page, website login page, etc. By simulating these different requests to visit the website, the present invention can more realistically simulate real-life user behavior.

[0111] 3.3 Step 3: Workload and Microservice Link Analysis

[0112] After generating a request and sending it to the request processor, the present invention not only sends the request to the target cluster, but also collects load information from the cloud server cluster. This load information mainly includes information such as the microservice's own latency, cloud server resource consumption, microservice request latency, and request success rate. This information is passed to the workload and link analyzer. Through critical path analysis of the microservice, the link relationship of the microservice in the current state can be obtained. Combined with the obtained link latency data, the critical path of the microservice can be obtained.

[0113] For key nodes, the present invention uses a decision tree-based classification approach to determine and derive results. By training a decision tree model tailored to dynamically changing link relationships, a precise determination of whether a node is a key node is obtained. This decision tree has three possible outcomes: key node, non-key node, and potential key node. Each key path includes at least one key node, while non-key nodes and potential key nodes may not be present in the key path.

[0114] After determining the location of key nodes, the present invention uses a trained deep learning model to make scheduling decisions for the cloud server cluster. In the present invention, reinforcement learning uses Deep Q-Learning, which is based on a deep learning neural network.

[0115] Deep Q-Learning is a common reinforcement learning method. Based on Q-Learning, the Q-Table generated through multiple iterations is converted into a neural network with all the corresponding parameters, which is then trained using deep learning. For Q-Learning, the present invention trains the model by calculating the Q-value after a series of actions. The calculation equation is shown in Formula 3:

[0116]

[0117] As can be seen from the equation above, the Q-value generated by being in state s and performing action a is the reward r(s,a) plus the highest possible Q-value of the next state s'. Here, γ is the discount, which controls the impact of future rewards on the Q-value of the current state.

[0118] The present invention iterates the equation in the above formula 3 to obtain formula 4, which is the equation in the final convergence case.

[0119] Q*(s,a)=∑ s′ P(s′∣s,a)(R(s,a,s′)+γmax a′ Q*(s′,a′)) (4)

[0120] For a neural network, the training goal of the present invention is the part of storing the past Q-value in the above equation 4, that is, equation 5.

[0121] γmax a′ Q*(s′,a′) (5)

[0122] 3.4 Step 4: Cluster Scheduling Strategy

[0123] After critical path and node judgment and model-based scheduling decision, the present invention uses three algorithms to schedule microservices in the cloud server cluster. The three scheduling strategies are as follows: Figure 2 As shown in the figure, the three scheduling strategies are horizontal expansion, horizontal expansion, and brownout control.

[0124] Horizontal scaling, also known as horizontal expansion, adjusts resources by adding or removing microservice replicas on cloud servers within the cluster to improve resource utilization and system availability. Cloud server clusters can reduce workload by increasing microservice replicas.

[0125] Vertical scaling: Also known as horizontal scaling, vertical scaling adjusts the processing capacity of the cloud server by adjusting the amount of CPU, memory, or network resources allocated to microservice instances. A cloud server cluster can reduce the workload of the cloud server cluster by increasing the resources dedicated to a single or multiple microservices on a single or multiple cloud servers within the cluster.

[0126] Control reduces microservice link latency by dynamically turning optional components on or off. Because control can affect the overall integrity of microservices to a certain extent, it is generally triggered and turns off optional components only in extreme cases, such as when a cloud service cluster is continuously in an overloaded state. Once the cloud service and cluster transition from extreme conditions to normal operation, the turned-off optional components will be turned back on by control.

[0127] Example 2

[0128] According to another embodiment of the present invention, a scheduling device based on microservice link analysis and reinforcement learning is provided, comprising:

[0129] The load generator is used to analyze and make decisions on the links of cloud server microservices based on the workload and link analyzer of the deep learning model, select the critical path and key nodes with the longest delay, and obtain cloud server load data;

[0130] Workload and Link Analyzer, which uses Deep Q-Learning-based reinforcement learning algorithms to train cloud server load data and develop deep learning models that adapt to different load conditions.

[0131] The cluster scheduler is used to schedule cloud server clusters using deep learning models. The scheduling methods include horizontal expansion, vertical expansion, and control.

[0132] The scheduling device based on microservice link analysis and reinforcement learning in the embodiment of the present invention schedules the cloud server cluster based on a deep learning model. Reinforcement learning can use three different scheduling methods: horizontal expansion, vertical expansion and control. The present invention uses a workload and link analyzer based on a deep learning model to analyze and make decisions on the links of microservices, and selects the key paths and key nodes with the longest delay. The present invention trains the cloud server load data based on the reinforcement learning algorithm of Deep Q-Learning, and uses the algorithm to train a deep learning model that is suitable for different load conditions. The present invention uses the maximum delay link in the microservice link to determine the position of the key node in the link. Compared with the microservice scheduling algorithm based on the longest link, the present invention solves the problem of possible mismatch between link length and microservice delay, and can obtain an optimization target for the delay itself. Based on this, resource scheduling of the cloud server can effectively alleviate the existing microservice link delay problem.

[0133] The following is a detailed description of the scheduling device based on microservice link analysis and reinforcement learning of the present invention using a specific embodiment:

[0134] As an integral component of modern IT technology, user-oriented cloud computing services offer numerous benefits. User-oriented microservices are typically supported by distributed cloud server clusters. When using these services, users are often highly sensitive to server link latency, as server link latency directly impacts the server's service quality, thus impacting the user experience. Link latency within a microservice cluster is caused by resource competition among microservices within the cluster. To address the issue of high link latency for microservices within cloud servers, this paper proposes an efficient, intelligent scheduling device based on reinforcement learning that performs online adjustments based on varying cloud server loads. This device uses methods including horizontal and vertical scaling to reallocate resources within the server. The purpose of this paper is to utilize a server load scheduling algorithm to address the impact of link latency on microservices within cloud servers. This paper utilizes the maximum-latency link within a microservice link to determine the location of key nodes within the link. Compared to microservice scheduling algorithms based on the longest link, this paper addresses the potential mismatch between link length and microservice latency, generating an optimization target for latency itself. This optimization target allows cloud server resource scheduling to effectively mitigate existing microservice link latency issues.

[0135] To achieve the above objectives, the present invention schedules cloud server clusters based on deep learning models. Reinforcement learning can use three different scheduling methods: horizontal expansion, vertical expansion, and control. The present invention uses a workload and link analyzer based on a deep learning model to analyze and make decisions on the links of microservices, and selects the links (critical paths) and nodes (critical nodes) with the longest delay. The present invention is based on the reinforcement learning algorithm of Deep Q-Learning, trains cloud server load data, and uses the algorithm to train a deep learning model that is suitable for different load conditions. Specifically:

[0136] (1) Data source introduction

[0137] The first set of data sets used in the present invention is the workload data set cluster-trace-v2018 of the cloud server cluster from Alibaba in 2018. This data set contains workload data spanning 8 days, which is generated by 4034 isomorphic servers. Each of the servers has 96 CPU cores, and their memory size is also the same. The present invention selected machine_usage.csv from the data set for analysis. The machine_usage.csv data set includes parameters such as machine number machine_id, timestamp time_stamp, CPU utilization cpu_util_percent of the current machine, and memory usage mem_util_percent of the machine at the current moment. The time interval for recording data is 10 seconds. The present invention removes some reference variables that often have null values, and assigns 0 to the null values of reference variables that occasionally have null values.

[0138] The present invention uses two experimental platforms, one of which is a microservice platform based on SockShop. SockShop simulates the user-facing part of an e-commerce website that sells socks. The purpose is to help users demonstrate and test microservices and cloud-native technologies. SockShop's microservices are designed to have a minimum estimated resource allocation, that is, the quota of each microservice is as small as possible during initialization. DNS is used between microservices to find other related microservices. When scheduling the cloud server for testing, a load balancer or service router can be inserted into the entire microservice framework according to the needs of the present invention. The SockShop version used in the present invention is based on the official version of Kubernetes, which includes a Jaeger-based microservice monitoring component in addition to the basic microservice components.

[0139] Another experimental platform used in this paper is train-ticket, a train ticket booking system based on a microservices architecture, comprising 41 microservices. The programming languages and frameworks used include Java, Python, and Go. train-ticket simulates the front-end and back-end of an online ticketing website, including functions such as ticket purchase, ticket sales, ticket query, ticket refund, and login. It also includes a Jaeger microservice detection component for monitoring system status and a workload balancer for controlling the overall stability of the microservices.

[0140] (2) Concept Definition

[0141] The system model of the present invention (such as Figure 1 The system (as shown in Figure 2) includes the following components: load generator, workload and link analyzer, and cluster scheduler.

[0142] Historical workload: Historical workload refers to the workload collected regularly from a running cloud server cluster by a specific server or built-in program. It mainly includes information such as timestamp, machine number, CPU utilization, and memory usage. In this paper, historical workload comes from the Alibaba dataset.

[0143] Load Generator: In our model, the load generator is comprised of a series of components, including workloads and processors, a Locust-based request generator, and a database storing historical workloads. These components coordinate to extract and process the required workload data and features from the historical database. The load generator then simulates real user usage and converts these into requests that are sent to the server cluster.

[0144] Workload and Link Analyzer: The main task of the workload and link analyzer is to analyze the requests sent from the cloud server and perform link analysis based on the link relationship of the microservices to identify the key nodes and critical paths. After that, the cluster scheduler with the deep learning / deep learning model deployed sends scheduling requests to the microservices of the cloud server cluster.

[0145] Cluster Scheduler: The cluster scheduler primarily operates on cloud server clusters, accepting scheduling requests from workload and link analyzers to schedule microservices within the corresponding cloud servers. This scheduling approach is categorized into three main types: vertical scaling, horizontal scaling, and brownout scheduling.

[0146] Critical Nodes and Critical Paths: A critical path refers to the path in a microservice link that has the highest sum of microservice latencies among all the longest non-repeated links in the entire system. A microservice cluster can have one or more critical paths. After finding the critical path, critical nodes are identified by testing the resource utilization of different microservices. This involves limiting resource consumption under a certain number of requests and observing the stability and latency of the entire system to distinguish between critical and non-critical nodes on the path.

[0147] (3) Solution

[0148] 3.1 Step 1: Preprocessing of source data

[0149] This step includes Figure 1Step 1 in the algorithm involves workload preprocessing and data cleaning, and processing the raw data obtained from the actual dataset. Whether the raw data is from Alibaba or Google cloud workloads, the present invention first removes columns containing empty data. This is because, regardless of whether a zero-padding scheme is used or the data is simply ignored, these redundant items will negatively impact the predicted data. Next, the present invention classifies the dataset according to time series and calculates the average value of each parameter with the same timestamp. This is achieved using a grouping function (Python Groupby function). Next, the present invention normalizes the Alibaba dataset. Normalization is a data processing method that reduces dimensionality. From a model optimization perspective, normalization can not only improve the model's convergence speed but also improve prediction accuracy. There are two forms of normalization: one is to convert numbers into decimals, that is, between 0 and 1. The other is to convert dimensional expressions into non-dimensional expressions, that is, convert them into scalars. The present invention chooses the first normalization method and uses MinMaxScaler to implement this function. MinMaxScaler operates based on the min-max scaling method, and the specific formulas are shown in Formulas 1 and 2 below.

[0150]

[0151] X scaled =X std *(X max -X min )+X min (2)

[0152] The present invention uses MinMaxScaler to transform each feature and scale each feature to a value between 0 and 1, where X represents the set of data to be processed, and X min and X max is the minimum and maximum data in the set, and the final processed data is represented by X scaled express.

[0153] 3.2 Step 2: Convert to the number of requests based on user behavior

[0154] After obtaining the processed Alibaba workload, the present invention simulates requests generated in the cluster to achieve the same effect as the original dataset. After collecting the relationship between the number of requests and the workload of the cloud server, the present invention uses a deep learning algorithm to obtain the data on the number of requests required for the entire experiment.

[0155] After this, the present invention analyzes user behavior and records the different locations of the website visited by the user based on the user's behavior. The different locations of the website visited by the user here refer to the website and its sub-sites based on the microservice homepage, such as the website homepage, website index page, website category page, website login page, etc. By simulating these different requests to visit the website, the present invention can more realistically simulate real-life user behavior.

[0156] 3.3 Step 3: Workload and Microservice Link Analysis

[0157] After generating a request and sending it to the request processor, the present invention not only sends the request to the target cluster, but also collects load information from the cloud server cluster. This load information mainly includes information such as the microservice's own latency, cloud server resource consumption, microservice request latency, and request success rate. This information is passed to the workload and link analyzer. Through critical path analysis of the microservice, the link relationship of the microservice in the current state can be obtained. Combined with the obtained link latency data, the critical path of the microservice can be obtained.

[0158] For key nodes, the present invention uses a decision tree-based classification approach to determine and derive results. By training a decision tree model tailored to dynamically changing link relationships, a precise determination of whether a node is a key node is obtained. This decision tree has three possible outcomes: key node, non-key node, and potential key node. Each key path includes at least one key node, while non-key nodes and potential key nodes may not be present in the key path.

[0159] After determining the location of key nodes, the present invention uses a trained deep learning model to make scheduling decisions for the cloud server cluster. In the present invention, reinforcement learning uses Deep Q-Learning, which is based on a deep learning neural network.

[0160] Deep Q-Learning is a commonly used reinforcement learning method. It is based on Q-Learning and converts the Q-Table generated by multiple iterations into a neural network with all the corresponding parameters, which is then submitted to deep learning for training. For Q-Learning, the present invention needs to train the model by calculating the Q-value after a series of actions. The calculation equation is shown in Formula 3:

[0161]

[0162] As can be seen from the equation above, the Q-value generated by being in state s and taking action a is the reward r(s,a) plus the highest possible Q-value of the next state s'. Here, γ is the discount, which controls the impact of future rewards on the Q-value of the current state.

[0163] The present invention iterates the equation in the above formula 3 to obtain formula 4, which is the equation in the final convergence case.

[0164] Q*(s,a)=∑ s′ P(s′∣s,a)(R(s,a,s′)+γmax a′ Q*(s′,a′)) (4)

[0165] For a neural network, the training goal of the present invention is the part of storing the past Q-value in the above equation 4, that is, equation 5.

[0166] γmax a′ Q*(s′,a′) (5)

[0167] 3.4 Step 4: Cluster Scheduling Strategy

[0168] After critical path and node judgment and model-based scheduling decision, the present invention uses three algorithms to schedule microservices in the cloud server cluster. The three scheduling strategies are as follows: Figure 2 As shown in the figure, the three scheduling strategies are horizontal expansion, horizontal expansion, and brownout control.

[0169] Horizontal scaling, also known as horizontal expansion, adjusts resources by adding or removing microservice replicas on cloud servers within the cluster to improve resource utilization and system availability. Cloud server clusters can reduce workload by increasing microservice replicas.

[0170] Vertical scaling: Also known as horizontal scaling, vertical scaling adjusts the processing capacity of the cloud server by adjusting the amount of CPU, memory, or network resources allocated to microservice instances. A cloud server cluster can reduce the workload of the cloud server cluster by increasing the resources dedicated to a single or multiple microservices on a single or multiple cloud servers within the cluster.

[0171] Control reduces microservice link latency by dynamically turning optional components on or off. Because control can affect the overall integrity of microservices to a certain extent, it is generally triggered and turns off optional components only in extreme cases, such as when a cloud service cluster is continuously in an overloaded state. Once the cloud service and cluster transition from extreme conditions to normal operation, the turned-off optional components will be turned back on by control.

[0172] Example 3

[0173] A storage medium storing a program file capable of implementing any of the above-mentioned scheduling methods based on microservice link analysis and reinforcement learning.

[0174] Example 4

[0175] A processor is used to run a program, wherein when the program is running, any one of the above-mentioned scheduling methods based on microservice link analysis and reinforcement learning is executed.

[0176] Compared with the existing technology, the present invention adopts a microservice link scheduling model based on reinforcement learning to allocate resources for the longest delay link of the cloud server. Compared with the method of directly selecting the longest link without analysis, the present invention can solve the problems in the field of cloud service scheduling with greater variability, more complex microservice links, and more sensitivity to link delay.

[0177] This paper uses workload datasets from Alibaba and Google's cloud data centers. Using a Locust-based load generator and a deep learning-based cloud server cluster scheduler, the paper compares several commonly used scheduling algorithms. The results demonstrate that the paper outperforms existing methods in cloud server link latency analysis and load scheduling.

[0178] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0179] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the system embodiments described above are only exemplary. For example, the division of units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0181] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs.

[0182] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0184] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A scheduling method based on microservice link analysis and reinforcement learning, characterized in that: The following steps are involved: The workload and link analyzer based on the deep learning model analyzes and makes decisions on the links of cloud server microservices, selects the key paths and key nodes with the longest delay, and obtains cloud server load data; Based on the Deep Q-Learning reinforcement learning algorithm, we train the cloud server load data and use it to train a deep learning model that adapts to different load conditions. After determining the location of key nodes, the trained deep learning model will be used to make scheduling decisions for the cloud server cluster. Scheduling methods include horizontal expansion, vertical expansion, and control. The workload and link analyzer based on the deep learning model analyzes and makes decisions on the links of the cloud server microservices, selects the key paths and key nodes with the longest delay, and obtains the cloud server load data including: Preprocess the workload and process the raw data obtained from the actual dataset to obtain the number of requests; After collecting the relationship between the number of requests and the workload of the cloud server, we used a deep learning algorithm to obtain the data on the number of requests required for the entire experiment. After generating a request and sending it to the request processor, the request is sent to the target cluster while the load information from the cloud server cluster is collected. The load information includes the latency of the microservice itself, the resource consumption of the cloud server, the request latency of the microservice, and the request success rate. The load information is passed to the workload and link analyzer. Through the critical path analysis of the microservices, the link relationship of the microservices in the current state is obtained. Combined with the obtained link delay data, the critical path of the microservices is obtained. The key nodes are judged and the results are obtained through the classification method based on the decision tree; After collecting the relationship between the number of requests and the workload of the cloud server, and obtaining data on the number of requests required for the entire experiment through a deep learning algorithm, the method further includes: Analyze user behavior and record the different locations where users visit the website based on their behavior. By simulating different requests to access the website, we can simulate the behavior of real-life users. The method of judging the key nodes and obtaining the results by the decision tree-based classification method includes: By training dynamically changing link relationships, a decision tree model is obtained to determine whether a node is a key node. The decision tree has three decision results: key node, non-key node, and potential key node. Among the three results, each key path includes at least one key node, and non-key nodes and potential key nodes do not exist in the key path.

2. The scheduling method based on microservice link analysis and reinforcement learning according to claim 1 is characterized in that: The workload is pre-processed, and the raw data obtained from the actual data set is processed to obtain the number of requests, including: First, we remove the columns containing empty data, classify the dataset into time series, and then use the grouping function to calculate the average value of each parameter with the same timestamp; Use MinMaxScaler to normalize the dataset; the operation of MinMaxScaler is based on the min-max scaling method, using MinMaxScaler to transform each feature and scale each feature to a value between 0 and 1.

3. The scheduling method based on microservice link analysis and reinforcement learning according to claim 1 is characterized in that: The horizontal expansion adjusts resources by adding or removing replicas of microservices on cloud servers within the cluster; The cloud server cluster reduces the workload of the cloud server cluster by increasing the number of replicas of microservices; The vertical expansion adjusts the processing service capacity of the current cloud server by adjusting the amount of CPU, memory or network resources allocated to the microservice; The cloud server cluster reduces the workload of the cloud server cluster by increasing the resources of a single cloud server or multiple cloud servers in the cluster for a single or multiple microservices; The control reduces microservice link latency by dynamically turning on or off optional components. When the cloud service and cluster transition from extreme conditions to normal working conditions, the turned-off optional components will be turned back on by the control.

4. A scheduling device based on microservice link analysis and reinforcement learning using the scheduling method based on microservice link analysis and reinforcement learning according to claim 1, characterized in that: include: The load generator is used to analyze and make decisions on the links of cloud server microservices based on the workload and link analyzer of the deep learning model, select the critical path and key nodes with the longest delay, and obtain cloud server load data; Workload and Link Analyzer, which uses Deep Q-Learning-based reinforcement learning algorithms to train cloud server load data and develop deep learning models that adapt to different load conditions. The cluster scheduler is used to schedule cloud server clusters using deep learning models. The scheduling methods include horizontal expansion, vertical expansion, and control.

Images