A micro-service scheduling method and device, computer equipment and storage medium

Abstract

The application relates to a micro-service scheduling method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring load data, converting the load data into a user request; collecting micro-service instance information in a cluster based on the user request, and performing load prediction according to the micro-service instance information; combining the micro-service instance information and the load prediction result, training a central network and a distributed deployment network in a system model by using a reinforcement learning algorithm, and using the distributed deployment network as a student network to perform imitative learning on the behavior of the central network; and when the imitative learning of the distributed deployment network converges, using a scheduler algorithm to execute a scheduling strategy of distributed micro-services in the cluster through the distributed deployment network. The embodiment of the application adopts an imitative learning technology, transfers the functions of resource scheduling and data collection to the distributed deployment network, reduces the burden of the central network, and realizes efficient decision-making of micro-service scheduling.

Description

Technical Field

[0001] This application belongs to the field of cloud computing resource management technology, and specifically relates to a microservice scheduling method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In the field of contemporary software design and development, microservice architecture has demonstrated its transformative advantages. The core characteristics of microservice architecture are modularity and decentralization. They communicate through well-defined APIs (Application Programming Interfaces), supporting seamless integration and compatibility with multiple technology stacks. They can decompose complex applications into small, independently deployable services. Each microservice focuses on a specific business function, enabling teams to develop their respective components simultaneously, thus achieving rapid development, deployment, and scalability.

[0003] The rapid growth of the microservices paradigm has also brought challenges to resource management. As the number of microservices within a system increases, ensuring optimal resource allocation and utilization becomes complex. In cloud computing and virtualization environments, application requirements are constantly changing. First, application load and performance requirements are typically not static; they fluctuate and change over time. For example, some periods may experience high traffic, while others have lower loads. Second, the rise of cloud computing and virtualization environments has made on-demand resource allocation easier, but it has also introduced challenges in resource management, including how to effectively allocate, monitor, and maintain resources. Against this backdrop, microservice auto-scaling technology has emerged to adapt to these dynamic performance demands, reduce resource waste, improve automated management, and enhance system fault tolerance. Microservice auto-scaling technology involves dynamically adjusting the instances or resources allocated to microservices based on real-time demand, ensuring efficient resource utilization while maintaining the desired performance level. Due to the intricate interdependencies between microservices, critical paths and nodes that impact performance often emerge. The dynamic nature of these dependencies requires a deep understanding of application behavior, workload patterns, and their impact on system performance. Determining the optimal scaling strategy that balances resource allocation among interconnected services while maintaining overall system stability has become a new challenge for microservice scheduling.

[0004] In recent years, many service scheduling algorithms have been proposed, aiming to optimize service latency in computation and ensure service quality. Existing service scheduling methods typically include the following three types:

[0005] (1) Threshold-based and heuristic service scheduling algorithm: This algorithm uses predefined rules for resource allocation. When the system's resource usage exceeds a specific threshold, the auto-scaler increases resource allocation. When resources are sufficient, request patterns are stable, and there are no drastic fluctuations, this algorithm can reduce resource conflicts between microservices, thereby improving service availability. However, this algorithm relies on thresholds to trigger resource scaling, limiting the accurate prediction and adaptability to load changes, and making it unable to flexibly respond to future load patterns and sudden events. Secondly, since scaling operations are only triggered after resource utilization reaches or exceeds the threshold, this strategy has a delayed response, which may lead to decreased system performance and a poor user experience. Furthermore, fixed thresholds can easily lead to frequent scaling operations when load fluctuations are small, resulting in resource waste and system instability. It also requires a large amount of prior human knowledge to set predefined thresholds, which may lead to over-adjustment and increased costs if the thresholds are not set properly. Finally, this algorithm is generally suitable for simple load patterns and is difficult to adapt to complex, multi-dimensional, and dynamic load scenarios such as multiple resource demands, seasonal fluctuations, and sudden events, limiting its effectiveness in complex environments.

[0006] (2) Machine learning-based auto-scaling algorithm; This algorithm is a dynamic and intelligent approach that uses machine learning algorithms to automatically adjust the resources allocated to microservices to cope with changing workloads. By analyzing historical data, performance metrics, and workload patterns, this algorithm enables microservices to scale up or down as needed, ensuring optimal resource utilization and maintaining stable application performance. This helps improve efficiency, reduce operating costs, and enhance the scalability of modern distributed systems built on microservice architectures.

[0007] (3) Deep Learning-Based Auto-Scaling Algorithm: This algorithm automates the scaling process of microservice resources using advanced deep learning techniques. By using neural networks for decision-making (e.g., reinforcement learning) and analyzing complex patterns in the data, it dynamically adjusts the resources allocated to microservices based on real-time demand. Furthermore, by continuously learning from historical and current workload data, the algorithm optimizes resource provisioning, enhances system responsiveness, and reduces resource waste, facilitating efficient management of microservices in dynamic environments and promoting seamless scalability and improved overall performance.

[0008] However, autoscaling algorithms based on machine learning and deep learning typically require significant computational resources and time, including environment simulation, data sampling, and policy optimization, resulting in high computational costs. Secondly, these algorithms require large amounts of sample data, meaning real-world applications may require substantial training time. Furthermore, the algorithms need to balance exploring unknown policies with utilizing known ones; insufficient exploration can lead to performance degradation, while overexploration can also have negative effects. The algorithms also heavily rely on accurate environmental models, which are challenging to model in complex and uncertain real-world environments. Some methods also require large amounts of data for learning, which may be impractical for resource-constrained systems or real-time applications. Finally, existing algorithms often support only a limited number of scaling operations due to the high degree of freedom in resource adjustment. Summary of the Invention

[0009] This application provides a microservice scheduling method, apparatus, computer device, and storage medium, which aims to at least partially solve one of the aforementioned technical problems in the prior art.

[0010] To address the above problems, this application provides the following technical solution:

[0011] A microservice scheduling method, comprising:

[0012] Obtain load data and convert the load data into user requests;

[0013] Based on user requests, collect information on microservice instances within the cluster, and perform load prediction based on the microservice instance information;

[0014] Combining the microservice instance information and load prediction results, reinforcement learning algorithms are used to train the central network and distributed deployment network in the system model, and the distributed deployment network is used as a student network to imitate and learn the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operations of microservices in parallel based on local information;

[0015] Once the imitation learning of the distributed deployment network converges, the scheduler algorithm is used through the distributed deployment network to execute the scheduling strategy of distributed microservices within the cluster.

[0016] The technical solution adopted in this application embodiment further includes: the distributed deployment network includes a deployment buffer and an imitation learner; the deployment buffer is used to store the policy communicated by the central network to train the distributed deployment network, and to store the state when the policy is recorded, and the policy-state pair is used to train the distributed deployment network through imitation learning; the imitation learner is used to independently scale each distributed deployment network.

[0017] The technical solution adopted in this application embodiment also includes: the distributed deployment network includes a learning mode and an action mode. In the learning mode, the central network explores the state and guides the distributed deployment network to take appropriate actions, the state including resource utilization. In the action mode, the distributed deployment network makes autonomous decisions based on the state and performs parallel scaling operations on the system.

[0018] The technical solution adopted in this application embodiment further includes: the reinforcement learning algorithm employs dual-delay deep deterministic policy gradient technology and a retraining notify mechanism; specifically, the reinforcement learning algorithm is used to train the central network and the distributed deployment network in the system model.

[0019] The training process includes a first phase and a second phase. The first phase is a central cluster exploration phase, and the second phase is a distributed scheduling phase. When the retraining notification is triggered by a trigger condition set by the central network, the system enters the first phase. In the first phase, the central network explores the state of the central cluster by selecting scaling operations for each distributed deployment network. The scaling operations include horizontal scaling, CPU scaling, memory scaling, and maintaining the current state. The distributed deployment networks and the central network begin training simultaneously with the state exploration. During training, the distributed deployment networks learn their own information and the mapping of scaling operations from the central network through imitation learning. When the retraining notification determines that a change is to be made, it indicates that the central network has conducted extensive state exploration, and the system switches from the first phase to the second phase. In the second phase, the responsibility for decision-making is transferred to the distributed deployment networks. The scope of the scaling operations remains unchanged, and policy-state pairs are continuously registered in the deployment buffer of the distributed deployment networks to update the evolving knowledge of the system. When the retraining notification is activated, the system switches from the second phase to the first phase, and the state exploration is restarted through the central network.

[0020] The technical solution adopted in this application embodiment also includes: the specific training process in the first stage includes:

[0021] The actor network and commentator network are initialized using a dual-delay deep deterministic policy gradient technique, and the buffer is replayed.

[0022] Select an action from the actor network and execute the action, detect the subsequent state, reward and episode termination state, and store the corresponding tuple (s, a, r, s, d) in a buffer;

[0023] When the buffer accumulates a set amount of data and reaches the update time, a batch of transition data is sampled from the buffer using a dual-delay deep deterministic strategy gradient technique.

[0024] Add a clipped black noise (∈), between -c and c, as shown in the following formula:

[0025] [a′=clip(π′(s′)+clip(∈,-c,c), aHigh, aLow)]

[0026] The double-Q learning algorithm is employed, where both Q functions use a single objective (y(r, s′, d)), which is achieved by adding an immediate reward r and applying it to both Q functions (Q′). θi′ The minimum value obtained from (s′, a′) is multiplied by the discount factor (γ) for calculation. If the subsequent state is a terminal state (d = 1), then the future reward is not considered, as shown in the following formula:

[0027]

[0028] Wherein, the loss is denoted as (L(θi)), and for the i-th commenter network, it is calculated as the difference between the target Q-value y and the Q-value predicted by the commenter network (Q... θi The average of the sum of squared differences between (s, a) spans a mini-batch of experiences B, containing a set of ((s, a, r, s′, d)) experiences, the number of which is denoted by |B|, as shown in the formula:

[0029]

[0030] Learn a more stable Q-function, update the gradient ascent step using the loss function at each policy update, and update the target network:

[0031]

[0032] The technical solution adopted in this application embodiment further includes: in the second stage, the imitation learning process of the distributed deployment network includes calculating the loss function and reward using the strategy provided by the central network, and focusing on two optimization objectives: Quality of Service (QoS) and stable physical machine resource status. The optimization objectives are achieved through a reward model, including response time (R). qos (rt)) and resource utilization rate (R util (u)).

[0033] The technical solution adopted in this application embodiment further includes: the Quality of Service (QoS) is obtained through Locust to obtain user latency, wherein the maximum tolerable latency (RT) is... max In the QoS reward function, normal system operation is rewarded with 1, and operation exceeding (RT) is rewarded with 1. max The performance of the service quality reward (R) is penalized, and the reward gradually approaches 0, eventually reaching zero. qos The formula for calculating (rt) is:

[0034]

[0035] For the physical machine resources in a stable state, the resource utilization rate is measured using the following formula;

[0036]

[0037] Where (K) and (R) represent the residual of the r-th resource type on the k-th physical machine and the k-th physical machine, respectively. The ideal utility of a certain resource type r represents the k-th physical machine;

[0038] The final reward value is a combination of response time and resource utilization:

[0039]

[0040] The ultimate optimization goal is to reduce response time while maintaining stable system operation.

[0041]

[0042] Another technical solution adopted in this application embodiment is: a microservice scheduling device, comprising:

[0043] Data acquisition module: used to acquire load data and convert the load data into user requests;

[0044] Load prediction module: used to collect microservice instance information within the cluster based on user requests, and to perform load prediction based on the microservice instance information;

[0045] The reinforcement learning algorithm training module is used to combine the microservice instance information and load prediction results to train the central network and distributed deployment network in the system model using reinforcement learning algorithms, and to use the distributed deployment network as a student network to imitate and learn the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operations of microservices in parallel based on local information;

[0046] Microservice scheduling module: After the imitation learning of the distributed deployment network converges, it uses the scheduler algorithm to execute the scheduling strategy of distributed microservices within the cluster through the distributed deployment network.

[0047] Another technical solution adopted in this application embodiment is: a computer device, the computer device including a processor and a memory coupled to the processor, wherein,

[0048] The memory stores program instructions for implementing the microservice scheduling method;

[0049] The processor is used to execute the program instructions stored in the memory to control the microservice scheduling method.

[0050] Another technical solution adopted in this application embodiment is: a storage medium storing processor-executable program instructions, the program instructions being used to execute the microservice scheduling method.

[0051] Compared to existing technologies, the beneficial effects of the embodiments of this application are as follows: The microservice scheduling method, apparatus, computer equipment, and storage medium of the embodiments of this application optimize the problem modeling of service scheduling in cloud computing by introducing a distributed framework and reinforcement learning algorithms. By employing imitation learning technology, the functions of resource scheduling and data collection are transferred to a distributed deployment network, thereby reducing the burden on the central network and achieving efficient decision-making in microservice scheduling. Considering the potential resource differences among nodes in the cluster and the latency characteristics of user requests, resource allocation is dynamically adjusted to ensure timely and efficient response to user requests even under high load conditions. Furthermore, the resource utilization of the cluster is considered to avoid overload and reduce service latency. Compared to existing technologies, the embodiments of this application have at least the following advantages:

[0052] 1. The embodiments of this application use a low-complexity reinforcement learning algorithm for microservice scheduling, which can speed up the decision-making process, thereby reducing service latency and enabling users to enjoy faster response and higher service efficiency.

[0053] 2. This application embodiment incorporates the resource utilization of the edge server into probabilistic decision-making, which can better balance resource allocation, thereby ensuring service quality, and the system can more intelligently adapt to constantly changing load demands.

[0054] 3. This application's embodiments introduce a distributed scheduling strategy, enabling multiple distributed deployment networks to work independently and collaboratively, ensuring the system can still operate effectively and scalably even with an increase in the number of users. Furthermore, because the distributed framework integrates information from multiple nodes, it can provide a more comprehensive view and a more accurate description of the system's behavior.

[0055] 4. The embodiments of this application allow for more freedom in resource adjustment, thereby more flexibly adapting to the needs of different applications. The system can adjust resources more frequently and in a more differentiated manner to cope with rapidly changing service demands. Attached Figure Description

[0056] Figure 1 This is a flowchart of a microservice scheduling method according to an embodiment of this application;

[0057] Figure 2 This is a schematic diagram of the microservice scheduling system framework according to an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of the microservice scheduling device structure according to an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of the computer device structure according to an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of the storage medium according to an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0062] The terms "first," "second," and "third" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or computer device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or computer devices.

[0063] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0064] Specifically, please refer to Figure 1 This is a flowchart of a microservice scheduling method according to an embodiment of this application. The microservice scheduling method according to an embodiment of this application includes the following steps:

[0065] S100: Obtain load data from the user behavior dataset within the cluster and use the Locust tool to perform traffic transformation, converting the load data into actual user requests;

[0066] In this step, the load data comes from Alibaba, which includes 8 days of workload tracking data from 4,000 machines.

[0067] S110: Based on user requests, collect microservice instance information through the microservice information collection module;

[0068] S120: Perform load prediction analysis on the system based on the collected microservice instance information;

[0069] S130: Combining the collected microservice instance information and load prediction results, use reinforcement learning algorithms to train the central network and distributed deployment network of the system model, and use the distributed deployment network as a student network to imitate the behavior of the central network until the imitation learning reaches convergence.

[0070] Please refer to the following in this step: Figure 2 This is a schematic diagram of the microservice scheduling system model structure according to an embodiment of this application. The microservice scheduling system model of this embodiment includes a central network and a distributed deployment network. When performing resource scheduling, the same scheduling strategy is used, which is referred to as the scheduler for convenience in this embodiment. The scheduler is a general scaling process of DPRC (distributed RPC), which obtains suggestions from the neural network to obtain a Q value to guide resource adjustment, including CPU utilization and memory utilization, as a feasible instruction embodiment within the DPRC framework. In DRPC, resource provisioning based on reinforcement learning is modeled as a Markov decision process, which regards the state st∈S as the current microservice system state and interprets the action a∈A as a scaling operation that modifies the system state and allocates resources. In this embodiment, M=(m1,m2,...,mi) is used to represent the physical machine that provides resources for microservices in the system. For physical machine mi, the amount of resources that can be allocated is represented as Ri=(ri,1,ri,2,...,ri,j), where j represents the resource type, such as CPU and memory. Finally, the set of operations that can be executed on physical machine mi is given by Ai = (ai,1,ai,2,...,ai,k), and supported operations include horizontal scaling, CPU scaling, memory scaling, and Brownout. The plus or minus sign before the action indicates adding or removing resources for a specific physical machine, respectively. "Brownout" is a binary indicator indicating whether brownout can be triggered on the physical machine. If this value is set to positive, the minimum allowed replicas for horizontal scaling will be set to 0. The system's overall action space A is the product of the action spaces of each physical machine.

[0071] Specifically, the training process for training the central network and the distributed deployment network using reinforcement learning (RL) algorithms includes two parts: exploration + reinforcement learning in the central network and imitation learning in the distributed deployment network. The use of multi-agent reinforcement learning often leads to suboptimal solutions; therefore, a teacher network is needed to suggest actions to another student network, study the global state, and formulate an optimal policy function to enhance system performance. Since reinforcement learning can provide a tight feedback loop, explore scaling operations, and formulate optimal policies independent of erroneous hypotheses, this embodiment uses reinforcement learning algorithms to train the collected data, and uses imitation learning techniques to train the distributed deployment network to learn from the central network. In this embodiment, the distributed deployment network is deployed in a distributed manner, enabling it to execute microservice scaling operations in parallel based on its local information. Specifically, the distributed deployment network includes a deployment buffer and an imitation learner. The deployment buffer stores the policies conveyed by the teacher network (i.e., the central network) to train the student networks (distributed deployment networks), and stores the state when the policies are recorded. These policy-state pairs are used to train the student networks through imitation learning, thereby replicating successful behavior in a given context and maintaining the policies transmitted by the central network. The imitation learner is used to independently scale each distributed deployment network. The distributed deployment network operates in two modes: a learning mode and an action mode. In the learning mode, the central network explores various states and guides the distributed deployment networks to take appropriate actions. In the action mode, the distributed deployment networks autonomously make decisions based on states such as resource utilization and perform parallel scaling operations on the system, enabling the distributed deployment networks to continuously learn and adapt to changing conditions, thereby optimizing resource allocation and system performance.

[0072] Specifically, the reinforcement learning algorithm in this application employs the following two techniques:

[0073] (1) Dual-Delay Deep Deterministic Policy Gradient (TD3); TD3 is an advanced model-free actor-commentator RL framework that overcomes the limitations of Q-value overestimation in RL, eliminates the need for accurate modeling of stationary distributed states (transitions between states), and renders the simulation of state transitions in model-based RL inefficient in the context of microservice updates. Furthermore, the actor-commentator framework, which combines policy-based and value-based approaches, is suitable for continuous stochastic environments, accelerates convergence, reduces variance, and provides a robust and efficient solution for managing the dynamic characteristics of container-based environments.

[0074] (2) Retraining Notifier: When the system needs to be retrained, a signal is sent through the retraining notifier. During the imitation learning process, the learner may be unable to make effective decisions, triggering the retraining notifier and initiating a retraining notification. Specifically, the retraining notifier is activated in the following three scenarios:

[0075] I. Insufficient exploration of teacher networks: If a teacher network encounters an unusual situation, it may lack confidence and may not be able to properly guide the student network, which may lead to the cloud service scaler misjudging the system's scaling needs.

[0076] Second, the learning state of the student network is suboptimal: In this case, although the central network learns an optimal policy function, the distributed network fails to establish a relationship between the optimal policy and the domain information it possesses.

[0077] Third, the knowledge held by the student network is outdated: For example, assuming that most users used to search for tickets, the student network may have relied too much on this pattern. If the request combination changes, the deployment of the student network will need to be retrained.

[0078] Furthermore, in this embodiment, the training process for training the central network and the distributed deployment network using reinforcement learning algorithms includes two phases: a central cluster exploration phase (first phase) and a distributed scheduling phase (second phase). The first phase begins when the retraining notifier is triggered by a condition set by the central network. In the first phase, the central network explores the state of the central cluster by selecting scaling operations for each distributed deployment network. These scaling operations include horizontal scaling, CPU scaling, memory scaling, and the option to maintain the current state. The distributed deployment networks begin their training phase simultaneously with the central network's state exploration. During this training, the distributed deployment networks learn their own information and the mapping of scaling operations from the central network through imitation learning. When the retraining notifier determines that a change is imminent, it indicates that the central network has conducted extensive state exploration and the distributed deployment networks have been sufficiently trained, and the system switches from the first phase to the second phase. In the second phase, the responsibility for decision-making is transferred to the distributed deployment networks, provided that the central network's state exploration is sufficient. The scope of scaling operations remains unchanged. Additionally, in the second phase, policy-state pairs are continuously registered in the deployment buffer of the distributed deployment networks to update the system's evolving knowledge. The system's dynamism is reflected in its ability to switch between these phases based on instructions provided by the retraining notify. When the retraining notify is activated, the system switches from phase two to phase one, thereby restarting state exploration via the central network. Conversely,

[0079] Furthermore, the specific training process in the first phase includes:

[0080] Step 1: Initialize the actor network and commentator network using TD3, and replay the buffer;

[0081] Step 2: Select an action from the actor network and execute the action, detect the subsequent state, reward and plot termination state, and store the corresponding tuple (s, a, r, s, d) in a buffer;

[0082] Step 3: When a set amount of data has accumulated and an update is imminent, a batch of transitional data is sampled from the buffer using TD3. This data includes data from both the central network and the distributed deployment network.

[0083] Step 4: In some cases, scaling may significantly reduce SLO violations, while circumventing the shrinking of communication time may result in better response times after several scaling operations. Since the variability of network connectivity can also cause this spike, a clipped black noise (∈) is added to mitigate this problem, between -c and c, as shown in Equation (1):

[0084] [a′=clip(π′(s′)+clip(∈,-c,c),aHigh,aLow)] (1)

[0085] Furthermore, since microservices are deployed in a distributed manner and communicate over cables, some scaling operations may experience significant reductions in communication time due to internet fluctuations, resulting in substantial rewards. To avoid overestimating the rewards, a sheared double Q-learning algorithm is employed. Both Q-functions use a single objective (y(r, s′, d)), which is calculated by adding an immediate reward r and applying it from the two Q-functions (Q′). θi′ The minimum value obtained from (s′, a′) is multiplied by the discount factor (γ) to calculate the reward. If the subsequent state is a terminal state (d = 1), then the future reward is not considered, as shown in formula (2):

[0086]

[0087] Wherein, the loss is denoted as (L(θi)), and for the i-th commenter network, it is calculated as the difference between the target Q-value y and the Q-value predicted by the commenter network (Q... θi The average of the sum of squared differences between (s, a) spans a mini-batch of experiences B, containing a set of ((s, a, r, s′, d)) experiences. The number of such experiences is denoted by |B|, as shown in the formula:

[0088]

[0089] Step 5: To minimize the variability of microservice communication and learn a more stable Q-function, update the gradient ascent step using the loss function at each policy update:

[0090]

[0091] Step 6: Update the target network.

[0092] In the second phase, the imitation learning process of the distributed deployment network involves calculating the loss function and reward using the strategy provided by the central network. During the imitation learning process, the main focus is on two optimization objectives: Quality of Service (QoS) and the stable state of physical machine resources. These optimization objectives are achieved through a reward model, including response time (R0). qos (rt)) and resource utilization rate (R util (u)). Specifically, for Quality of Service (QoS), user latency is obtained through Locust, where the maximum tolerable latency (RT) is... max This is predefined in the QoS reward function. Normal system operation rewards 1, while exceeding (RT)... max Performance will be penalized, and rewards will gradually approach 0, thus preventing SLO violations and ultimately reducing the Service Quality Reward (R). qos (rt) can be obtained from the following formula:

[0093]

[0094] For a physical machine in a stable resource state, the resource utilization rate is calculated using the formula (R... util The design model in (u) is used for measurement, where (K) and (R) represent the residual of the r-th resource type on the k-th physical machine and the k-th physical machine, respectively. The predefined utility represents the ideal utility of a certain resource type r for the k-th physical machine. Approaching this ideal utilization rate will be rewarded; resource insufficiency or waste will be discouraged.

[0095]

[0096] The final reward value is a combination of response time and resource utilization, as shown in formula (7):

[0097]

[0098] The ultimate optimization goal is to reduce response time while maintaining stable system operation, as shown in formula (8):

[0099]

[0100] In this application embodiment, the policy score provided by the distributed deployment network is used as another optimization objective of the reinforcement learning algorithm. It is expected that the distributed deployment network will be as close as possible to the policy score provided by the central network during training, that is, to minimize the mean squared error.

[0101] S140: After the imitation learning of the distributed deployment network converges, the scheduler algorithm is used through the distributed deployment network to execute the scheduling strategy of distributed microservices within the cluster.

[0102] In this step, when the retraining notifier determines that the system has entered a stable state, that is, the distributed deployment network has acquired enough scheduling strategies from the central network through imitation learning, the cluster enters the distributed scheduling phase, and each distributed deployment network uses the scheduler algorithm to execute the scheduling tasks of distributed microservices within the cluster.

[0103] Based on the above, the microservice scheduling method of this application optimizes the problem modeling of service scheduling in cloud computing by introducing a distributed framework and reinforcement learning algorithms. It employs imitation learning technology to transfer resource scheduling and data collection functions to a distributed deployment network, thereby reducing the burden on the central network and achieving efficient decision-making in microservice scheduling. Considering the potential resource differences among nodes in the cluster and the latency characteristics of user requests, it dynamically adjusts resource allocation to ensure timely and efficient response to user requests even under high load conditions. It also considers cluster resource utilization to avoid overload and reduce service latency. Compared to existing technologies, this application embodiment has at least the following advantages:

[0104] 1. The embodiments of this application use a low-complexity reinforcement learning algorithm for microservice scheduling, which can speed up the decision-making process, thereby reducing service latency and enabling users to enjoy faster response and higher service efficiency.

[0105] 2. This application embodiment incorporates the resource utilization of the edge server into probabilistic decision-making, which can better balance resource allocation, thereby ensuring service quality, and the system can more intelligently adapt to constantly changing load demands.

[0106] 3. This application's embodiments introduce a distributed scheduling strategy, enabling multiple distributed deployment networks to work independently and collaboratively, ensuring the system can still operate effectively and scalably even with an increase in the number of users. Furthermore, because the distributed framework integrates information from multiple nodes, it can provide a more comprehensive view and a more accurate description of the system's behavior.

[0107] 4. The embodiments of this application allow for more freedom in resource adjustment, thereby more flexibly adapting to the needs of different applications. The system can adjust resources more frequently and in a more differentiated manner to cope with rapidly changing service demands.

[0108] Please see Figure 3 This is a schematic diagram of the microservice scheduling device structure according to an embodiment of this application. The microservice scheduling device 40 according to an embodiment of this application includes:

[0109] Data acquisition module 41: used to acquire load data and convert the load data into user requests;

[0110] Load prediction module 42: used to collect microservice instance information in the cluster based on user requests, and perform load prediction based on the microservice instance information;

[0111] Reinforcement learning algorithm training module 43: used to combine the microservice instance information and load prediction results, and use reinforcement learning algorithm to train the central network and distributed deployment network in the system model, and use the distributed deployment network as a student network to imitate the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operation of microservices in parallel based on local information;

[0112] Microservice scheduling module 44: After the imitation learning of the distributed deployment network converges, it uses the scheduler algorithm to execute the scheduling strategy of distributed microservices within the cluster through the distributed deployment network.

[0113] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0114] The apparatus provided in this application can be applied to the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0115] Please see Figure 4 This is a schematic diagram of a computer device structure according to an embodiment of this application. The computer device 50 includes:

[0116] Memory 51 storing executable program instructions;

[0117] Processor 52 connected to memory 51;

[0118] The processor 52 is used to call the executable program instructions stored in the memory 51 and perform the following steps: acquire load data and convert the load data into user requests; based on the user requests, collect microservice instance information within the cluster and perform load prediction based on the microservice instance information; combine the microservice instance information and the load prediction results, use a reinforcement learning algorithm to train the central network and the distributed deployment network in the system model, and use the distributed deployment network as a student network to imitate the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operations of microservices in parallel based on local information; when the imitation learning of the distributed deployment network converges, the scheduler algorithm is used through the distributed deployment network to execute the scheduling strategy of distributed microservices within the cluster.

[0119] The processor 52 can also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0120] Please see Figure 5This is a schematic diagram of the structure of the storage medium in an embodiment of this application. The storage medium in this embodiment stores program instructions 61 capable of implementing the following steps: acquiring load data and converting the load data into user requests; collecting microservice instance information within the cluster based on the user requests and performing load prediction based on the microservice instance information; combining the microservice instance information and the load prediction results, training the central network and the distributed deployment network in the system model using a reinforcement learning algorithm, and using the distributed deployment network as a student network to imitate the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operations of microservices in parallel based on local information; when the imitation learning of the distributed deployment network converges, the scheduler algorithm is used through the distributed deployment network to execute the scheduling strategy of the distributed microservices within the cluster. The program instructions 61 can be stored in the above-mentioned storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network computer device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage media include: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program instructions, or terminal computer devices such as computers, servers, mobile phones, and tablets. Servers can be standalone servers or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, apparatuses, or units, and may be electrical, mechanical, or other forms.

[0122] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A microservice scheduling method, characterized in that, include: Obtain load data and convert the load data into user requests; Based on user requests, collect information on microservice instances within the cluster, and perform load prediction based on the microservice instance information; Combining the microservice instance information and load prediction results, reinforcement learning algorithms are used to train the central network and distributed deployment network in the system model, and the distributed deployment network is used as a student network to imitate and learn the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operations of microservices in parallel based on local information; Once the imitation learning of the distributed deployment network converges, the scheduler algorithm is used through the distributed deployment network to execute the scheduling strategy for distributed microservices within the cluster; wherein: The reinforcement learning algorithm employs a dual-delay deep deterministic policy gradient technique and a retraining informant. Specifically, the reinforcement learning algorithm is used to train the central network and the distributed deployment network in the system model. The training process includes a first phase and a second phase. The first phase is the central cluster exploration phase, and the second phase is the distributed scheduling phase. The first phase begins when the retraining notification is triggered by a condition set by the central network. In the first phase, the central network explores the state of the central cluster by selecting scaling operations for each distributed deployment network. The scaling operations include horizontal scaling, CPU scaling, memory scaling, and maintaining the current state. The distributed deployment networks begin training simultaneously with the central network's state exploration. During training, the distributed deployment networks learn their own information and the mapping of scaling operations from the central network through imitation learning. When the retraining notification determines that a change is imminent, it indicates that the central network has conducted extensive state exploration, and the system switches from the first phase to the second phase. In the second phase, the responsibility for decision-making is transferred to the distributed deployment networks. The scope of the scaling operations remains unchanged, and policy-state pairs are continuously registered in the deployment buffers of the distributed deployment networks to update the evolving knowledge of the system. When the retraining notification is activated, the system switches from the second phase to the first phase, and the state exploration is restarted through the central network.

2. The microservice scheduling method according to claim 1, characterized in that, The distributed deployment network includes a deployment buffer and an imitation learner. The deployment buffer stores policies communicated by the central network to train the distributed deployment network and stores the states of the policies. Policy-state pairs are used to train the distributed deployment network through imitation learning. The imitation learner is used to independently scale each distributed deployment network.

3. The microservice scheduling method according to claim 2, characterized in that, The distributed deployment network includes a learning mode and an action mode. In the learning mode, the central network explores the state and guides the distributed deployment network to take appropriate actions, the state including resource utilization. In the action mode, the distributed deployment network autonomously makes decisions based on the state and performs parallel scaling operations on the system.

4. A microservice scheduling device, which executes the microservice scheduling method according to claim 1, characterized in that, include: Data acquisition module: used to acquire load data and convert the load data into user requests; Load prediction module: used to collect microservice instance information within the cluster based on user requests, and to perform load prediction based on the microservice instance information; The reinforcement learning algorithm training module is used to combine the microservice instance information and load prediction results to train the central network and distributed deployment network in the system model using reinforcement learning algorithms, and to use the distributed deployment network as a student network to imitate and learn the behavior of the central network; wherein, the distributed deployment network is deployed in a distributed manner and can execute the expansion operations of microservices in parallel based on local information; Microservice scheduling module: After the imitation learning of the distributed deployment network converges, it uses the scheduler algorithm to execute the scheduling strategy of distributed microservices within the cluster through the distributed deployment network.

5. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, wherein, The memory stores program instructions for implementing the microservice scheduling method according to any one of claims 1-3; The processor is used to execute the program instructions stored in the memory to control the microservice scheduling method.

6. A storage medium, characterized in that, The system stores processor-executable program instructions for performing the microservice scheduling method according to any one of claims 1 to 3.

Images