A computer resource scheduling system based on reinforcement learning under a micro-service architecture
A computer resource scheduling system employing the Q-learning reinforcement learning algorithm in a microservice architecture solves the space explosion problem of RL models by grouping and dynamically adjusting the number of instances, thereby improving the accuracy and speed of resource scheduling and optimizing container resource allocation and service quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIV
- Filing Date
- 2022-07-30
- Publication Date
- 2026-07-21
AI Technical Summary
Existing RL models suffer from space explosion in microservice architectures and fail to effectively balance the accuracy and speed of resource scheduling, especially in terms of insufficient integration of vertical and horizontal scaling.
A computer resource scheduling system based on the Q-learning reinforcement learning algorithm is adopted. Instances are divided into Agent group, trial group and threshold group by a grouper. Combined with vertical and horizontal scaling actions, the system optimizes resource allocation by using a monitor, decision maker, executor and parameter adjuster, and dynamically adjusts the number of instances to adapt to load changes.
It effectively solves the space explosion problem of RL models, improves the accuracy and speed of resource scheduling, optimizes container resource allocation, improves service quality and resource utilization, and adapts to complex load changes.
Smart Images
Figure CN115237571B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cloud service instance elastic scaling technology, specifically to a computer resource scheduling system based on reinforcement learning under a microservice architecture. Background Technology
[0002] In cloud platforms, container technology, represented by Docker, has been widely adopted due to its lightweight and convenient characteristics. Compared to virtual machines that require deploying an entire operating system to deploy applications, containers are very lightweight and start up quickly. Containers do not need to virtualize an entire operating system; they only need to virtualize a small-scale environment, and they start up very quickly, consuming virtually no additional system resources besides running the application within them. In cloud computing environments, large-scale containers need to be managed, and while Docker focuses only on providing the containers and images themselves, Kubernetes, with its powerful container orchestration capabilities, has become a leader in container cluster management systems. However, the workload in large-scale cloud computing platforms is uncertain, especially in user-facing applications. Given limited hardware resources and uncertain workloads, how to fully utilize computing resources, achieve elastic scaling of computing resources to ensure stable operation of the cloud platform under sudden load changes, and improve average service response time and service quality are key issues that need to be considered.
[0003] In container environments, the research and implementation of elastic scaling technology, a core platform-level function, has become an urgent need. Ensuring the quality of service (QoS) in typical internet flash-crowd scenarios remains challenging. For service requesters, response time is one of the most important performance indicators of QoS. However, reducing resource usage while meeting average request response time is a difficult problem. This is because service request loads in the internet environment have self-similar, bursty, and variable characteristics, making it difficult to predict future service request loads. Over-allocation of resources, such as deploying microservices based on peak load demand, leads to significant resource waste during off-peak periods; conversely, under-allocation, such as deploying microservices based on average load demand, causes service overload during peak loads, increasing queuing times and failing to meet the request response time SLA. The elastic scaling capability of container services is key to addressing this challenge, but how to scale quickly and accurately to handle bursty service request loads remains a hot research topic and a difficult challenge.
[0004] Current autoscaling methods can be broadly categorized into reactive and predictive approaches. Reactive methods primarily include threshold-based strategies (Kubernetes' built-in scaling strategies), queuing theory, and cybernetics; predictive methods encompass time series analysis, deep learning, and reinforcement learning. Because they make scaling decisions in advance, predictive methods offer better timeliness than reactive methods. Furthermore, reinforcement learning-based scaling methods are adaptive and robust, requiring no prior knowledge and ensuring relatively stable resource utilization and operation even with dynamically changing workloads. Therefore, reinforcement learning (such as Q-learning algorithms) has become a promising approach for autoscaling in cloud environments.
[0005] On the other hand, elastic scaling algorithms can be divided into two types based on their resource scaling methods: vertical scaling and horizontal scaling. Vertical scaling refers to changing the amount of computing resources allocated to a single instance, while horizontal scaling refers to adding or removing instances. Each method has its advantages and disadvantages: vertical scaling is usually faster and smoother, and can handle situations where a single instance encounters problems, but its capabilities are limited; horizontal scaling adjusts application capabilities by increasing or decreasing the number of application instances, is usually slower, and does not react promptly to environmental changes. Furthermore, each instance requires a certain amount of basic resources to maintain its operation, but it can scale more broadly, ensuring that the application's capabilities meet its needs.
[0006] A good scaling strategy should take into account the advantages of both vertical and horizontal scaling. However, current models used for elastic scaling algorithms all suffer from the following problems: First, existing reinforcement learning models for elastic scaling only use horizontal scaling and ignore vertical scaling, which makes it difficult to ensure that the application's capabilities meet the needs when scaling over a large range. Second, although there are already a few techniques that apply reinforcement learning to vertical scaling, the large number of instances in cloud platforms leads to a space explosion problem during reinforcement learning.
[0007] Therefore, how to solve the space explosion problem in existing RL models, while balancing the accuracy and speed of resource scheduling, has become an urgent problem to be solved. Summary of the Invention
[0008] To address the shortcomings of the existing technologies, this invention provides a computer resource scheduling system based on reinforcement learning under a microservice architecture, which can solve the problem of space explosion in existing RL models, while also taking into account the accuracy and speed of resource scheduling.
[0009] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A computer resource scheduling system based on reinforcement learning in a microservice architecture; each instance monitored by the computer resource scheduling system is equipped with a vertical scaling module for resource capacity; the computer resource scheduling system includes a grouper, a monitor, a decision maker, an executor, and a parameter adjuster; the monitor is used to monitor the latest status information of each instance; the status information includes CPU resource utilization and the average response time of requests. The grouper is used to logically group each instance according to preset grouping requirements before each scaling. The grouping requirements are to divide the instances into several Agent groups, several trial groups and one threshold group. The number of instances in each Agent group is a preset value k, the number of instances in each trial group is equal to but not equal to k, and the number of instances in the threshold group is less than that in the Agent groups and the trial groups. Before each scaling operation, the decision-maker constructs group state vectors for each agent group and each test group based on the latest state information of each instance within each agent group and each test group. It then queries the corresponding optimal action in the Q-table of the pre-defined Q-learning reinforcement learning algorithm based on each group state vector. The optimal action includes vertical scaling and horizontal scaling of each instance within the agent group or test group. All optimal actions are then packaged into an execution strategy and sent to the executor. After receiving the execution strategy, the executor performs container resource scaling actions according to preset execution requirements. The execution requirements are as follows: for vertical scaling in the best actions, control the corresponding instance's resource capacity vertical scaling module to expand or shrink accordingly; for horizontal scaling in the best actions, sum the horizontal scaling of all instances in all best actions to obtain the total horizontal scaling value, and then perform the corresponding horizontal scaling within the threshold group according to the total horizontal scaling value. The parameter adjuster is used to obtain the latest status of all instances monitored by the monitor after each execution of the execution strategy by the executor, and to calculate the reward for each best action based on the latest status of all instances and the Q-learning reward function R and update the Q table. The parameter adjuster is also used to analyze the rewards of the trial group and the agent group. If the average reward of the trial group is greater than the average reward of the agent group and the difference exceeds the preset value, the k value is updated to the number of instances in the trial group.
[0010] Basic scheme principles and effects: 1. Before each scaling, the grouper logically groups (i.e., virtual grouping) each instance. In this way, the application instances are logically divided into several agent groups, several trial groups and one threshold group. The same reinforcement learning model is trained together by all groups, which greatly reduces the size of the state space during the training process and can effectively solve the space explosion problem of existing RL models.
[0011] 2. To address complex load changes, the container scaling mechanism combines vertical and horizontal scaling actions, cleverly integrating previous grouping operations. Vertical scaling is performed by instances within the Agent group and the probe group, while horizontal scaling is handled by the threshold group, balancing the accuracy and speed of resource scheduling. In this process, grouping operations simultaneously achieve multiple benefits.
[0012] 3. In this method, the vertical scaling action occurs on each instance of the Agent group and the probe group, which can optimize container resource allocation more finely and improve the resource utilization of the container while ensuring service quality.
[0013] 4. To address long-term, complex load changes and improve the model's adaptability, this invention employs a dynamic grouping method. The setup of the trial group is ingenious. During parallel training, it allows observation of whether the number of instances in the current trial group is better adapted to the current environment than that of the Agent group. If the average reward of the trial group is greater than the average reward of the Agent group, and the difference exceeds a preset value, it indicates that the trial group performs significantly better than the Agent group. In this case, adjusting the number of instances in the trial group within the Agent group allows the number of instances in the Agent group to better adapt to the new environment. Since the Agent group is the primary scheduling target, this dynamic grouping method allows the model to continuously adapt to new load environments during long-term learning (improving from static learning to dynamically generating the size of the learning state space and action space), thus enhancing the model's adaptability.
[0014] In summary, this invention can solve the problem of space explosion in existing RL models, while also taking into account the accuracy and speed of resource scheduling.
[0015] Preferably, when the actuator performs horizontal scaling in the optimal action, if the total horizontal scaling value is positive, then the horizontal expansion corresponding to the total horizontal scaling value is directly performed within the threshold group; if the total horizontal scaling value is negative, then the actuator performs a reactive scaling operation based on the current information of all instances.
[0016] Beneficial effects: If the total horizontal scaling value is positive, directly executing the corresponding expansion ensures timeliness and accuracy. However, if the total horizontal scaling value is negative, direct execution (i.e., using a predictive approach) may result in excessive scaling down, affecting system stability and thus service quality. Therefore, in this case, reactive scaling down, although wasting some computing resources, maximizes service quality stability. Furthermore, since vertical scaling of each instance is already executed predictively, the wasted computing resources are actually very small.
[0017] Preferably, when the grouper logically groups each instance, it also allocates several second trial groups; the number of instances in each second trial group is equal and not equal to k and the number of instances in the trial group; the number of instances in the threshold group is also less than that in the second trial group; before each scaling, the decision-maker also constructs the group state vector of each second trial group and queries the corresponding best action. The parameter adjuster also analyzes the rewards of the second trial group and the Agent group. If the average reward of the trial group or the second trial group is greater than the average reward of the Agent group and the difference exceeds a preset value, the k value is updated to the number of instances within the trial group or the second trial group. If the average rewards of both the trial group and the second trial group are greater than the average reward of the Agent group and the difference exceeds a preset value, the trial group and the second trial group with the larger average reward are selected as the parameter update group, and the k value is updated to the number of instances within the parameter update group.
[0018] Beneficial effects: On the one hand, adding a second trial group allows for parallel training of the reinforcement learning model, which improves its learning efficiency; on the other hand, adding a second trial group allows for better and faster adaptation to changes in the load environment and a more suitable grouping method for the agent groups.
[0019] Preferably, the number of the trial group and the second trial group are equal.
[0020] Preferably, the number of both the trial group and the second trial group is 2.
[0021] Beneficial effects: Since it is only used as an experiment, two groups are sufficient to achieve the goal. This number can be increased as much as possible to ensure the effectiveness of reinforcement learning.
[0022] Preferably, the number of instances within an Agent group is between the number of instances in the first and second test groups.
[0023] Beneficial effects: This setup allows for timely detection when the number of instances in an Agent group is too large or too small, resulting in an inability to adapt to the environment effectively.
[0024] Preferably, the trial group and the second trial group have the same number of instances, where one group has k+1 instances and the other has k-1 instances.
[0025] Beneficial effect: This quantity setting allows for more precise adjustment of the number of instances within an Agent group.
[0026] Preferably, when the grouper logically groups each instance, it first calculates the number of Agent groups, trial groups, and second trial groups, and then, based on the latest status information of each instance, it assigns each instance to each Agent group, trial group, and second trial group respectively through a preset clustering algorithm, and then assigns the remaining instances to the threshold group.
[0027] Beneficial effects: Compared with random grouping, grouping instances with similar characteristics into the same group is more conducive to accelerating model convergence and improving the learning effect of reinforcement models.
[0028] Preferably, the clustering algorithm is the k-means clustering algorithm.
[0029] Preferably, the initial value of k is greater than or equal to 3 and less than or equal to 5.
[0030] Beneficial effect: Such initial values can solve the problem of space explosion in reinforcement learning while ensuring the effectiveness of reinforcement learning. Attached Figure Description
[0031] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein: Figure 1 This is a logical block diagram of the computer resource scheduling system based on reinforcement learning under the microservice architecture of this invention. Figure 2 This is a diagram illustrating the overall architecture of the computer resource scheduling system in the embodiment. Figure 3 This is a schematic diagram of the reinforcement learning model structure in the embodiment. Detailed Implementation
[0032] The following detailed explanation illustrates the specific implementation methods: Example: like Figure 1 , Figure 2 and Figure 3 As shown, this embodiment discloses a computer resource scheduling system based on reinforcement learning under a microservice architecture. It should be noted that each instance monitored by the computer resource scheduling system is equipped with a vertical scaling module for resource capacity. The task performed by the vertical scaling module is to perform vertical scheduling tasks such as expanding or shrinking the computing resources in the computer. In a computer processing system, the inclusion of a vertical scaling module is a basic functional configuration and belongs to the prior art in this field, and will not be elaborated further here.
[0033] The computer resource scheduling system includes a grouper, a monitor, a decision-maker, an executor, and a parameter adjuster. It should be noted that the grouper, monitor, decision-maker, executor, and parameter adjuster are all functional modules within the computer resource scheduling system and are actual electronic hardware. The monitor is used to monitor the latest status information of each instance; this status information includes CPU (Central Processing Unit) resource utilization and the average response time of requests.
[0034] The grouper logically groups instances according to preset grouping requirements before each scaling operation. These requirements divide the instances into several Agent (autonomous agent instance) groups, several trial groups, several second trial groups, and one threshold group. Each Agent group contains a preset number of instances (k), each trial group contains k+1 instances, each second trial group contains k-1 instances, and the threshold group contains fewer than k-1 instances. The number of instances in each trial group and the second trial group is equal, and the initial value of k is greater than or equal to 3 and less than or equal to 5. In this embodiment, there are two trial groups and two second trial groups, and the initial value of k is 4.
[0035] In practice, the grouper first calculates the number of Agent groups, and then, based on the latest status information of each instance, uses a preset k-means clustering algorithm to assign each instance to the Agent group, the trial group, and the second trial group, and then assigns the remaining instances to the threshold group.
[0036] Before each scaling operation, the decision-maker constructs group state vectors for each agent group, trial group, and second trial group based on the latest state information of each instance within each agent group, trial group, and second trial group. Then, it queries the corresponding optimal action from the pre-defined Q-table of the Q-learning reinforcement learning algorithm based on each group state vector. The optimal action includes vertical and horizontal scaling of instances within each agent group or trial group. All optimal actions are packaged into an execution strategy and sent to the executor. In practice, resource utilization and average request response time are discretized to facilitate the practical application of the reinforcement model's learning.
[0037] The executor, upon receiving the execution strategy, performs container resource scaling actions according to preset execution requirements. These requirements are: for vertical scaling in the optimal actions, control the corresponding instance's resource capacity vertical scaling module to expand or shrink accordingly; for horizontal scaling in the optimal actions, sum the horizontal scaling values of all instances in all optimal actions to obtain the total horizontal scaling value, and then perform the corresponding horizontal scaling within a threshold group based on this total value. Specifically, when the executor performs horizontal scaling in the optimal actions, if the total horizontal scaling value is positive, it directly performs the horizontal expansion corresponding to the total horizontal scaling value within the threshold group; if the total horizontal scaling value is negative, the executor performs a reactive scaling operation based on the current information of all instances. In practice, the reactive scaling operation can employ threshold-based strategies (Kubernetes built-in scaling strategies), queuing theories, etc., all of which are conventional techniques in this field and will not be elaborated upon here.
[0038] The parameter adjuster is used to obtain the latest status of all instances monitored by the monitor after each execution strategy is completed by the executor. Based on the latest status of all instances and the Q-learning reward function R, it calculates the reward for each optimal action and updates the Q-table. The parameter adjuster also analyzes the rewards of the trial group, the second trial group, and the agent group. If the average reward of the trial group or the second trial group is greater than the average reward of the agent group and the difference exceeds a preset value, the k value is updated to the number of instances within the trial group or the second trial group. If the average rewards of both the trial group and the second trial group are greater than the average reward of the agent group and the difference exceeds a preset value, the trial group and the second trial group with the larger average reward are selected as the parameter update group, and the k value is updated to the number of instances within the parameter update group.
[0039] In this system, before each scaling operation, the grouper logically groups (i.e., virtually groups) each instance. This logically divides the application instances into several Agent groups, several trial groups, several second trial groups, and one threshold group. Training the same reinforcement learning model using all groups significantly reduces the state space size during training, effectively solving the space explosion problem of existing RL models. For complex load changes, the container scaling action combines vertical and horizontal scaling actions, cleverly integrating the previous grouping operation. Vertical scaling is performed by instances within the Agent group, trial group, and second trial group, while horizontal scaling is performed by the threshold group, balancing the accuracy and speed of resource scheduling. In this process, the grouping operation achieves multiple effects simultaneously. Furthermore, since the vertical scaling action occurs on each instance within the Agent group, trial group, and second trial group, it allows for finer-grained optimization of container resource allocation, improving container resource utilization while ensuring service quality. When performing horizontal scaling, if the total horizontal scaling value is positive, directly executing the corresponding expansion ensures timeliness and accuracy. However, if the total horizontal scaling value is negative, direct execution (i.e., using a predictive approach) may result in excessive scaling down, affecting system stability and thus service quality. Therefore, in this case, reactive scaling down is used. Although it wastes some computing resources, it maximizes the stability of service quality. Furthermore, since vertical scaling of each instance is already executed predictively, the wasted computing resources are actually very small.
[0040] In addition, to address long-term complex load changes and improve the model's adaptability, this invention employs a dynamic grouping method. The setup of the trial group is ingenious. While training in parallel, it allows observation of whether the number of instances in the current group is better adapted to the current environment than the Agent group. If the average reward of the trial group or the second trial group is greater than the average reward of the Agent group, and the difference exceeds a preset value, it indicates that the trial group or the second trial group performs significantly better than the Agent group. At this point, adjusting the number of instances in the Agent group or the trial group allows the number of instances in the Agent group to better adapt to the new environment. Since the Agent group is the primary scheduling target, this dynamic grouping method allows the model to continuously adapt to new load environments during long-term learning (improving from static learning to dynamically generating the learning state space and action space size), thus improving the model's adaptability. Furthermore, since the number of instances in the trial group is one more than that in the Agent group, while the number of instances in the second trial group is one less than that in the Agent group, it can be detected promptly when the number of instances in the Agent group is too large or too small, leading to a failure to adapt to the environment. Moreover, adjusting the number by one each time ensures the accuracy of the adjustment.
[0041] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.
Claims
1. A computer resource scheduling system based on reinforcement learning under a microservice architecture, characterized in that: Each instance monitored by the computer resource scheduling system is equipped with a vertical scaling module for resource capacity; the computer resource scheduling system includes a grouper, a monitor, a decision-maker, an executor, and a parameter adjuster; the monitor is used to monitor the latest status information of each instance; the status information includes CPU resource utilization and the average response time of requests. The grouper is used to logically group each instance according to preset grouping requirements before each scaling. The grouping requirements are to divide the instances into several Agent groups, several trial groups and one threshold group. The number of instances in each Agent group is a preset value k, the number of instances in each trial group is equal to but not equal to k, and the number of instances in the threshold group is less than that in the Agent groups and the trial groups. Before each scaling operation, the decision-maker constructs group state vectors for each agent group and each test group based on the latest state information of each instance within each agent group and each test group. It then queries the corresponding optimal action in the Q-table of the pre-defined Q-learning reinforcement learning algorithm based on each group state vector. The optimal action includes vertical scaling and horizontal scaling of each instance within the agent group or test group. All optimal actions are then packaged into an execution strategy and sent to the executor. After receiving the execution strategy, the executor performs container resource scaling actions according to preset execution requirements. The execution requirements are as follows: for vertical scaling in the best actions, control the corresponding instance's resource capacity vertical scaling module to expand or shrink accordingly; for horizontal scaling in the best actions, sum the horizontal scaling of all instances in all best actions to obtain the total horizontal scaling value, and then perform the corresponding horizontal scaling within the threshold group according to the total horizontal scaling value. The parameter adjuster is used to obtain the latest status of all instances monitored by the monitor after each execution of the execution strategy by the executor, and to calculate the reward for each best action based on the latest status of all instances and the Q-learning reward function R and update the Q table; the parameter adjuster is also used to analyze the rewards of the trial group and the agent group. If the average reward of the trial group is greater than the average reward of the agent group and the difference exceeds the preset value, the k value is updated to the number of instances in the trial group. The initial value of k is greater than or equal to 3 and less than or equal to 5.
2. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 1, characterized in that: When the executor performs horizontal scaling in the optimal action, if the total horizontal scaling value is positive, it directly performs horizontal expansion corresponding to the total horizontal scaling value within the threshold group; if the total horizontal scaling value is negative, the executor performs reactive scaling operation based on the current information of all instances.
3. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 1, characterized in that: When the grouper logically groups each instance, it also allocates several second trial groups; the number of instances in each second trial group is equal and not equal to k and the number of instances in the trial group; the number of instances in the threshold group is also less than that in the second trial group; before each scaling, the decision-maker also constructs the group state vector of each second trial group and queries the corresponding best action. The parameter adjuster also analyzes the rewards of the second trial group and the Agent group. If the average reward of the trial group or the second trial group is greater than the average reward of the Agent group and the difference exceeds a preset value, the k value is updated to the number of instances within the trial group or the second trial group. If the average rewards of both the trial group and the second trial group are greater than the average reward of the Agent group and the difference exceeds a preset value, the trial group and the second trial group with the larger average reward are selected as the parameter update group, and the k value is updated to the number of instances within the parameter update group.
4. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 3, characterized in that: The number of the trial group and the second trial group are equal.
5. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 4, characterized in that: The number of the trial group and the second trial group is 2 each.
6. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 5, characterized in that: The number of instances within the Agent group is between the number of instances in the trial group and the number of instances in the second trial group.
7. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 6, characterized in that: The first group is a trial group, and the second group is a second group. One group has k+1 instances, and the other has k-1 instances.
8. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 3, characterized in that: When the grouper logically groups each instance, it first calculates the number of Agent groups, trial groups, and second trial groups. Then, based on the latest status information of each instance, it uses a preset clustering algorithm to assign each instance to its respective Agent group, trial group, and second trial group. Finally, it assigns the remaining instances to the threshold group.
9. The computer resource scheduling system based on reinforcement learning under a microservice architecture as described in claim 8, characterized in that: The clustering algorithm is the k-means clustering algorithm.