An adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning

Abstract

The present invention relates to an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning, which belongs to the field of cloud computing resource management. The method proposes an adaptive multi-granularity cloud resource orchestration framework AMG-RL, which adaptively groups cloud tasks into K task clusters, predicts the resource requirements of different task clusters, and performs multi-granularity cloud resource orchestration based on the resource requirements of the divided different task clusters and the corresponding task clusters to achieve task resource allocation and scheduling; it also includes performance feedback on the strategy of multi-granularity cloud resource orchestration to tune the strategy of multi-granularity cloud resource orchestration. The present invention can adapt to the diversity of cloud tasks, reveal the similarities and complementarities between tasks, lay the foundation for differentiated resource allocation, and accurately predict future load changes, and can maximize resource utilization efficiency and reduce costs while meeting task performance constraints, and continuously improve the long-term performance of the system through strategy improvement.

Description

Technical Field

[0001] The present invention belongs to the field of cloud computing resource management, and in particular relates to an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning. Background Art

[0002] With the widespread application and rapid development of cloud computing, how to efficiently manage increasingly complex cloud infrastructure resources and meet the diverse needs of users and applications has become a key issue that cloud service providers need to solve. Traditional static resource configuration methods and coarse-grained dynamic adjustment strategies face many challenges and are difficult to adapt to the high dynamics, high heterogeneity and high uncertainty of the cloud environment. The main problems include:

[0003] (1) Cloud workloads present complex and dynamic characteristics, while static resource allocation methods lack the ability to adapt to load changes, resulting in a prominent imbalance between resource supply and demand.

[0004] (2) There are a large number of heterogeneous tasks in the cloud environment. Coarse-grained resource scheduling methods cannot match personalized task requirements and performance goals, resulting in problems such as resource waste and task delays.

[0005] (3) The uncertainty of cloud systems, such as sudden load behavior, software and hardware failures, etc., poses great challenges to resource management, and there is an urgent need for intelligent exception handling and dynamic decision-making mechanisms.

[0006] (4) The trade-off between cloud resource utilization efficiency and energy cost is becoming increasingly prominent, and there is an urgent need to coordinate and optimize multi-dimensional heterogeneous resources such as computing, storage, and networking to balance system performance and economic benefits.

[0007] In response to the above pain points, the industry and academia have conducted fruitful research. Some scholars have proposed an online resource allocation method based on reinforcement learning in cloud environments, CARLO, which adapts to load changes by learning sequential decision strategies. Some institutions have designed a two-layer resource scheduling framework DRESS for heterogeneous workloads, combining task clustering and deep reinforcement learning technology to achieve fine-grained dynamic configuration of cloud resources. Other scholars have proposed a multi-objective cloud resource optimization model CRUZE that maximizes resource utilization and minimizes energy consumption to solve the joint optimization problem of server switching and task placement.

[0008] Although existing work has made encouraging progress in intelligent cloud resource management, how to organically integrate multiple intelligent computing paradigms, collaboratively optimize scheduling decisions across multiple granularities such as data centers, racks, servers, virtual machines, and balance multiple performance goals are still key problems that need to be solved. Summary of the invention

[0009] The purpose of the present invention is to solve the problem of intelligent scheduling and resource optimization of massive heterogeneous workloads in increasingly complex cloud environments, provide an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning, and propose an adaptive multi-granularity cloud resource orchestration framework AMG-RL, which innovatively combines integrated clustering, time series prediction, and hierarchical deep reinforcement learning to achieve end-to-end intelligent management and control of cloud resources.

[0010] To achieve the above objectives, the technical solution of the present invention is: an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning, an adaptive multi-granularity cloud resource orchestration framework AMG-RL is proposed, cloud tasks are adaptively grouped into K task clusters, the resource requirements of different task clusters are predicted, and multi-granularity cloud resource orchestration is performed based on the divided different task clusters and the resource requirements of the corresponding task clusters to realize task resource allocation and scheduling.

[0011] In one embodiment of the present invention, it also includes providing performance feedback on the strategy of multi-granularity cloud resource orchestration to optimize the strategy of multi-granularity cloud resource orchestration.

[0012] In one embodiment of the present invention, the method comprises the following steps:

[0013] Step S1, integrated fuzzy clustering: the cloud tasks are clustered using an integrated fuzzy C-means algorithm; the input of the integrated fuzzy C-means algorithm is a d-dimensional vector set X representing the resource consumption characteristics of the task, the number of clusters K, and the integration scale r; for each base learner, the fuzzy membership matrix and cluster center are randomly initialized, and the objective function is iteratively optimized until convergence; finally, the comprehensive membership matrix is ​​obtained by combining the clustering results and solving the quadratic programming problem, and the tasks are hard divided and grouped into different K task clusters;

[0014] Step S2, cluster-level workload prediction: for each task cluster, an independent ARIMA time series model is constructed to predict the future resource demand of each task cluster; the resource usage sequence of each task cluster is tested for stationarity, the order of the ARIMA time series model is determined using the autocorrelation and partial correlation functions or the information criterion, the ARIMA time series model parameters are estimated by maximizing the likelihood function, and multi-step resource demand forecast values ​​are generated;

[0015] Step S3, hierarchical deep reinforcement learning orchestration: design a hierarchical deep reinforcement learning architecture, and collaboratively learn multi-granularity resource allocation strategies through the global data center controller and the local cluster controller; model the cloud environment as a hierarchical Markov decision process, define the global state space, multi-granularity action space, transfer function and multi-objective reward function; use the deep Q network algorithm to train the data center controller and cluster controller, the data center controller is responsible for coarse-grained decision-making, and the cluster controller is responsible for fine-grained task resource allocation, and achieve strategy coordination between controllers by setting sub-goals.

[0016] In one embodiment of the present invention, the method further includes the following steps:

[0017] Step S4, performance feedback: Continuously collect multiple feedback signals, including indicators of resource utilization, load prediction deviation, task response time, energy consumption, and SLA breach rate, to evaluate the resource orchestration performance of AMG-RL and to tune the strategy of multi-granularity cloud resource orchestration.

[0018] In one embodiment of the present invention, step S1 is specifically implemented as follows:

[0019] The input of the integrated fuzzy C-means algorithm is a d-dimensional vector set of task resource consumption characteristics , the number of clusters K and the ensemble size r, where represents the d-dimensional feature vector of the i-th sample; the hard partitioning result of the output task of the integrated fuzzy C-means algorithm is ; For the base learner , , randomly initialize the fuzzy membership matrix and the cluster center matrix ; Within the maximum number of iterations, alternately optimize the membership degree and cluster center until the convergence condition is met;

[0020] Membership Indicates the degree of belonging of sample i to the kth task cluster, and the update formula is:

[0021]

[0022] Cluster Center Represents the sample mean vector of the kth task cluster, and the update formula is:

[0023]

[0024] in is the fuzzification degree parameter. After r base learners are trained independently, they are integrated. The matching degree between a sample i and the kth cluster center is defined as:

[0025]

[0026] The final comprehensive membership Obtained by solving the following optimization problem:

[0027]

[0028] The optimization problem is transformed into a quadratic programming problem with linear constraints and solved. Finally, the hard clustering labels of the task are obtained by thresholding:

[0029]

[0030] Output divides the cloud tasks into K task clusters ,The task characteristics within each task cluster are similar and the resource requirements are consistent, but there are large differences between clusters.

[0031] In one embodiment of the present invention, step S2 is specifically implemented as follows:

[0032] The ARIMA time series model is used to capture the temporal dynamics of the overall resource consumption of each task cluster; for the kth task cluster, its resource consumption sequence is obtained ,in represents the sum of resource consumption of all tasks in the kth task cluster at time t, and T represents the number of time periods; the ARIMA time series model assumes that the resource consumption at the current moment is the sum of the past The weighted sum of the periods, taking into account the past The impact of the forecast error in the future; is the difference operator, It means to make d-order difference on the sequence; then The time series model is defined as:

[0033]

[0034] in

[0035]

[0036] The above formula is The polynomial sum of the autoregressive coefficients The moving average coefficient polynomial of order, is the autoregression coefficient, is the moving average coefficient, is a constant term, B is a lag operator, , j is the number of time lag steps, is a zero-mean white noise sequence; perform a stationarity test on the resource consumption sequence Y. If it is not stationary, perform a differential operation until it is stationary; determine the optimal order based on the sample autocorrelation coefficient ACF and partial autocorrelation coefficient PACF graphs and , use maximum likelihood estimation or least squares method to solve the ARIMA time series model parameters , , ; After model training is completed, resource consumption for the next h period Make a recursive prediction:

[0037]

[0038] in Express The predicted value is the predicted value of resource usage of the kth task cluster at time T+h. Represents conditional expectation, and finally outputs the resource demand estimate of each task cluster in several future time windows , h is the prediction step, and H is the prediction range, that is, the number of prediction periods required.

[0039] In one embodiment of the present invention, step S3 is specifically implemented as follows:

[0040] The cloud environment is modeled as a hierarchical Markov decision process (MDP), where the global MDP corresponds to coarse-grained cross-cluster resource coordination and the local MDP corresponds to fine-grained intra-cluster / task-level resource allocation. The global MDP is defined as a six-tuple :

[0041] State space S: The global system state s includes the resource utilization of each task cluster , load forecast value , average task response time , Energy consumption , SLA default rate ;

[0042] Action Space : Global scheduling action This includes increasing or decreasing the number of physical machines in different clusters, adjusting VM resource configuration, and migrating VMs;

[0043] Transition probability : Defined as performing an action in state s Then transfer to the next state s ’ Probability ; Considering the uncertainty of cloud environment, the transfer function is a random mapping;

[0044] Reward Function : Comprehensively consider the performance indicators of the system and design a multi-objective reward function , weighing resource utilization, response time, energy consumption, and SLA factors;

[0045] Discount Factor : Controls the importance of future rewards, The larger it is, the more the strategy focuses on long-term returns;

[0046] The local MDP is defined as :

[0047] State Space :state Contains fine-grained monitoring information within a cluster;

[0048] Action Space :action This includes adjusting task priorities within the cluster, changing task resource quotas, and placing and migrating tasks;

[0049] Reward Function : Partial Reward Focus on the performance and fairness factors of tasks within the cluster;

[0050] , are the transition probability and discount factor respectively;

[0051] Based on global MDP and local MDP, hierarchical deep reinforcement learning is used to learn resource scheduling strategies from bottom to top and end to end; the global scheduler uses the DQN algorithm, takes the system state s as input, and outputs coarse-grained scheduling decisions ; The local scheduler uses the DQN algorithm to further refine the task resource allocation and scheduling within the cluster based on the global decision; the two-level agents achieve strategy coordination through sub-goal decomposition; the DQN algorithm in the global scheduler uses the following Q value iterative update rules:

[0052]

[0053] in is the learning rate, is the state-action value function, evaluating the action taken in state s Long-term cumulative rewards;

[0054] The DQN algorithm Q value update in the local scheduler introduces the global scheduling decision as an additional input:

[0055]

[0056] in is the learning rate, a global decision Affect the state transition and reward of the local MDP; by As the conditional input of the local Q network, the collaborative optimization of the hierarchical strategy is achieved.

[0057] In one embodiment of the present invention, step S4 is specifically implemented as follows:

[0058] Continuously collect multi-granular monitoring data, including resource usage, task queuing and running status, and energy consumption at the cluster / node / VM / task level; on this basis, calculate a series of macro and micro KPI indicators, including:

[0059] Average resource utilization:

[0060] Average task waiting time:

[0061] Average task completion rate:

[0062] in, The resource utilization of the kth cluster at time t, where T represents the number of time cycles and i is the task index. represents the start execution time of task i, represents the submission time of task i, represents the completion time of task i, is the deadline of task i, Indicates the total number of tasks, It is an indicator function. When the actual completion time of the task minus the submission time is greater than the deadline, the indicator function returns 1, otherwise it returns 0.

[0063] Compared with the prior art, the present invention has the following beneficial effects: The AMG-RL proposed in the present invention has demonstrated excellent performance in the simulation experiment of real loads. Compared with the mainstream heuristic algorithm and single-agent DRL method, it has achieved significant performance advantages in terms of resource utilization improvement, task performance guarantee, energy consumption optimization, adaptability and generalization ability, which are mainly reflected in the following three aspects:

[0064] 1. Improve the utilization of cloud environment resources in multiple dimensions

[0065] AMG-RL cleverly combines integrated fuzzy clustering and ARIMA demand forecasting to accurately characterize the resource usage patterns of different task clusters and maximize resource utilization efficiency while ensuring QoS.

[0066] 2. Reduce energy consumption while ensuring mission performance

[0067] While improving resource utilization, AMG-RL does not sacrifice the QoS of the task.

[0068] 3. Excellent adaptability and generalization capabilities

[0069] The scheduling knowledge learned by AMG-RL from one workload has a certain universality and can significantly reduce the adaptation cost in a new environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 This is the overall architecture diagram of the adaptive multi-granularity cloud resource orchestration framework AMG-RL of the present invention.

[0071] Figure 2 This is a diagram of the hierarchical deep reinforcement learning (DRL) architecture of the present invention.

[0072] Figure 3The comparison curves showing how CPU and memory utilization change with the number of tasks under different algorithms.

[0073] Figure 4 Comparison between algorithms showing average task waiting time, response time, and SLA breach rate.

[0074] Figure 5 Energy consumption-latency performance comparison. DETAILED DESCRIPTION

[0075] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0076] The present invention provides an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning, and proposes an adaptive multi-granularity cloud resource orchestration framework AMG-RL, which adaptively groups cloud tasks into K task clusters, predicts the resource requirements of different task clusters, and performs multi-granularity cloud resource orchestration based on the divided different task clusters and the resource requirements of the corresponding task clusters to achieve task resource allocation and scheduling. It also includes performance feedback on the strategy of multi-granularity cloud resource orchestration to optimize the strategy of multi-granularity cloud resource orchestration. The method comprises the following steps:

[0077] Step S1, integrated fuzzy clustering: the cloud tasks are clustered using an integrated fuzzy C-means algorithm; the input of the integrated fuzzy C-means algorithm is a d-dimensional vector set X representing the resource consumption characteristics of the task, the number of clusters K, and the integration scale r; for each base learner, the fuzzy membership matrix and cluster center are randomly initialized, and the objective function is iteratively optimized until convergence; finally, the comprehensive membership matrix is ​​obtained by combining the clustering results and solving the quadratic programming problem, and the tasks are hard divided and grouped into different K task clusters;

[0078] Step S2, cluster-level workload prediction: for each task cluster, an independent ARIMA time series model is constructed to predict the future resource demand of each task cluster; the resource usage sequence of each task cluster is tested for stationarity, the order of the ARIMA time series model is determined using the autocorrelation and partial correlation functions or the information criterion, the ARIMA time series model parameters are estimated by maximizing the likelihood function, and multi-step resource demand forecast values ​​are generated;

[0079] Step S3, hierarchical deep reinforcement learning orchestration: design a hierarchical deep reinforcement learning architecture, and collaboratively learn multi-granularity resource allocation strategies through the global data center controller and the local cluster controller; model the cloud environment as a hierarchical Markov decision process, define the global state space, multi-granularity action space, transfer function and multi-objective reward function; use the deep Q network algorithm to train the data center controller and cluster controller, the data center controller is responsible for coarse-grained decision-making, and the cluster controller is responsible for fine-grained task resource allocation, and achieve strategy coordination between controllers by setting sub-goals;

[0080] Step S4, performance feedback: Continuously collect multiple feedback signals, including indicators of resource utilization, load prediction deviation, task response time, energy consumption, and SLA breach rate, to evaluate the resource orchestration performance of AMG-RL and to tune the strategy of multi-granularity cloud resource orchestration.

[0081] The following is the specific implementation process of the present invention.

[0082] like Figure 1 As shown, the present invention provides an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning, and proposes an adaptive multi-granularity cloud resource orchestration framework AMG-RL, which consists of four core modules: integrated fuzzy clustering, cluster-level workload prediction, hierarchical deep reinforcement learning orchestration, and performance feedback loop.

[0083] 1. Integrated fuzzy clustering module

[0084] This module uses an improved fuzzy C-means clustering algorithm to adaptively group massive heterogeneous cloud tasks. With the help of ensemble learning, the results of multiple clusterers are weighted and fused to improve clustering robustness. The ensemble fuzzy clustering process is specifically implemented as follows.

[0085] The input of the integrated fuzzy C-means algorithm is a d-dimensional vector set of task resource consumption characteristics , the number of clusters K and the ensemble size r, where represents the d-dimensional feature vector of the i-th sample; the hard partitioning result of the output task of the integrated fuzzy C-means algorithm is ; For the base learner , , randomly initialize the fuzzy membership matrix and the cluster center matrix ; Within the maximum number of iterations, alternately optimize the membership degree and cluster center until the convergence condition is met;

[0086] Membership Indicates the degree of belonging of sample i to the kth task cluster, and the update formula is:

[0087]

[0088] Cluster Center Represents the sample mean vector of the kth task cluster, and the update formula is:

[0089]

[0090] in is the fuzzification degree parameter. After r base learners are trained independently, they are integrated. The matching degree between a sample i and the kth cluster center is defined as:

[0091]

[0092] The final comprehensive membership Obtained by solving the following optimization problem:

[0093]

[0094] The optimization problem is transformed into a quadratic programming problem with linear constraints and solved. Finally, the hard clustering labels of the task are obtained by thresholding:

[0095]

[0096] Output divides the cloud tasks into K task clusters ,The task features within each task cluster are similar and the resource requirements are consistent, but there are large differences among clusters.,The clustering results lay the foundation for fine-grained resource allocation.

[0097] 2. Cluster-level workload prediction module Based on integrated clustering, this module builds personalized resource demand prediction models for different task clusters. Considering the scale effect and periodicity of cloud load, this module uses the autoregressive differential moving average (ARIMA) time series model to capture the temporal dynamics of the overall resource consumption of each task cluster. The prediction process is specifically implemented as follows.

[0098] For the kth task cluster, obtain its resource consumption sequence ,in represents the sum of resource consumption of all tasks in the kth task cluster at time t, and T represents the number of time periods; the ARIMA time series model assumes that the resource consumption at the current moment is the sum of the past The weighted sum of the periods, taking into account the past The impact of the forecast error in the future; is the difference operator, It means to make d-order difference on the sequence; then The time series model is defined as:

[0099]

[0100] in

[0101]

[0102] The above formula is The polynomial sum of the autoregressive coefficients The moving average coefficient polynomial of order, is the autoregression coefficient, is the moving average coefficient, is a constant term, B is a lag operator, , j is the number of time lag steps, is a zero-mean white noise sequence; perform a stationarity test on the resource consumption sequence Y. If it is not stationary, perform a differential operation until it is stationary; determine the optimal order based on the sample autocorrelation coefficient ACF and partial autocorrelation coefficient PACF graphs and , use maximum likelihood estimation or least squares method to solve the ARIMA time series model parameters , , ; After model training is completed, resource consumption for the next h period Make a recursive prediction:

[0103]

[0104] in Express The predicted value is the predicted value of resource usage of the kth task cluster at time T+h. Represents conditional expectation, and finally outputs the resource demand estimate of each task cluster in several future time windows , h is the prediction step length, and H is the prediction range, i.e., the number of prediction periods required. This fine-grained demand prediction helps to plan resource quotas in advance and meet the service quality requirements of different tasks.

[0105] 3. Hierarchical Deep Reinforcement Learning Scheduling Module Based on the task clustering and demand forecasting, this module designs a hierarchical reinforcement learning architecture to autonomously learn multi-granularity resource scheduling strategies. The specific implementation is as follows.

[0106] The cloud environment is modeled as a hierarchical Markov decision process (MDP), where the global MDP corresponds to coarse-grained cross-cluster resource coordination and the local MDP corresponds to fine-grained intra-cluster / task-level resource allocation. The global MDP is defined as a six-tuple :

[0107] State space S: The global system state s includes the resource utilization of each task cluster , load forecast value , average task response time , Energy consumption , SLA default rate ;

[0108] Action Space : Global scheduling action This includes increasing or decreasing the number of physical machines in different clusters, adjusting VM resource configuration, and migrating VMs;

[0109] Transition probability : Defined as performing an action in state s Then transfer to the next state s ’ Probability ; Considering the uncertainty of cloud environment, the transfer function is a random mapping;

[0110] Reward Function : Comprehensively consider the performance indicators of the system and design a multi-objective reward function , weighing resource utilization, response time, energy consumption, and SLA factors;

[0111] Discount Factor : Controls the importance of future rewards, The larger it is, the more the strategy focuses on long-term returns;

[0112] The local MDP is defined as :

[0113] State Space :state Contains fine-grained monitoring information within a cluster;

[0114] Action Space :action This includes adjusting task priorities within the cluster, changing task resource quotas, and placing and migrating tasks;

[0115] Reward Function : Partial Reward Focus on the performance and fairness factors of tasks within the cluster;

[0116] , are the transition probability and discount factor respectively;

[0117] Based on global MDP and local MDP, hierarchical deep reinforcement learning is used to learn resource orchestration strategies from bottom to top and end to end; Figure 2 As shown in Figure 2, the global scheduler uses the DQN algorithm, takes the system state s as input, and outputs a coarse-grained scheduling decision. ; The local scheduler uses the DQN algorithm to further refine the task resource allocation and scheduling within the cluster based on the global decision; the two-level agents achieve strategy coordination through sub-goal decomposition; the DQN algorithm in the global scheduler uses the following Q value iterative update rules:

[0118]

[0119] in is the learning rate, is the state-action value function, evaluating the action taken in state s Long-term cumulative rewards;

[0120] The DQN algorithm Q value update in the local scheduler introduces the global scheduling decision as an additional input:

[0121]

[0122] in is the learning rate, a global decision Affect the state transition and reward of the local MDP; by As the conditional input of the local Q network, the collaborative optimization of the hierarchical strategy is achieved.

[0123] The training process of hierarchical DRL orchestration is as follows: the input is the hierarchical MDP of the cloud environment, the parameters of the global and local DQN, the preset sub-goal set, etc. The output is the two-level optimal orchestration strategy obtained through training. Specifically, in each training round, the agent interacts with the cloud environment for multiple scheduling cycles. In each cycle t, the global scheduler Making scheduling decisions , decompose the sub-goals into , used to guide the local scheduler. The scheduler in each cluster is in the local state and global decision making Based on this, further optimize the scheduling action of this cluster . After the cloud environment performs a series of resource adjustments, it enters the next state and returns global and local rewards. These transfer samples are stored in two-level experience buffer pools. The scheduler regularly samples training data from the buffer pool and updates the Q network parameters based on the time difference error. Through policy exploration, multi-granularity scheduling decisions are continuously optimized, and finally the joint optimal strategy is obtained. The innovation of this module is: 1) The cloud environment is finely modeled through hierarchical MDP, which reduces the state-action space and reduces the difficulty of exploration; 2) End-to-end learning of cross-granular optimal scheduling strategies significantly improves sample efficiency; 3) Collaborative optimization of multiple performance indicators to maximize resource utilization while meeting QoS constraints. A large number of simulation experiments show that this method achieves significant performance improvement in complex cloud environments compared with single-agent DRL and heuristic scheduling algorithms.

[0124] 4. Performance Feedback Loop Module

[0125] This module focuses on monitoring and evaluating the impact of scheduling decisions on cloud system performance, and provides feedback for online tuning of orchestration strategies. As shown in Figure 1, this module continuously collects multi-granular monitoring data, including cluster / node / VM / task level resource usage (such as CPU, memory, I / O, etc.), task queuing and running status (such as ready queue length, task start / completion time), energy consumption (such as CPU power, air conditioning power consumption), etc. The specific implementation is as follows.

[0126] Continuously collect multi-granular monitoring data, including resource usage, task queuing and running status, and energy consumption at the cluster / node / VM / task level; on this basis, calculate a series of macro and micro KPI indicators, including:

[0127] Average resource utilization:

[0128] Average task waiting time:

[0129] Average task completion rate:

[0130] in, The resource utilization of the kth cluster at time t, where T represents the number of time cycles and i is the task index. represents the start execution time of task i, represents the submission time of task i, represents the completion time of task i, is the deadline of task i, Indicates the total number of tasks, is an indicative function. When the actual completion time of the task minus the submission time is greater than the deadline, the indicative function returns 1, otherwise it returns 0. These KPI indicators describe the overall operating status of the cloud system under the action of the multi-granularity cloud resource orchestration strategy π in multiple dimensions, and provide feedback to optimize the multi-granularity cloud resource orchestration strategy.

[0131] Example:

[0132] As shown in Figure 1, assume that there are N tasks to be scheduled in the cloud environment, and each task has a d-dimensional resource consumption characteristic. The goal of the task scheduling problem is to minimize the system energy consumption E and maximize the resource utilization , while ensuring the average response time of the task Does not exceed the preset threshold .

[0133] First, the fuzzy clustering module is integrated to adaptively cluster N tasks. Enter the improved FCM algorithm, set the number of clusters K and the integration scale After multiple rounds of iterative optimization, the membership matrix of r clustering results is obtained. The comprehensive membership is calculated, and the tasks are hard divided to obtain K task clusters. The characteristics of tasks within each cluster are similar, while the differences between clusters are large.

[0134] Next, for each task cluster ,The cluster-level workload prediction module builds a time series model of its resource usage intensity. The total resource consumption of all tasks in the historical T time windows . Perform stationarity test and difference processing on the sequence, and use sample autocorrelation and partial correlation coefficients to determine the optimal ARIMA model order. After estimating the model parameters based on the maximum likelihood method, the resource requirements for the next H windows are Make recursive predictions.

[0135] Based on the task clustering and demand forecast, the hierarchical DRL scheduling module starts to train the two-level resource scheduling strategy. Each training round contains multiple scheduling cycles. In each cycle t, the global DQN takes the system state is the input and output is the coarse-grained scheduling action , assign task clusters to different nodes and determine the resource quota of each node. The local DQN uses the local state of cluster k and global decision making The cloud environment executes these scheduling instructions and returns the global reward and local rewards DQN performs gradient updates on the Q network parameters. Generated by the policy network Greedy exploration, the two-level agents gradually optimize their scheduling strategies.

[0136] Finally, the performance feedback loop monitors the operation status of the cloud environment in real time and evaluates the effectiveness of the scheduling strategy π. This module collects multi-granular monitoring data such as tasks, VMs, and physical machines, and calculates the average resource utilization , Average response time and energy consumption KPI indicators such as π and π describe the impact of π on the overall performance of the system. To achieve adaptive optimization, this module feeds back the KPI to the DRL orchestration module, dynamically adjusts the reward weights of each target, guides the strategy search direction, and inputs the KPI changes as additional state information into the agent to enhance the strategy's ability to perceive the environment.

[0137] Each module works together to output the optimal resource scheduling strategy for multiple performance goals. A large number of simulation experiments show that AMG-RL fully exploits the heterogeneity of tasks and resources in the cloud environment, and achieves more refined resource matching through task clustering and demand prediction; hierarchical DRL orchestration reduces the state-action space and significantly improves the efficiency of policy learning; the KPI feedback loop enhances the adaptability of the policy, enabling it to dynamically adapt to changes in load and environment. Compared with mainstream heuristic and single-agent DRL scheduling algorithms, AMG-RL has achieved a 20%-50% performance improvement in indicators such as resource utilization, task response time, and energy efficiency, demonstrating the superiority of the intelligent adaptive scheduling paradigm.

[0138] The AMG-RL framework proposed in this paper provides a new idea for the intelligent scheduling of massive heterogeneous cloud environments. By integrating multi-intelligent computing technologies and adaptively learning multi-granularity optimal scheduling strategies, AMG-RL is expected to become a key enabling technology for the next generation of cloud computing resource management and promote the intelligent development of cloud computing. Future research directions include incorporating dynamic disturbance factors, task dependencies, and communication topology in the cloud environment, considering more scheduling objectives such as fairness and robustness, and exploring the application of cutting-edge technologies such as distributed multi-agent reinforcement learning in cloud resource scheduling.

[0139] The AMG-RL proposed in this paper shows excellent performance in the simulation experiment of real load. Compared with the mainstream heuristic algorithm and single-agent DRL method, it has achieved significant performance advantages in terms of resource utilization improvement, task performance guarantee, energy consumption optimization, adaptability and generalization ability, which are mainly reflected in the following three aspects:

[0140] 1. Improve the utilization of cloud environment resources in multiple dimensions

[0141] AMG-RL cleverly combines integrated fuzzy clustering and ARIMA demand forecasting to accurately describe the resource usage patterns of different task clusters, maximizing resource utilization efficiency while ensuring QoS. Figure 3 It is shown that by performing a fine-grained analysis of the CPU and memory usage distribution of different task clusters, the resource usage distribution curve obtained by AMG-RL is closest to the actual task consumption (TaskUsage), reflecting its advantage in fine-grained resource supply and demand matching. Figure 4 ,During the dynamic scheduling of Google cluster tasks, the average resource utilization of AMG-RL is always 10%-20% higher than other methods and remains stable, thanks to its ability to continuously learn and optimize the dynamic scheduling strategy through hierarchical ,DRL.

[0142] 2. Reduce energy consumption while ensuring mission performance

[0143] While improving resource utilization, AMG-RL does not sacrifice the QoS of tasks. Figure 5 It is shown that AMG-RL reduces the total energy consumption of the data center by 10%-20%. This is due to its global optimization of energy consumption-performance balance based on meeting the SLA requirements of local tasks through task clustering and hierarchical scheduling. Figure 5 The results show that in the two-dimensional performance space of energy-delay trade-off, the performance curve of AMG-RL completely "envelopes" other algorithms, confirming its significant advantages in multi-objective optimization. It is worth mentioning that even under strict SLA requirements (such as delay ≤ 100ms), AMG-RL can stabilize the average CPU utilization at a high level of more than 70%, while other methods have to over-allocate resources to meet SLA, and the utilization rate drops significantly.

[0144] 3. Excellent adaptability and generalization capabilities

[0145] Figure 4 The excellent stability of AMG-RL in dynamic task scheduling has been preliminarily demonstrated. Its adaptive ability in dealing with cluster anomalies was further investigated. When a short-term task burst occurs, AMG-RL can make adjustments quickly, restoring the average task response to normal levels in less than 1 / 3 of the time, while other methods show obvious jitter and instability. This is attributed to the online policy self-optimization capability given to AMG-RL by the KPI feedback loop. The present invention also tested the generalization of AMG-RL on heterogeneous workloads of Microsoft and Alibaba. The results show that the scheduling strategy trained on Google data can be directly migrated and maintain excellent performance, which is better than the baseline method trained from scratch overall. This confirms that the scheduling knowledge learned by AMG-RL from a workload has a certain universality and can significantly reduce the adaptation cost in a new environment.

[0146] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. An adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning, characterized in that: An adaptive multi-granularity cloud resource orchestration framework AMG-RL is proposed. The cloud tasks are adaptively grouped into K task clusters, and the resource requirements of different task clusters are predicted. Based on the divided different task clusters and the resource requirements of the corresponding task clusters, multi-granularity cloud resource orchestration is performed to achieve task resource allocation and scheduling. The method also includes performance feedback on the strategy of multi-granularity cloud resource orchestration to optimize the strategy of multi-granularity cloud resource orchestration. The method comprises the following steps: Step S1, integrated fuzzy clustering: the cloud tasks are clustered using an integrated fuzzy C-means algorithm; the input of the integrated fuzzy C-means algorithm is a d-dimensional vector set X representing the resource consumption characteristics of the task, the number of clusters K, and the integration scale r; for each base learner, the fuzzy membership matrix and cluster center are randomly initialized, and the objective function is iteratively optimized until convergence; finally, the comprehensive membership matrix is ​​obtained by combining the clustering results and solving the quadratic programming problem, and the tasks are hard divided and grouped into different K task clusters; Step S2, cluster-level workload prediction: for each task cluster, an independent ARIMA time series model is constructed to predict the future resource demand of each task cluster; the resource usage sequence of each task cluster is tested for stationarity, the order of the ARIMA time series model is determined using the autocorrelation and partial correlation functions or the information criterion, the ARIMA time series model parameters are estimated by maximizing the likelihood function, and multi-step resource demand forecast values ​​are generated; Step S3: Hierarchical Deep Reinforcement Learning Orchestration: Design a hierarchical deep reinforcement learning architecture to collaboratively learn multi-granularity resource allocation strategies through the global data center controller and the local cluster controller; The cloud environment is modeled as a hierarchical Markov decision process, and the global state space, multi-granularity action space, transfer function and multi-objective reward function are defined. The deep Q-network algorithm is used to train the data center controller and cluster controller. The data center controller is responsible for coarse-grained decision-making, and the cluster controller is responsible for fine-grained task resource allocation. Strategy coordination between controllers is achieved by setting sub-goals.

2. According to claim 1, an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning is characterized in that: The following steps are also included: Step S4, performance feedback: Continuously collect multiple feedback signals, including indicators of resource utilization, load prediction deviation, task response time, energy consumption, and SLA breach rate, to evaluate the resource orchestration performance of AMG-RL and to tune the strategy of multi-granularity cloud resource orchestration.

3. According to claim 2, an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning is characterized in that: Step S1 is specifically implemented as follows: The input of the integrated fuzzy C-means algorithm is a d-dimensional vector set of task resource consumption characteristics , the number of clusters K and the ensemble size r, where represents the i-th sample, and the i-th sample is a d-dimensional feature vector; the hard partitioning result of the output task of the integrated fuzzy C-means algorithm is ; For the base learner , , randomly initialize the fuzzy membership matrix and the cluster center matrix ; Within the maximum number of iterations, alternately optimize the membership degree and cluster center until the convergence condition is met; Membership Indicates the degree of belonging of sample i to the kth task cluster, and the update formula is: ； Cluster Center Represents the sample mean vector of the kth task cluster, and the update formula is: ； in is the fuzzification degree parameter. After r base learners are trained independently, they are integrated. The matching degree between a sample i and the kth cluster center is defined as: ； The final comprehensive membership Obtained by solving the following optimization problem: ； The optimization problem is transformed into a quadratic programming problem with linear constraints and solved. Finally, the hard clustering labels of the task are obtained by thresholding: ； Output divides the cloud tasks into K task clusters ,The task characteristics within each task cluster are similar and the resource requirements are consistent, but there are large differences between clusters.

4. According to claim 3, the method for adaptive multi-granularity cloud resource orchestration based on hierarchical reinforcement learning is characterized in that: Step S2 is specifically implemented as follows: The ARIMA time series model is used to capture the temporal dynamics of the overall resource consumption of each task cluster; for the kth task cluster, its resource consumption sequence is obtained ,in represents the sum of resource consumption of all tasks in the kth task cluster at time t, and T represents the number of time periods; the ARIMA time series model assumes that the resource consumption at the current moment is the sum of the past The weighted sum of the periods, taking into account the past The impact of the forecast error in the future; is the difference operator, It means to make d-order difference on the sequence; then The time series model is defined as: ； in ； The above formula is The polynomial sum of the autoregressive coefficients The moving average coefficient polynomial of order, is the autoregression coefficient, is the moving average coefficient, is a constant term, B is a lag operator, , j is the number of time lag steps, is a zero-mean white noise sequence; perform a stationarity test on the resource consumption sequence Y. If it is not stationary, perform a differential operation until it is stationary; determine the optimal order based on the sample autocorrelation coefficient ACF and partial autocorrelation coefficient PACF graphs and , use maximum likelihood estimation or least squares method to solve the ARIMA time series model parameters , , ; After model training is completed, resource consumption for the next h period Make a recursive prediction: ； in Express The predicted value is the predicted value of resource usage of the kth task cluster at time T+h. Represents conditional expectation, and finally outputs the resource demand estimate of each task cluster in several future time windows , h is the prediction step, and H is the prediction range, that is, the number of prediction periods required.

5. According to claim 4, an adaptive multi-granularity cloud resource orchestration method based on hierarchical reinforcement learning is characterized in that: Step S3 is specifically implemented as follows: The cloud environment is modeled as a hierarchical Markov decision process (MDP), where the global MDP corresponds to the coarse-grained cross-cluster resource coordination and the local MDP corresponds to the fine-grained intra-cluster / task-level resource allocation. The global MDP is defined as a six-tuple : State space S: The global system state s includes the resource utilization of each task cluster , load forecast value , average task response time , Energy consumption , SLA default rate ; Action Space : Global scheduling action This includes increasing or decreasing the number of physical machines in different clusters, adjusting VM resource configuration, and migrating VMs; Transition probability : Defined as performing an action in state s Then transfer to the next state s ’ Probability ; Considering the uncertainty of cloud environment, the transfer function is a random mapping; Reward Function : Comprehensively consider the performance indicators of the system and design a multi-objective reward function , weighing resource utilization, response time, energy consumption, and SLA factors; Discount Factor : Controls the importance of future rewards, The larger it is, the more the strategy focuses on long-term returns; The local MDP is defined as : State Space :state Contains fine-grained monitoring information within a cluster; Action Space :action This includes adjusting task priorities within the cluster, changing task resource quotas, and placing and migrating tasks; Reward Function : Partial Reward Focus on the performance and fairness factors of tasks within the cluster; , are the transition probability and discount factor respectively; Based on global MDP and local MDP, hierarchical deep reinforcement learning is used to learn resource orchestration strategies from bottom to top and end to end. The global scheduler uses the DQN algorithm, takes the system state s as input, and outputs a coarse-grained scheduling decision ; The local scheduler uses the DQN algorithm to further refine the task resource allocation and scheduling within the cluster based on the global decision; the two-level agents achieve strategy coordination through sub-goal decomposition; the DQN algorithm in the global scheduler uses the following Q value iterative update rules: ； in is the learning rate, is the state-action value function, evaluating the action taken in state s Long-term cumulative rewards; The DQN algorithm Q value update in the local scheduler introduces the global scheduling decision as an additional input: ； in is the learning rate, a global decision Affect the state transition and reward of the local MDP; by As the conditional input of the local Q network, the collaborative optimization of the hierarchical strategy is achieved.

6. The method for adaptive multi-granularity cloud resource orchestration based on hierarchical reinforcement learning according to claim 3 is characterized in that: Step S4 is specifically implemented as follows: Continuously collect multi-granular monitoring data, including resource usage, task queuing and running status, and energy consumption at the cluster / node / VM / task level; on this basis, calculate a series of macro and micro KPI indicators, including: Average resource utilization: ; Average task waiting time: ; Average task completion rate: ; in, The resource utilization of the kth cluster at time t, where T represents the number of time cycles and I is the task index. Indicates the start execution time of task I, Indicates the submission time of task I, represents the completion time of task I, is the deadline of Task I, Represents the total number of tasks, It is an indicator function. When the actual completion time of the task minus the submission time is greater than the deadline, the indicator function returns 1, otherwise it returns 0.

Images