Deep reinforcement learning cloud platform resource adjustment method based on prediction of user workload

Abstract

The invention relates to the technical field of cloud computing, and provides a deep reinforcement learning cloud platform resource adjustment method based on user workload prediction. The method aims at solving the problem of adjustment hysteresis caused by the fact that load changes cannot be predicted in a traditional response type resource adjustment strategy, so that the rule violation rate of a service level target (SLO) is effectively reduced, and the resource utilization rate is improved. According to the main scheme, the method comprises the following steps: firstly, predicting a future workload trend through a long short-term memory (LSTM) network; combining a prediction result with a current system state (such as response delay and copy number) to serve as input of a deep reinforcement learning agent; the intelligent agent outputs a resource adjustment action through a strategy network; and finally, optimizing the decision model according to a composite revenue function integrating the service quality, the resource cost and the adjustment stability. The method can be widely applied to cloud native application under a micro-service architecture, and automatic and prospective elastic resource scaling management is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing and provides a method for adjusting resources of a deep reinforcement learning cloud platform based on predicting user workload. Background Art

[0002] With the increasing popularity of cloud computing, autoscaling technology has become crucial for ensuring the quality of service (QoS) of cloud applications and saving resource costs. However, user-facing cloud applications often face dynamically changing or even drastically fluctuating workloads, which poses a significant challenge in maintaining stable service quality within pre-set service level objectives (SLOs).

[0003] Traditional resource adjustment methods, such as KubeMeteor's native Horizontal Pod Autoscaler (HPA), mostly employ threshold-based reactive strategies. These methods trigger resource adjustments only when monitored metrics such as CPU or memory exceed thresholds. Because container startup and application initialization require time, this passive response mechanism inherently suffers from lag. When load suddenly increases, the system cannot immediately scale up, resulting in increased latency in user request responses and reduced service quality. When load decreases, allocated resources cannot be scaled back in time, resulting in wasted resources. Therefore, overcoming adjustment lag and achieving proactive and forward-looking resource management is a pressing technical challenge in cloud resource scheduling. Summary of the Invention

[0004] The purpose of this invention is to solve the problem of response lag caused by the lack of forward-looking load prediction capabilities in traditional cloud platform resource adjustment methods, as well as the resulting high service level objective (SLO) violation rate and low resource utilization. By integrating time series feature embedding, multi-step load prediction and deep reinforcement learning decision models, proactive resource dynamic optimization based on future workload trends is achieved.

[0005] To achieve this objective, the present invention provides a method for adjusting resources on a deep reinforcement learning cloud platform based on predicted user workload, comprising the following steps:

[0006] S1: Obtain historical workload data and current real-time operating status of cloud applications. The historical workload data is used for prediction, and the real-time operating status is used for decision-making.

[0007] S2: Based on the historical workload data, a workload trend for one or more future time steps is predicted using a load prediction model to obtain a predicted workload value.

[0008] S3: Combine the predicted workload value obtained by S2 with the current real-time operating status obtained by S1 (including request response delay, number of container replicas, etc.) to form a composite state vector, which is used as the input of the resource adjustment decision model.

[0009] S4: Outputting a resource adjustment action through the resource adjustment decision model, where the action is an instruction to increase, decrease, or keep unchanged the number of container instantiation replicas for one or more microservices.

[0010] S5: After executing the resource adjustment action output by S4 in the cloud platform, a compound benefit value is calculated based on the system's service quality, resource cost, and the adjustment action itself, and the network parameters of the resource adjustment decision model are updated based on the benefit value to continuously optimize the decision strategy.

[0011] In the above method, in S1, the detailed description of resource status acquisition is as follows:

[0012] S11: Deploy Prometheus and Istio as resource monitoring tools to pull underlying resource allocation and usage metrics from containers and nodes. The resource monitoring tools record the current cloud application status every minute as follows:

[0013] X t ={Timestap,CPU,Memory,latency,request}

[0014] Timestap is the time in the format of {HH:MM}, CPU is the CPU utilization of the current Deployment, Memory is the memory utilization of the current Deployment, Latency is the average response time of user requests within one minute, and Request is the number of user requests within one minute.

[0015] S12: Different applications have different resource requirements. To mask the differences in minimum resource requirements between different deployments, normalize the CPU and memory values.

[0016]

[0017] In the above method, in S2, the steps of historical time series data feature processing are as follows:

[0018] S21: Temporal Feature Embedding. To enable the neural network model to understand and utilize the temporal information in the raw data, we first process the timestamps in the historical workload data acquired in S1 using the Time2Vec embedding technique. This technique converts the timestamps into a high-dimensional vector that captures both the linear and periodic patterns implicit in the time series data.

[0019]

[0020] Where t2v(TS)[i] is the i-th component of the output vector, ω i and φ i To learn parameters, the model continuously optimizes the parameters to continuously adapt to the specific periodic patterns in the time series data.

[0021] S22: Construct model input features. The embedding vector generated by S21, which already contains the time pattern, is concatenated with other original indicators in the historical workload data to form a more information-rich and unified feature vector E t = {t2v(TS), CPU, Memory, Latency, Request}. This vector serves as the final input to the prediction model. By capturing information from time series features through t2V and combining user workload with time series features, the model can better understand the relationship between user workload and time series data.

[0022] S23: Perform multi-step sequence prediction. The input feature vector E obtained in S22 is fed into a pre-trained prediction model composed of a multi-layer long short-term memory network (LSTM). Leveraging LSTM's powerful ability to capture time series dependencies, the model outputs independent load prediction results for multiple consecutive time steps in the future.

[0023]

[0024] Among them H (i) =LSTM (i) (H (i-1) ) is the hidden state of the previous LSTM layer, which is the feature information passed to the next LSTM layer. x is the step size for multi-step forecasts.

[0025] In order to solve the problem of excessive valuation deviation in single-step prediction and obtain a more stable and reliable judgment of future load trends, L x Independent prediction results for all future time steps generated by the LSTM model Perform arithmetic mean calculation. The final average value is the smoothed predicted workload value. It will be used in subsequent S3 resource adjustment decisions.

[0026] The resource adjustment decision process in S3 in the above method is described in detail as follows:

[0027] S31: Construct a composite state vector. At the beginning of each decision cycle, data from different information sources are integrated to construct a composite state vector S that can fully reflect the current and future trends of the system. x The vector includes at least three core parts: S311: Future load part: The predicted workload value generated by step S2, representing the future workload trend

[0028] S312: Current service quality part: the real-time average request response delay Delay obtained in step S1, representing the current service performance.

[0029] S32: Current resource configuration part: the number of instantiated copies of each microservice container Pod_num obtained in step S1, which represents the current resource investment.

[0030] S32: Generate adjustment actions through the policy network. The composite state vector St constructed in S31 is used as input and fed into a pre-trained policy network (Actor) in the resource adjustment decision model based on the soft actor-critic (SAC) framework. The policy network outputs an adjustment action A for different resources based on the input state. t ={a1, a2, ..., a n} and finally select a specific a from this distribution by random sampling i The action (by increasing or decreasing the Pod replica instances of the Deployment) is used as the instruction to be executed in step S4.

[0031] In the above method, the cloud application resource replica adjustment method in S4 includes the following steps:

[0032] S41: First, the abstract action representing the adjustment strategy output by the decision model in step S3 is parsed. This action is usually a discrete value representing the change in the number of replicas (for example, +1, -1, 0, +2, etc.). The system adds this change to the actual number of running replicas of the current microservice obtained from S1 to calculate the target number of replicas K that needs to be achieved after this adjustment.

[0033] K=Pod_num,+max(max_num,min(min_num,a i ))

[0034] S42: Next, the platform issues an update command to the container orchestration platform (such as Kubemetes) by calling its application programming interface (API), passing the target number of replicas calculated in S41 as a parameter. This command explicitly specifies the Deployment resource corresponding to the target microservice to be adjusted and updates the replicas field (.spec.replicas) in its configuration to the new target value. Upon receiving this command, the platform's controller automatically executes subsequent Pod creation or destruction operations until the number of running replicas in the system matches the new target value, completing a complete autoscaling operation.

[0035] Modeling the reward return of resource adjustment actions, modeling the benefits of resource adjustment, and helping the model understand resource adjustment actions. Specifically, the resource adjustment action reward return and loss function calculation in S5 of the above method are described as follows:

[0036] S51: After executing the adjustment action of S4 and waiting for the system to transition to the new state, collect the performance indicators during this period and calculate a quantitative compound benefit value R according to the preset multi-objective benefit function. t The return value is composed of the weighted sum of three parts:

[0037] R t =w qos ·r qos +w res ·r res +w act ·r act

[0038] W qos +w res +w act =1

[0039] where r qos As a service quality penalty, when the real-time response delay Delany exceeds the service level target threshold Delany slo , this term is a negative value:

[0040] r qos =-max(0,dd slo )

[0041] r res This is a resource cost penalty, which is proportional to the adjusted number of replicas, Pod_num, and is a negative penalty:

[0042] r res =-|Pod_num|

[0043] r actTo adjust the action penalty, this item is related to the amplitude of the adjustment action |a i | is proportional to a negative penalty, whose size is r act =-|a i |, used to suppress overly frequent or drastic resource scaling.

[0044] S52: The complete interaction data in this decision cycle, including the state S before adjustment t , Action A t , the profit value R calculated in S51 i And the adjusted new state S i+1 , packaged together into an experience tuple t , A t , R t , S t+1 >. The tuple is then stored in a finite-sized memory buffer called the experience replay pool. During training, rather than using the newly generated experience directly, a mini-batch of experience tuples is randomly sampled from the pool for model updates. This aims to break the temporal correlation between data samples and improve the stability and efficiency of the training process.

[0045] S53: Using the small batch of experience tuples extracted in S52, the network parameters of the resource adjustment decision model (SAC model) are iteratively updated.

[0046] S531: First, use the target value network (TargetNetwork) to calculate the target Q value y used to guide the update t ,y i Denotes the target Q value, calculated according to the Bellman method. Then, the mean square error loss between the Q value output by the current two value networks and the target Q value is minimized:

[0047]

[0048] S532: Different from traditional reinforcement learning, which finds the optimal strategy by maximizing the reward return of the policy function, it balances the optimal action reward return and the action exploration degree, so that it explores different strategies to avoid the model falling into the local optimum. Its policy function π * Defined as:

[0049]

[0050] Based on the judgment given by the value network, by minimizing a specific loss function:

[0051]

[0052] where H(π(·|s​t )) represents the strategy π in state s t This process guides the policy network to prefer actions that can obtain higher Q values, while ensuring sufficient exploration by maximizing the policy entropy.

[0053] Because the present invention adopts the above technical means, it has the following beneficial effects:

[0054] 1. Proactive Load Forecasting: This mechanism uses long short-term memory (LSTM) networks and Time2Vec embedding technology to deeply learn and analyze the temporal patterns of historical workload data, accurately predicting future load trends. This approach transforms traditional "passive response" scaling into "proactive pre-adjustment," fundamentally resolving the resource adjustment lag caused by time-consuming container startup.

[0055] 2. Forward-Looking Intelligent Decision-Making Model: The deep reinforcement learning decision-making model (SAC-PARE) constructed by this invention not only takes into account the current system state (such as latency and number of replicas) but also innovatively incorporates predicted future load values. This gives the decision-making agent foresight, enabling it to make optimal resource adjustment decisions in advance based on upcoming load peaks or troughs, thus avoiding the performance fluctuations of traditional methods when faced with sudden load fluctuations.

[0056] 3. Multi-Objective Composite Reward Function: We designed a composite reward function that comprehensively considers quality of service (QoS), resource costs, and the stability of adjustment actions. By assigning differently weighted penalties to SLO violations, resource usage, and frequent adjustments, this function guides the model to learn a balanced strategy. This avoids sacrificing user experience or system stability for a single objective, ensuring that adjustments more closely align with actual production needs.

[0057] 4. Dual Improvements in Performance and Efficiency: Combining these innovations, this invention achieves significant results in both ensuring service quality and improving resource efficiency. Experiments have shown that compared to the traditional threshold-based resource adjustment method (HPA), this invention can reduce service level objective (SLO) violations by an average of 17% while simultaneously improving resource utilization by 10%. This demonstrates that this invention not only provides a more stable and reliable user experience, but also effectively reduces cloud resource rental costs for enterprises.

[0058] In summary, the present invention significantly improves the intelligence level and efficiency of cloud platform resource adjustment through an innovative active load prediction method, a forward-looking reinforcement learning decision model, and a multi-objective composite benefit function design, effectively overcoming the inherent defects of traditional elastic scaling methods. It has important practical value for building a stable, efficient, and economical modern cloud-native application system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 : Overall flow chart of the resource dynamic adjustment system proposed in the present invention.

[0060] Figure 2 : Detailed structure diagram of the load prediction module, showing the process from timing input to predicted value output.

[0061] Figure 3 : The structural diagram of the resource adjustment decision module, showing the actor-critic framework based on SAC.

[0062] Figure 4 : A graph showing the request rate changes within 24 hours for the Wikipedia access tracking dataset used in the experiment.

[0063] Figure 5 : Comparison chart of average response time fluctuations of the method of the present invention and the comparative method during the experiment.

[0064] Figure 6 : Box plots of response delay distribution of three methods.

[0065] Figure 7 : Comparison of CPU and memory utilization changes of the three methods during the experiment.

[0066] Figure 8 : Box plots of CPU and memory utilization distribution of the three methods. DETAILED DESCRIPTION

[0067] The following is a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with certain specific embodiments, it should be noted that the present invention is not limited to these embodiments. On the contrary, modifications or equivalent substitutions of the present invention are intended to fall within the scope of the claims of the present invention.

[0068] In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention can also be implemented without these specific details.

[0069] The overall system architecture of the present invention is as follows Figure 1 As shown, it follows a cycle of monitoring, analysis, and execution. The monitoring module collects cloud application operating indicators through tools such as Prometheus and Istio; the execution module adjusts resources by calling the control interface of cloud platforms such as Kubemetes; the core of the present invention lies in the analysis link, namely the SAC-PARE model. This invention proposes a deep reinforcement learning cloud platform resource adjustment method based on predicting user workloads, including:

[0070] 1. Modeling the resource adjustment problem

[0071] We model the resource adjustment process as a Markov decision process (MDP), which consists of three core elements: state, action, and payoff:

[0072] State S.: In the adjustment period t, the environment state perceived by the agent consists of three parts: the predicted future workload y t , the current user request response delay Delany, and the number of container instances of each microservice Pod_num.

[0073] Action A t : The actions that the agent can perform are to adjust the number of container instantiation copies of each microservice, such as adding one copy, reducing one copy, or keeping it unchanged.

[0074] Reward R t :When executing action A t After that, the system will calculate a compound benefit value based on the performance of the application. This benefit takes into account three aspects: service quality, resource cost, and adjustment frequency.

[0075] 2. SAC-PARE model design

[0076] This model mainly consists of two parts: load prediction module and resource adjustment decision module.

[0077] 2.1 Load Forecasting Module

[0078] like Figure 2 As shown, this module is designed to predict future load based on historical data.

[0079] Input processing: Input is the most recent L x The historical workload sequence is a time-step sequence. Each time-step data contains the actual workload (e.g., CPU and memory usage) and a timestamp (e.g., day or hour). To enable the neural network to understand time information, we use the Time2Vec embedding layer to process the timestamps and convert them into vectors that can express linear and periodic patterns.

[0080] Prediction Network: The processed feature vector is fed into a multi-layer LSTM network. LSTM is particularly well-suited for processing time series data due to its gating mechanism. The model performs multi-step predictions and averages multiple predictions to obtain a smoother and more accurate representation of the future workload y. t .

[0081] 2.2 Resource Adjustment Decision Module

[0082] like Figure 3 As shown in Figure 2, this module is responsible for formulating specific scaling strategies.

[0083] Model Framework: This framework uses the Soft Actor-Critic (SAC) reinforcement learning framework. SAC is one of the most advanced off-policy algorithms, known for its high sample efficiency and stability.

[0084] Network structure:

[0085] Policy network (actor): input is the current state S t , output the probability distribution of the adjustment action. The final resource adjustment action is generated by random sampling.

[0086] Value Network (Critic): Double Q-Networks is used to evaluate the state S t Next, perform action A t The overestimation problem is alleviated by taking the smaller value of the two Q network outputs.

[0087] Training process: The goal of training is to maximize a cumulative reward including entropy, the formula is:

[0088]

[0089] 3. Experimental Verification

[0090] We conducted experiments by deploying the open-source microservice benchmark application Sock-Shop on a Kubernetes cluster consisting of three virtual machines. We used a real Wikipedia access trace dataset to simulate user request load.

[0091] Performance comparison: Figure 5 and Figure 6 As shown, the SAC-PARE method of the present invention performs best in response delay, with not only the lowest median but also the narrowest fluctuation range, proving its superiority in ensuring QoS stability.

[0092] Resource efficiency: e.g. Figure 7 and Figure 8 As shown in the figure, SAC-PARE achieves higher resource utilization (5%-10% higher on average) than the comparison methods HPA and standard SAC while ensuring QoS.

[0093] Example 1

[0094] To achieve this objective, the present invention provides a method for adjusting resources on a deep reinforcement learning cloud platform based on predicted user workload, comprising the following steps:

[0095] S1: Obtain historical workload data and current real-time operating status of cloud applications. The historical workload data is used for prediction, and the real-time operating status is used for decision-making.

[0096] S2: Based on the historical workload data, a workload trend for one or more future time steps is predicted using a load prediction model to obtain a predicted workload value.

[0097] S3: Combine the predicted workload value obtained by S2 with the current real-time operating status obtained by S1 (including request response delay, number of container replicas, etc.) to form a composite state vector, which is used as the input of the resource adjustment decision model.

[0098] S4: Outputting a resource adjustment action through the resource adjustment decision model, where the action is an instruction to increase, decrease, or keep unchanged the number of container instantiation replicas for one or more microservices.

[0099] S5: After executing the resource adjustment action output by S4 in the cloud platform, a compound benefit value is calculated based on the system's service quality, resource cost, and the adjustment action itself, and the network parameters of the resource adjustment decision model are updated based on the benefit value to continuously optimize the decision strategy.

[0100] In the above method, in S1, the detailed description of resource status acquisition is as follows:

[0101] S11: Deploy Prometheus and Istio as resource monitoring tools to pull underlying resource allocation and usage metrics from containers and nodes. The resource monitoring tools record the current cloud application status every minute as follows:

[0102] X t ={Timestap,CPU,Memory,latency,request}

[0103] Timestap is the time in the format of {HH:MM}, CPU is the CPU utilization of the current Deployment, Memory is the memory utilization of the current Deployment, Latency is the average response time of user requests within one minute, and Request is the number of user requests within one minute.

[0104] S12: Different applications have different resource requirements. To mask the differences in minimum resource requirements between different deployments, normalize the CPU and memory values.

[0105]

[0106] In the above method, in S2, the steps of historical time series data feature processing are as follows:

[0107] S21: Temporal Feature Embedding. To enable the neural network model to understand and utilize the temporal information in the raw data, we first process the timestamps in the historical workload data acquired in S1 using the Time2Vec embedding technique. This technique converts the timestamps into a high-dimensional vector that captures both the linear and periodic patterns implicit in the time series data.

[0108]

[0109] Where t2v(TS)[i] is the i-th component of the output vector, ω i and φ i To learn parameters, the model continuously optimizes the parameters to continuously adapt to the specific periodic patterns in the time series data.

[0110] S22: Construct model input features. The embedding vector generated by S21, which already contains the time pattern, is concatenated with other original indicators in the historical workload data to form a more information-rich and unified feature vector E t ={t2v(TS), CPU, Memory, latency, request}. This vector will serve as the final input of the prediction model.

[0111] S23: Perform multi-step sequence prediction. The input feature vector E obtained in S22 is fed into a pre-trained prediction model composed of a multi-layer long short-term memory network (LSTM). Leveraging LSTM's powerful ability to capture time series dependencies, the model outputs independent load prediction results for multiple consecutive time steps in the future.

[0112]

[0113] Among them H (i) =LSTM (i) (H (i-1) ) is the hidden state of the previous LSTM layer, which is the feature information passed to the next LSTM layer. x is the step size for multi-step forecasts.

[0114] In order to solve the problem of excessive valuation deviation in single-step prediction and obtain a more stable and reliable judgment of future load trends, L x Independent prediction results for all future time steps generated by the LSTM model Perform arithmetic mean calculation. The final average value is the smoothed predicted workload value. It will be used in subsequent S3 resource adjustment decisions.

[0115] The resource adjustment decision process in S3 in the above method is described in detail as follows:

[0116] S31: Construct a composite state vector. At the beginning of each decision cycle, data from different information sources are integrated to construct a composite state vector S that can fully reflect the current and future trends of the system. x The vector consists of at least three core parts:

[0117] S311: Future load part: the predicted workload value generated by step S2, representing the future workload trend

[0118] S312: Current service quality part: the real-time average request response delay Delay obtained in step S1, representing the current service performance.

[0119] S32: Current resource configuration part: the number of instantiated copies of each microservice container Pod_num obtained in step S1, which represents the current resource investment.

[0120] S32: Generate adjustment actions through the policy network. The composite state vector St constructed in S31 is used as input and fed into a pre-trained policy network (Actor) in the resource adjustment decision model based on the soft actor-critic (SAC) framework. The policy network outputs an adjustment action A for different resources based on the input state. t ={a1, a2, ..., a n} and finally select a specific a from this distribution by random sampling i The action (by increasing or decreasing the Pod replica instances of the Deployment) is used as the instruction to be executed in step S4.

[0121] In the above method, the cloud application resource replica adjustment method in S4 includes the following steps:

[0122] S41: First, the abstract action representing the adjustment strategy output by the decision model in step S3 is parsed. This action is usually a discrete value representing the change in the number of replicas (for example, +1, -1, 0, +2, etc.). The system adds this change to the actual number of running replicas of the current microservice obtained from S1 to calculate the target number of replicas K that needs to be achieved after this adjustment.

[0123] K=Pod_num,+max(max_num,min(min_num,a i ))

[0124] S42: Next, the platform issues an update command to the container orchestration platform (such as Kubemetes) by calling its application programming interface (API), passing the target number of replicas calculated in S41 as a parameter. This command explicitly specifies the Deployment resource corresponding to the target microservice to be adjusted and updates the replicas field (.spec.replicas) in its configuration to the new target value. Upon receiving this command, the platform's controller automatically executes subsequent Pod creation or destruction operations until the number of running replicas in the system matches the new target value, completing a complete autoscaling operation.

[0125] In the above method, the reward return and loss function calculation of the resource adjustment action in S5 are described as follows:

[0126] S51: After executing the adjustment action of S4 and waiting for the system to transition to the new state, collect the performance indicators during this period and calculate a quantitative compound benefit value R according to the preset multi-objective benefit function. t The return value is composed of the weighted sum of three parts:

[0127] R t =w qos ·r qos +w res ·r res +w act ·r act

[0128] w qos +w res +w act =1

[0129] where r qos As a service quality penalty, when the real-time response delay Delany exceeds the service level target threshold Delany slo , this term is a negative value:

[0130] r qos =-max(0,dd slo )

[0131] r res This is a resource cost penalty, which is proportional to the adjusted number of replicas, Pod_num, and is a negative penalty:

[0132] r res =-|Pod_num|

[0133] r act To adjust the action penalty, this item is related to the amplitude of the adjustment action |a i | is proportional to a negative penalty, whose size is ract =-|a i |, used to suppress overly frequent or drastic resource scaling.

[0134] S52: The complete interaction data in this decision cycle, including the state S before adjustment t , Action A t , the profit value R calculated in S51 t And the adjusted new state S t+1 , packaged together into an experience tuple t , A t , R t , S i+1 >,. The tuple is then stored in a finite-sized memory buffer called the experience replay pool. During training, rather than using the newly generated experience directly, a mini-batch of experience tuples is randomly sampled from the pool for model updates. This aims to break the temporal correlation between data samples and improve the stability and efficiency of the training process.

[0135] S53: Using the small batch of experience tuples extracted in S52, the network parameters of the resource adjustment decision model (SAC model) are iteratively updated.

[0136] S531: First, use the target value network (TargetNetwork) to calculate the target Q value y used to guide the update t ,y i Denotes the target Q value, calculated according to the Bellman method. Then, the mean square error loss between the Q value output by the current two value networks and the target Q value is minimized:

[0137]

[0138] S532: Different from traditional reinforcement learning, which finds the optimal strategy by maximizing the reward return of the policy function, it balances the optimal action reward return and the action exploration degree, so that it explores different strategies to avoid the model falling into the local optimum. Its policy function π * Defined as:

[0139]

[0140] Based on the judgment given by the value network, by minimizing a specific loss function:

[0141]

[0142] where H(π(·|s t )) represents the strategy π in state s t ​This process guides the policy network to prefer actions that can obtain higher Q values, while ensuring sufficient exploration by maximizing the policy entropy.

[0143] This paper introduces a multi-step load prediction mechanism based on LSTM and Time2Vec to solve the hysteresis problem caused by the lack of foresight in traditional resource adjustment strategies, thereby effectively reducing the service level objective (SLO) violation rate and improving resource utilization:

[0144] The present invention constructs an active load prediction system through steps S1-S2, and its technical means include:

[0145] Time series feature embedding (S21): Use Time2Vec technology to convert timestamps into high-dimensional vectors, and use the learnable parameter ω i and φ i Capturing periodic and linear patterns enables the model to identify regular trends in workload changes.

[0146] Multi-step forecast fusion (S23): This method uses an LSTM network to perform time series modeling on historical data and combines the arithmetic average of multi-step forecast results to eliminate single-step forecast bias and generate more stable estimates of future load trends.

[0147] This technical approach addresses the limitations of traditional responsive strategies (such as HPA) that rely on real-time indicator triggers. By predicting load change trends in advance, resource adjustment preparations are completed before container instantiation. For example, when it is predicted that the load will surge in a certain period of time in the future, the system can expand capacity in advance to avoid exceeding the response delay; when it is predicted that the load will drop, it can actively reduce capacity to reduce idle resources. This "pre-adjustment" mode significantly reduces the risk of SLO violations caused by container startup delays, and at the same time avoids the problem of over-allocation or under-allocation of resources through accurate prediction.

[0148] This paper constructs a composite state vector that integrates future load forecasts and current status, and combines it with a deep reinforcement learning decision model in the SAC framework to solve the problem of traditional methods that are difficult to balance service quality and resource costs in dynamic load scenarios, thereby achieving multi-objective optimization:

[0149] The present invention designs a forward-looking intelligent decision-making system through steps S3-S5, and its technical means include:

[0150] Composite State Vector Construction (S31): This integrates the future load forecast (S311), current service quality indicators (S312), and resource allocation status (S32) into a unified input, enabling the decision model to consider both short-term fluctuations and long-term trends. For example, when a future load peak is predicted but current latency is within the specified range, the model can adjust resources in advance to avoid subsequent performance degradation. When a load trough is predicted and the current number of replicas is redundant, the model can proactively scale down resources to reduce resource costs.

[0151] Multi-objective compound benefit function design (S51): By weighting and summing service quality penalties, resource cost penalties, and adjustment action penalties, the model is guided to optimize resource utilization while ensuring the SLO. For example, when an adjustment action causes excessive fluctuations in the number of replicas, the penalty term will inhibit frequent scaling; when resource costs are too high, the model will prioritize more economical adjustment strategies.

[0152] This technical approach overcomes the limitations of traditional methods that focus solely on a single metric (such as CPU utilization). By dynamically balancing service quality, resource costs, and adjustment stability, it enables the decision-making model to achieve global optimization in complex scenarios. For example, during sudden traffic surges, the system prioritizes ensuring the SLO; during periods of stable load, refined adjustments are made to improve resource utilization. This multi-objective collaborative optimization mechanism significantly improves the adaptability and cost-effectiveness of the cloud platform in dynamic load scenarios.

[0153] In summary, the present invention successfully transforms resource adjustment from a passive response mode to an active pre-adjustment mode by introducing load prediction and an improved deep reinforcement learning decision model, effectively solving the hysteresis problem of resource allocation and achieving dual optimization of service quality and resource cost in a complex and changeable cloud environment.

Claims

1. A method for adjusting resources on a deep reinforcement learning cloud platform based on predicting user workload, characterized in that: The following steps are involved: S1: Obtain historical workload data and current real-time operating status of cloud applications. Historical workload data is used for prediction, and real-time operating status is used for decision-making. S2: Based on the historical workload data, using a load prediction model, predict the workload trend for one or more future time steps to obtain a predicted workload value; S3: Combine the predicted workload value obtained in S2 with the current real-time operating status obtained in S1 to form a composite state vector, which is used as the input of the resource adjustment decision model; S4: Outputting a resource adjustment action through the resource adjustment decision model, where the action is an instruction to increase, decrease, or keep unchanged the number of instantiated copies of one or more containers of one or more microservices; S5: After executing the resource adjustment action output by S4 in the cloud platform, a compound benefit value is calculated based on the system's service quality, resource cost, and the adjustment action itself, and the network parameters of the resource adjustment decision model are updated based on the benefit value to continuously optimize the decision strategy.

2. A method for adjusting resources on a deep reinforcement learning cloud platform based on predicted user workload according to claim 1, characterized in that: Step S1 includes: S11: Deploy Prometheus and Istio as resource monitoring tools to pull underlying resource allocation and usage metrics from containers and nodes. The resource monitoring tools record the current cloud application status every minute as follows: X t ={Timestap,CPU,Memory,latency,request} Timestap is the time in the format of {HH:MM}, CPU is the CPU utilization of the current Deployment, Memory is the memory utilization of the current Deployment, Latency is the average response time of user requests within one minute, and Request is the number of user requests within one minute. S12: Different applications have different resource requirements. To mask the differences in minimum resource requirements between different deployments, normalize the CPU and memory values. Usage indicates the actual CPU or memory usage of the current Deployment; Request indicates the resource request amount of the Deployment.

3. The method for adjusting resources of a deep reinforcement learning cloud platform based on predicting user workload according to claim 1, characterized in that: Step S2 includes: S21: Temporal feature embedding: The time stamps in the historical workload data obtained from S1 are processed using the Time2Vec embedding technology to convert the timestamps into a high-dimensional vector to capture the linear and periodic patterns implicit in the time series data: Where t2v(TS)[i] is the i-th component of the output vector, ω i and φ i is a learnable parameter; S22: Construct model input features: The embedding vector generated by S21 is concatenated with other original indicators in the historical workload data to form the feature vector E t ={t2v(TS),CPU,Memory,latency,request}, vector E t It will serve as the final input to the prediction model; S23: Perform multi-step sequence prediction: The input feature vector E obtained by S22 t The data is fed into a pre-trained multi-layer LSTM prediction model to output independent load prediction results for multiple consecutive time steps in the future. Among them H (i) =LSTM (i) (H (i-1) ) is the hidden state of the previous LSTM layer, which is the feature information passed to the next LSTM layer. x is the step size for multi-step forecasts.

4. The method for adjusting resources of a deep reinforcement learning cloud platform based on predicting user workload according to claim 1, characterized in that: Step S3 includes: S31: Construct composite state vector: At the beginning of each decision cycle, data from different information sources are integrated to construct a composite state vector S that can fully reflect the current and future trends of the system. x , vector S x It includes at least three core parts: S311: Future load part: The predicted workload value representing the future workload trend generated by step S2 S312: Current service quality part: the real-time average request response delay Delay obtained in step S1, which represents the current service performance; S32: Current resource configuration part: the number of instantiated copies of each microservice container Pod_num representing the current resource input obtained in step S1; S32: Generate adjustment actions through the policy network: The composite state vector S constructed in S31 x As input, it is fed into a pre-trained policy network in the resource adjustment decision model based on the soft actor-critic framework. The policy network outputs an adjustment action A for different resources according to the input state. t ={a1, a2, ..., a n } and finally select a specific a from this distribution by random sampling i The action is used as the instruction to be executed in step S4.

5. The method for adjusting resources of a deep reinforcement learning cloud platform based on predicting user workload according to claim 1, characterized in that: Step S4 includes: S41: The abstract action representing the adjustment strategy output by the decision model in step S3 is parsed. This action represents the discrete value of the change in the number of replicas. The system adds this change to the actual number of running replicas of the current microservice obtained from S1 to calculate the target number of replicas K to be achieved after this adjustment. The formula is: K=Pod_num+max(max_num,min(min_num,a i )) Among them, Pod_num is the number of instantiated copies of the current microservice container, a i is the action value, S42: By calling the application programming interface (API) of the container orchestration platform, the target number of replicas calculated by S41 is used as a parameter, and an update instruction is issued to the platform. The Deployment resources corresponding to the target microservices that need to be adjusted are clearly specified, and the replica number field .spec.replicas in its configuration is updated to the new target value K. After the platform receives this instruction, its own controller Controller will automatically execute subsequent Pod creation or destruction operations until the number of replicas running in the system is consistent with the new target value, thus completing a complete elastic scaling operation.

6. The method for adjusting resources of a deep reinforcement learning cloud platform based on predicting user workload according to claim 1, characterized in that: Step S5 includes: S51: After executing the adjustment action of S4 and waiting for the system to transition to the new state, collect the performance indicators during this period and calculate a quantitative compound benefit value R according to the preset multi-objective benefit function. t , the return value is composed of the weighted sum of three parts: R t =w qos ·r qos +w res ·r res +w act ·r act In qos +in res +in act =1 where r qos As a service quality penalty, when the real-time response delay Delany exceeds the service level target threshold Delany slo , this term is a negative value: r qos =-max(0,d-d slo ) Where d represents the real-time response delay, that is, the average response time of the current user request of the system, d slo Indicates the service level objective threshold, that is, the preset upper limit of the maximum allowed response delay. max indicates the maximum value. r res This is a resource cost penalty, which is proportional to the adjusted number of replicas, Pod_num, and is a negative penalty: r res =-|Pod_num| r act To adjust the action penalty, this item is related to the amplitude of the adjustment action |a i | is proportional to a negative penalty, whose size is r act =-|a i |, used to suppress overly frequent or drastic resource scaling; S52: The complete interaction data in this decision cycle, including the state S before adjustment t , Action A t , the profit value R calculated in S51 t And the adjusted new state S t+1 , packaged together into an experience tuple t , A t , R t , S t+1 >, and store it in the experience replay pool;​ S53: Use the small batch data extracted from the experience replay pool to iteratively update the network parameters of the resource adjustment decision model, including: S531: Use the target value network to calculate the target Q value y used to guide the update t , by minimizing the mean square error loss between the Q value output by the current two value networks and the target Q value: In state s i and action a i Next, the parameter is ω j The Q value estimated by the value network, y i represents the target Q value, calculated according to the Bellman method; S532: Balance the best action reward and action exploration, so that it can explore different strategies and avoid the model falling into local optimality. Its strategy function π * Defined as: Based on the judgment given by the value network, by minimizing a specific loss function: where H(π(·|s t )) represents the strategy π in state s t This process guides the policy network to prefer actions that can obtain higher Q values, while ensuring sufficient exploration by maximizing the policy entropy.

Images