Micro-service load combination prediction method based on depth deterministic strategy gradient

Abstract

The invention relates to the technical field of intelligent operation and maintenance resource management, and provides a micro-service load combination prediction method based on a depth deterministic strategy gradient. The method comprises the steps that firstly, historical load data are normalized, a pre-training model pool composed of various heterogeneous basic models is input, and each basic model independently generates a load prediction value and model loss; then, the normalized historical load data and the model loss at the previous moment are spliced into a state vector, and the state vector is input into a DDPG pre-trained model combination decision module; and the model combination decision module outputs a model combination score according to the state, and generates a model combination weight through Softmax processing. And carrying out weighted fusion on the model combination weight and the load prediction value of each model to generate a final load prediction result. According to the method, feature capture comprehensiveness is guaranteed through diversity of the model pool, dynamic weight optimization based on real-time performance feedback is realized by utilizing reinforcement learning, the limitation that a single model insufficiently captures multiple features of a micro-service load is effectively overcome, and prediction accuracy is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance resource management technology, and more specifically, to a microservice load combination prediction method based on deep deterministic policy gradient. Background Technology

[0002] Cloud-native architecture is a new paradigm for software development and operation in the cloud computing era. As one of the core technologies in this field, microservice architecture decouples software systems into functionally independent service modules, offering high flexibility, scalability, and maintainability, thus gaining popularity in an increasing number of enterprises. Each microservice deploys several replicas to share request load, known as microservice instances. Elastic scaling is a key characteristic of microservice architecture, allowing the number of microservice instances to automatically adjust based on load changes in the microservice system to meet business needs and maintain stable operation. Load forecasting is the foundation of microservice elastic scaling. By sensing future load changes, load forecasting guides the system to adjust resource scale in advance, thereby maintaining stable system operation.

[0003] However, the composition of microservice load is complex, exhibiting characteristics of randomness, trends, and multiple periods, which pose challenges to accurate prediction of microservice load. The randomness of microservice load refers to the potential impact of uncertain factors during its operation, such as retaliatory shopping behavior by users on e-commerce platforms following sudden news events. The trend of microservice load refers to the overall direction or trend of change over a long period; for example, the rapid growth of an e-commerce platform and its increasing market demand may lead to a long-term upward trend in its microservice load. The multiple periods of microservice load refer to the cyclical changes in microservice load across different time scales; for example, e-commerce platforms typically experience high loads on weekends or fixed promotional days each year due to increased order volume. Therefore, achieving accurate, multi-dimensional, and comprehensive prediction of microservice load to proactively meet the elastic scaling requirements of microservices has become an important issue.

[0004] In recent years, many time series forecasting models have been gradually applied to load forecasting tasks, but some problems still exist. For example, LSTM-based methods can capture some degree of load dependency, but they are not good at capturing the multi-periodicity of load; Transformer-based methods model series data at different time scales and have some ability to capture the multi-periodicity of load, but they suffer from memory decay, leading to the neglect of long-term load trends. These time series forecasting models use a single model and are trained for a specific characteristic of the load, lacking a comprehensive modeling of the complex characteristics of the load. Therefore, the accuracy of predictions still has some shortcomings. Summary of the Invention

[0005] This invention addresses the technical problem of insufficient prediction accuracy using a single model due to the complex composition of microservice loads. It provides a microservice load combination prediction method based on deep deterministic strategy gradients, which improves the accuracy of microservice load prediction by capturing the complex characteristics of microservice loads from multiple dimensions.

[0006] According to a first aspect of the present invention, a microservice load composition prediction method based on deep deterministic policy gradient is provided, comprising: S1, after normalizing the historical load data, input it into the pre-trained load prediction model pool. The load prediction model pool includes multiple basic prediction models. Each basic prediction model independently generates load prediction values ​​and model loss based on the normalized historical load data. S2, combine the normalized historical load data and the model loss set of the previous time step to generate a state vector, and input the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool. S3, the weighted combination of the model combination weights and the load prediction values ​​of each basic prediction model are weighted to obtain the final load prediction result.

[0007] Based on the above technical solution, the present invention can also be improved as follows.

[0008] Optionally, before step S1, the method further includes the step of constructing and pre-training the load prediction model pool, including: Based on the multi-dimensional characteristics of microservice load, a load prediction model pool is constructed using multiple deep learning-based time series prediction models. These multiple deep learning-based time series prediction models include at least the following: LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), BiLSTM (Bidirectional Long Short-Term Memory), TCN (Temporal Convolutional Network), Transformer, and Informer; Each of the basic prediction models is trained using normalized historical load window data as input, and each of the basic prediction models independently outputs the load prediction value and model loss for the corresponding prediction load window.

[0009] Optionally, in step S1, the historical load data is normalized using the min-max normalization method, with the following formula:

[0010] in, The load data before normalization at time i. This represents the normalized load data at time i. This represents the minimum load on the dataset. This indicates the maximum load on the dataset.

[0011] Optionally, in step S1, the model set M in the load prediction model pool captures normalized historical load data. The time series dependencies are determined, and the model prediction set of length H is output at time t:

[0012] in, This refers to the load prediction values ​​of multiple basic models in the load prediction model pool at a prediction time step i.

[0013] Optionally, in step S2, the state vector is obtained by concatenating normalized historical load window data and the model loss of the load prediction model pool at the previous time step, and is expressed as:

[0014] in, This represents normalized historical load window data. This represents the model loss of the load prediction model pool at the previous time step. Represented as:

[0015] in, Indicates the previous moment. j The model loss of each model. j ∈[1, m ], m This represents the number of basic models in the load prediction model pool.

[0016] Optionally, in step S2, the step of inputting the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool includes: The state vector An Actor network is input to a model combination decision module pre-trained using a deep deterministic policy gradient algorithm, and the Actor network outputs actions in its continuous action space. The action Composed of the model combination score at each prediction time step:

[0017] in, Indicates the first i Model combination score for each prediction time step H For the forecast load window size; for each forecast time step i Model combination score Represented as:

[0018] in, R represents the model combination score of the j-th basic model in the load forecasting model pool at the i-th forecasting time step, where R represents the real number field. For the model combination score Perform Softmax processing and map it to the model combination weights. :

[0019]

[0020] in, This represents the combined model weights at prediction time step i; By combining the model combination weights at each prediction time step, the model combination weights output by the prediction load window of the model combination decision module are obtained: .

[0021] Optionally, step S3 includes: The final load prediction value of the basic prediction model with prediction step size i is obtained by multiplying the load prediction value of each basic prediction model in the load prediction model pool with its corresponding model combination weight. :

[0022] The final load forecast result is obtained by combining the final load forecast values ​​from all the basic forecast models. : .

[0023] Optionally, it also includes a process of pre-training the agent of the model combination decision module using a deep deterministic policy gradient algorithm, wherein the agent is based on an Actor-Critic network structure, including an Actor main network, a target Actor network, a Critic main network and a target Critic network, wherein the Actor main network is used to deterministically output actions based on the current state, and the Critic main network is used to evaluate the value of state-action pairs. The training process includes: (1). Collect experience quadruples through interaction between the agent and the environment. The data is then stored in the experience replay pool, and subsequently, small batches of experience data are randomly sampled from the pool for network updates; among which, Let be the state vector at time t. Let t be the action at time t. To enable the agent to perform actions at time step t The immediate reward obtained afterwards This is the state vector at time step t+1; (2). Update the Critic main network, including: Calculate the TD target:

[0024] in, Let i be the target of TD, i be the current time, and i+1 be the next time. This is the reward value at the current moment. As a discount factor, The value evaluation function for the target Critic network. For the state at the next moment, The target Actor network in the next time step state The output value, For the learnable parameters of the target Actor network, The learnable parameters of the target Critic network. This indicates that the target Actor network and the target Critic network are used to determine the state at the next time step. To assess its future long-term value; The difference between the TD target and the output value of the Critic main network is used as the loss function L:

[0025] in, These are the learnable parameters of the Critic main network. This is the output value of the Critic main network. n The number of samples in the current batch of experience data sampled from the experience replay pool; The Critic master network value function is optimized by minimizing the loss function L, and the weights of the Critic master network are updated accordingly. (3) Update the target Critic network, including: Based on a soft update mechanism, the parameters of the target Critic network are slowly copied from the master Critic network to update the parameters of the target Critic network, as follows:

[0026] in, For the update range, ; (4). Update the Actor main network, including: The parameters of the Actor's main network are updated using policy gradients, as follows:

[0027] in, Represents the policy gradient. J For a given strategy, n The number of samples in the current batch of experience data sampled from the experience replay pool. Indicates the gradient calculation point. This represents the gradient of the Critic main network's output value with respect to the input action 'a'. The action output by the Actor's main network. This represents the learnable parameters of the Actor main network. Indicates the action output by the Actor main network. Relative to its own parameters The gradient, calculated during state. Pick ; (5) Update the target Actor network, including: Based on the soft update mechanism, the parameters of the main Actor network are slowly copied to update the parameters of the target Actor network, which is represented as: .

[0028] Optionally, during the training process of the agent, random noise is introduced into the actions generated by the policy. Let's explore:

[0029] in, Let t be the action output by the Actor's main network. This is the result after adding random noise to the action output of the Actor main network at time t.

[0030] According to a second aspect of the present invention, a microservice load composition prediction system based on deep deterministic policy gradient is provided, comprising: The data preprocessing module is configured to receive historical load data and perform normalization processing to generate normalized historical load data. The load prediction model pool module, connected to the data preprocessing module, includes multiple parallel basic prediction models. Each basic prediction model is configured to receive the normalized historical load data and independently generate the corresponding load prediction value and model loss. The state construction module is configured to combine normalized historical load data from the data preprocessing module with the model loss set from the previous time step from the load prediction model pool module to generate a state vector. A model combination decision module, connected to the state construction module, uses an agent pre-trained with a deep deterministic policy gradient algorithm. This agent is configured to receive the state vector and output dynamically generated model combination weights. The load prediction synthesis module is connected to the load prediction model pool module and the model combination decision module, respectively, and is configured to perform a weighted combination of the model combination weights and the load prediction values ​​generated by each basic prediction model to output the final load prediction result.

[0031] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, wherein the processor is configured to implement the steps of the above-described microservice load combination prediction method based on deep deterministic policy gradient when executing a computer management program stored in the memory.

[0032] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having a computer management program stored thereon, which, when executed by a processor, implements the steps of the above-described microservice load combination prediction method based on deep deterministic policy gradient.

[0033] This invention provides a microservice load compositing prediction method, system, electronic device, and storage medium based on deep deterministic policy gradient. First, it utilizes a heterogeneous pool of load prediction models to generate preliminary multi-perspective predictions and model losses reflecting recent accuracy from normalized historical load data in parallel. Second, these model losses are concatenated with historical load data to form a state vector, which is then input into a model compositing decision module pre-trained based on the DDPG algorithm. This model compositing decision module enables the agent to learn a deterministic policy, dynamically outputting optimal model compositing weights based on the concatenated current state vector. Finally, through weighted fusion, the predictive capabilities of each basic model are complemented, overcoming the inherent limitation of a single model in comprehensively capturing the multi-dimensional and complex characteristics of microservice load. Through a performance-oriented adaptive weighting mechanism, the accuracy and robustness of the final load prediction results are improved. Attached Figure Description

[0034] Figure 1 A flowchart of a microservice load combination prediction method based on deep deterministic policy gradient provided by the present invention; Figure 2 This is a schematic diagram of the load prediction model pool training process in one embodiment. Figure 3 This is a schematic diagram of the model combination decision module structure in one embodiment; Figure 4 This is a schematic diagram illustrating the training of the model combination decision module based on the deep deterministic policy gradient algorithm in one embodiment. Figure 5 A block diagram of a microservice load combination prediction system based on deep deterministic policy gradient provided by the present invention; Figure 6 A schematic diagram of a possible hardware structure of an electronic device provided by the present invention; Figure 7 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0035] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0036] Figure 1 A flowchart of a microservice load combination prediction method based on deep deterministic policy gradient provided by this invention is shown below. Figure 1 As shown, the method includes steps S1 to S3: S1, after normalizing the historical load data, input it into the pre-trained load prediction model pool. The load prediction model pool includes multiple basic prediction models. Each basic prediction model independently generates a load prediction value and model loss based on the normalized historical load data.

[0037] This step normalizes the input raw historical load data to eliminate dimensional differences and improve data stability. Then, leveraging model diversity, the processed normalized historical load data is input in parallel into a pre-trained model pool composed of various heterogeneous base models such as LSTM, GRU, and Transformer. Each base model runs independently, learning in parallel the complex temporal patterns (such as short-term dependence, long-term trends, and periodicity) of the normalized historical load data from different dimensions. Multiple base models generate their respective load prediction values ​​for future load prediction windows, and the model loss, reflecting their recent prediction accuracy, is calculated. The load prediction values ​​serve as candidates for combination, while the model loss serves as a real-time, quantitative performance metric, laying the data foundation for intelligent weight allocation in subsequent steps.

[0038] S2, combine the normalized historical load data and the model loss set of the previous time step to generate a state vector, and input the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool.

[0039] This step concatenates the normalized historical load data from step S1 with the model loss sets of each base model from the previous time step into a comprehensive state vector. This vector simultaneously contains the environmental state (load data) and the agent's historical performance (model loss), enabling it to perceive environmental changes and model performance. Then, this state vector is input into a model combination decision module pre-trained using the Deep Deterministic Policy Gradient (DDPG) algorithm. The agent in this module dynamically outputs a set of model combination scores based on the current state. After processing by the Softmax layer, the scores are converted into normalized combination weights of each base model at each time step within the prediction window.

[0040] S3, the weighted combination of the model combination weights and the load prediction values ​​of each basic prediction model are weighted to obtain the final load prediction result.

[0041] This step delivers the final prediction output by weighted summing of the prediction matrices from the load prediction model pool generated in step S1 and the dynamically generated model combination weights from step S2. Specifically, at each future time step within the prediction window, the predicted values ​​of each model are multiplied by their corresponding dynamic weights and then summed to integrate the strengths of each model, resulting in a final, more accurate sequence of load prediction results. At any given time, the model with the best recent performance (lower loss) is automatically assigned a higher weight, thereby improving the overall accuracy of the final prediction result and effectively overcoming the limitations of a single model when facing the complex and variable characteristics of microservice loads.

[0042] Understandably, given the shortcomings in the background technology, this invention proposes a microservice load composition prediction method based on deep deterministic policy gradients. This method comprises a load prediction model pool and a model combination decision module, forming an intelligent load predictor based on deep reinforcement learning (DRL), abbreviated as DDForcaster. By constructing a load prediction model pool composed of multiple heterogeneous base models, preliminary prediction results and model losses from multiple perspectives are generated. The model losses, along with historical load data, are then used as a state vector and input into the model combination decision module pre-trained based on the DDPG algorithm. This allows the system to dynamically and adaptively allocate optimal model combination weights based on real-time load characteristics and the recent prediction performance of each model. This effectively overcomes the limitations of a single model in capturing the complex characteristics of microservice load, such as randomness, trends, and multi-periodicity, thus improving the accuracy and robustness of the final load prediction results.

[0043] In one possible embodiment, before step S1, the method further includes a step of constructing and pre-training the load prediction model pool, as follows: Based on the multi-dimensional characteristics of microservice load, a load prediction model pool is constructed using multiple deep learning-based time series prediction models, represented as follows:

[0044] Among them, among them, Indicates the first j A basic model m The total number of basic models, for example Figure 2 The basic model shown ~ .

[0045] To increase the diversity of the base models, this embodiment selects several time series forecasting models of different types with significant differences in predictive capabilities as the base models for the load forecasting model pool. For example... Figure 2 As shown, DDForcaster constructs a load forecasting model pool that includes six basic models. The basic models of the load forecasting model pool are as follows: 1. LSTM (Long Short-Term Memory): An improved recurrent neural network architecture that introduces memory cells to store information and is equipped with gate controllers to control the flow of information.

[0046] 2. GRU (Gated Recurrent Unit): An improved recurrent neural network architecture consisting of two gate control units: a reset gate and an update gate.

[0047] 3. BiLSTM (Bidirectional Long Short-Term Memory): An improved recurrent neural network architecture consisting of a forward LSTM and a backward LSTM.

[0048] 4. TCN (Temporal Convolutional Network): A deep learning architecture that stacks multiple convolutional layers, uses residual connections, and uses convolutional kernels and dilation factors to control the network's receptive field.

[0049] 5. Transformer: A deep learning architecture that uses a self-attention mechanism and consists of two parts: an encoder and a decoder, both of which are stacked from multiple Transformer layers with the same structure.

[0050] 6. Informer: An improved Transformer architecture that combines self-attention, a global autoregressive model, and random sampling.

[0051] The base models in the load prediction model pool can be trained independently and simultaneously with the DDPG agents in the model combination decision module, generating load prediction values ​​for their respective prediction load windows. The training process is as follows: Figure 2 As shown. The input to the load forecasting model pool is historical load window data of length T:

[0052] in, Let R represent the historical load data at time i, where R represents the real number field.

[0053] However, microservice loads are random, and their values ​​fluctuate significantly over time. Therefore, directly using raw load data for prediction may affect the model's prediction accuracy. To mitigate the impact of load data fluctuations on model prediction accuracy, make the data more consistent with the model's assumptions, and improve model performance and convergence speed, this embodiment uses a min-max normalization method to normalize the historical load window data, obtaining normalized historical load data. :

[0054] in, The normalized load data at time i is represented as:

[0055] The load data before normalization at time i. This represents the normalized load data at time i. This represents the minimum load on the dataset. This indicates the maximum load on the dataset.

[0056] Then, the model set M in the load forecasting model pool captures normalized historical load data. Time series dependencies, such as Figure 2 As shown, the output is a model prediction set of length H at time t. :

[0057] in, For the load prediction model pool at prediction time step i m The load forecast value of the basic model can be expressed as:

[0058] in, Indicates the prediction time step i, the i-th j Load prediction values ​​for each basic model.

[0059] This embodiment uses This represents the actual load value within the predicted load window:

[0060] During the training of the load forecasting model pool, Mean Square Error (MSE) is used as the loss function to train each base model. MSE represents the loss function at time step t in the load forecasting model pool. j Model loss of the basic models:

[0061] The model's loss function is used to evaluate the difference between the model's predictions and the true values. A smaller loss function value indicates that the model's predictions are closer to the true values, and the model's performance is better; conversely, a larger loss function value indicates that the model's predictions differ significantly from the true values, and the model's performance is worse. Therefore, the model losses of each base model are passed to the model combination decision module to indicate the predictive ability of the base models, serving as part of the input state information of the DDForcaster agent.

[0062] In one possible implementation, DDForcaster models the model combination decision problem of the load forecasting model pool as an MDP (Markov Decision Process), and uses reinforcement learning to implement the model combination decision process. The structure of the model combination decision module is as follows: Figure 3 As shown.

[0063] The state space refers to the set of all possible states. In step S2, two vectors are concatenated to generate state information. That is, the DDPG agent obtains two types of data as states to complete the model combination weight decision: one part is the load data of the historical load window, which contains the input workload information used for prediction; the other part is the model loss set of the previous time step, that is, the model loss of each base model in the load prediction model pool at the previous time step, which is used to indicate the historical prediction performance of the base models.

[0064] Therefore, in step S2, the state vector at time t It is obtained by concatenating the normalized historical load window data and the model loss of the load prediction model pool at the previous time step, and is expressed as:

[0065] in, This represents normalized historical load window data. This represents the model loss of the load prediction model pool at the previous time step. Represented as:

[0066] in, Indicates the previous moment. j The model loss of each model. j ∈[1, m ], m To predict the number of base models in the model pool.

[0067] The action space is the set of all possible actions that the agent in the model's combinatorial decision-making module can take in each state. The state vector... The agent is fed into a model pre-trained using a deep deterministic policy gradient algorithm to combine decision modules. The agent outputs actions in its continuous action space. The motion space in this embodiment is designed as a continuous motion space. At time step t, the motion... This describes the model combination score of each basic model in the load forecasting model pool, used to calculate the final load forecast value for the forecasting load window. Action Composed of the model combination score at each prediction time step, expressed as:

[0068] in, Indicates the first i Model combination score for each prediction time step H For the predicted load window size. Each prediction time step. i Model combination score Represented as:

[0069] in, This indicates that at the i-th prediction time step, the i-th prediction model in the prediction model pool... j The model combination of the basic models is a numerical value, where R represents the real number field.

[0070] DDForcaster adds a Softmax layer after the Actor network in the model combination decision module to obtain the model combination weights. This maps the output model prediction scores to a probability distribution, ensuring that each element's value ranges from 0 to 1, and the sum of all elements is 1. At each prediction time step, the model combination scores are processed by Softmax to obtain the model combination weights. :

[0071]

[0072] in, This represents the model combination weights at prediction time step i.

[0073] After Softmax processing, the sum of the weights of all basic models in the load prediction model pool is 1, which is represented as:

[0074] Finally, by combining the model combination weights at each prediction time step, the model combination weights output by the prediction load window of the model combination decision module are obtained, expressed as: .

[0075] In one possible implementation, combined Figure 3 As shown, the model prediction set for the predicted load window in the load prediction model pool is represented as follows:

[0076] in, For the prediction model pool at prediction time step i m The load forecast values ​​of the basic models are expressed as follows:

[0077] In step S3, DDForcaster uses the model prediction set of the predicted load window and the model combination weights output by the model combination decision module to calculate the final output of the entire load forecasting method, i.e., the final load forecasting result. :

[0078] in, The final load prediction for step size i is obtained by multiplying the load prediction of each base model in the prediction model pool by its corresponding model combination weights: .

[0079] The reward function is used to evaluate the quality of an agent's actions during interaction with the environment. This embodiment uses... This represents the actual load value within the predicted load window:

[0080] in, This represents the actual load value for the predicted step size i.

[0081] DDForcaster outputs the predicted load value using the predicted load window. The reward function is designed based on the difference between the predicted load value and the actual load value. Compared with the actual load value When the difference between the predicted load value and the true load value is large, it indicates that the load prediction task has been performed poorly, and the agent is given a poor reward, thus penalizing the decision. Conversely, when the difference between the predicted load value and the true load value is small, it indicates that the load prediction task has been performed well, and the agent is given a high reward, thus rewarding the decision.

[0082] Therefore, DDForcaster's reward function is designed as follows:

[0083] Considering that the output actions of the model combinatorial decision-making module are continuous values, this embodiment of DDForcaster selects the deterministic policy gradient algorithm, which is used to solve the decision-making problem of continuous action space sequences, to train the model combinatorial decision-making agent. Its algorithm structure is as follows: Figure 4 As shown.

[0084] like Figure 4 As shown, the agent of the model combination decision module is based on the Actor-Critic network structure, including the Actor main network, the target Actor network, the Critic main network, and the target Critic network. The Actor main network is used to deterministically output actions based on the current state, and the Critic main network is used to evaluate the value of the state-action pair.

[0085] DDForcaster uses an experience replay mechanism, setting up an experience replay pool to store experience quadruples. Specifically, through the interaction between the intelligent agent and the environment, an experience quadruple is collected. This data is stored in the experience replay pool, and then, during training, small batches of experience data are randomly sampled from the experience replay pool for network updates. Let be the state vector at time t. Let t be the action at time t. To enable the agent to perform actions at time step t The immediate reward obtained afterwards Let be the state vector at time step t+1.

[0086] In DDForcaster, the Critic network is used to fit the value function. The Critic main network takes the current state as input. and current action Output value function The target Critic network has the same network structure as the Critic network, but different parameters. The target Critic network takes the next time-step state as input. and the next moment's action Output the value function at the next time step. Used to calculate TD target:

[0087] in, Let i be the target of TD, i be the current time, and i+1 be the next time. This is the reward value at the current moment. As a discount factor, The output value of the target Critic network. For the state at the next moment, The output value of the target Actor network. For the learnable parameters of the target Actor network, The learnable parameters of the target Critic network. This indicates that the target Actor network and the target Critic network are used to determine the state at the next time step. The future long-term value is assessed.

[0088] The Critic main network uses the TD error, which is the difference between the TD target and the Critic main network output, as a loss function for parameter updates.

[0089] in, These are the learnable parameters of the Critic main network. This is the output value of the Critic main network. n The number of samples in the current batch of experience data sampled from the experience replay pool.

[0090] During the update process, the loss function L is minimized to optimize the estimation of the Critic main network value function and update the weights of the Critic main network. And based on the "soft" update mechanism, the parameters of the target Critic network are slowly copied from the master Critic network to update the parameters of the target Critic network. , represented as:

[0091] in, For the update range, This means that the target value is updated slowly to improve the stability of learning.

[0092] DDForcaster's Actor network, acting as a policy network, deterministically maps states to specific actions to specify the current policy. The current state of the Actor's main network input. Output the current action Update its parameters using policy gradients:

[0093] in, Represents the policy gradient. J For a given strategy, n The number of samples in the current batch of experience data sampled from the experience replay pool. Indicates the gradient calculation point. This indicates that the output value of the Critic main network is relative to the input action. gradient, The action output by the Actor's main network. This represents the learnable parameters of the Actor main network. Indicates the action output by the Actor main network. Relative to its own parameters The gradient, calculated during state. Pick .

[0094] The target Actor network has the same network structure as the main Actor network, but different parameters; its input is the next time-step state. Output the action at the next moment. Similar to the parameter update steps of the target Critic network, a "soft" update mechanism is adopted, which updates the target Actor network parameters by slowly copying the parameters of the Actor master network, as shown below: .

[0095] In DDForcaster, because the policy of DDPG is deterministic, it may not try enough actions in the initial interactions to find useful learning signals, which limits its exploration. Therefore, during the agent's training process, random noise is introduced into the actions generated by the policy. Let's explore:

[0096] in, Let t be the action output by the Actor's main network. This is the result after adding random noise to the action output of the Actor main network at time t.

[0097] In summary, the specific process of DDForcaster's DDPG algorithm can be simply described as follows: The algorithm initially initializes the network weights of the prediction model pool and the model combination decision module, and initializes the experience replay pool of the model combination decision module.

[0098] In each round of training, the model loss vector is first initialized. And the noise N of the agent's action exploration.

[0099] At each time step, first, acquire the historical load window data of the input. The data is then normalized and input into the prediction model pool to obtain the model prediction set. Then, combine the normalized data from the historical load window. The model loss set from the previous time step Generate a state vector, input it into the Actor network, and add exploration noise. N Get Action , for action The combined weights of the model are obtained after performing Softmax processing. Model combination weights and model prediction set Calculate the predicted load results for the final predicted load window. DDForcaster calculates rewards by comparing the final predicted load with the actual load value. Finally, the experience quadruple Store the data in the experience replay pool and sample it to train and update the Actor network, Critic network, and their target network. Train and update the model network parameters of the prediction model pool by minimizing the loss function, and record the current model loss set. Used for generating state information for the next time step.

[0100] Figure 5 A structural diagram of a microservice load composition prediction system based on deep deterministic policy gradients is provided for an embodiment of the present invention, as shown below. Figure 5 As shown, a microservice load composition prediction system based on deep deterministic policy gradient includes a data preprocessing module, a prediction model pool module, a state construction module, a model combination decision module, and a load prediction synthesis module, wherein: The data preprocessing module is configured to receive historical load data and perform normalization processing to generate normalized historical load data. The prediction model pool module, connected to the data preprocessing module, includes multiple parallel basic prediction models. Each basic prediction model is configured to receive the normalized historical load data and independently generate the corresponding load prediction value and model loss. The state construction module is configured to combine normalized historical load data from the data preprocessing module with the model loss set from the previous time step from the prediction model pool module to generate a state vector. A model combination decision module, connected to the state construction module, uses an agent pre-trained with a deep deterministic policy gradient algorithm. This agent is configured to receive the state vector and output dynamically generated model combination weights. The load prediction synthesis module is connected to the prediction model pool module and the model combination decision module, respectively, and is configured to perform a weighted combination of the model combination weights and the load prediction values ​​generated by each basic prediction model to output the final load prediction result.

[0101] It is understood that the microservice load combination prediction system based on deep deterministic policy gradient provided by the present invention corresponds to the microservice load combination prediction method based on deep deterministic policy gradient provided in the foregoing embodiments. The relevant technical features of the microservice load combination prediction system based on deep deterministic policy gradient can be referred to the relevant technical features of the microservice load combination prediction method based on deep deterministic policy gradient, and will not be repeated here.

[0102] Please see Figure 6 , Figure 6 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 6 As shown, this embodiment of the invention provides an electronic device 600, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, it performs the following steps: S1, after normalizing the historical load data, input it into the pre-trained load prediction model pool. The load prediction model pool includes multiple basic prediction models. Each basic prediction model independently generates load prediction values ​​and model loss based on the normalized historical load data. S2, combine the normalized historical load data and the model loss set of the previous time step to generate a state vector, and input the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool. S3, the weighted combination of the model combination weights and the load prediction values ​​of each basic prediction model are weighted to obtain the final load prediction result.

[0103] Please see Figure 7 , Figure 7 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 7As shown, this embodiment provides a computer-readable storage medium 700, on which a computer program 611 is stored. When the computer program 611 is executed by a processor, it performs the following steps: S1, after normalizing the historical load data, input it into the pre-trained load prediction model pool. The load prediction model pool includes multiple basic prediction models. Each basic prediction model independently generates load prediction values ​​and model loss based on the normalized historical load data. S2, combine the normalized historical load data and the model loss set of the previous time step to generate a state vector, and input the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool. S3, the weighted combination of the model combination weights and the load prediction values ​​of each basic prediction model are weighted to obtain the final load prediction result.

[0104] This invention provides a microservice load combination prediction method, system, and storage medium based on deep deterministic policy gradient. It consists of two parts: a load prediction model pool and a model combination decision module, which constitute an intelligent load predictor based on deep reinforcement learning (DRL). It takes the historical time series of microservice workloads as input and outputs the future prediction sequence of microservice workloads.

[0105] The load forecasting model pool takes historical load window data as input and outputs two parts: load forecast values ​​and model loss. The load forecasting model pool consists of... m It consists of 3 basic models, each with an input length of 1. T Historical load window data, independently trained to generate a length of H The load forecast value of the predicted load window is generated, and the current time period is also generated. m The model loss is used to evaluate the predictive power of the base models in the model pool. The load forecast is passed to the forecast load window to calculate the final load forecast result, while the model loss is passed to the model combination decision module for model combination decision.

[0106] The core of the model combination decision module is the agent, which makes decisions based on current historical load data and the predictive capabilities of the model pool, guiding the model combination scheme in the load prediction model pool. The agent's input consists of two parts: the first part is a string of length... T The historical load window data, and another part is a length of mThe model loss of the load prediction model pool is obtained by the agent making decisions on the current state through policy learning, generating model combination scores, and then processing them through the Softmax layer to obtain model combination weights. The model combination weights are sent to the load prediction window and calculated together with the load prediction values ​​output by each basic model to obtain the final load prediction value.

[0107] This invention generates multi-dimensional preliminary prediction results by constructing a heterogeneous prediction model pool. The model loss and historical load data are used as state inputs to the DDPG agent. Deep reinforcement learning is used to dynamically generate model combination weights, and finally, adaptive prediction is achieved through weighted fusion. This scheme ensures comprehensive feature capture through model pool diversity, achieves accurate weight allocation through a real-time loss feedback mechanism, and optimizes long-term prediction performance through DDPG's sequential decision-making capabilities. This significantly improves the adaptability to the multi-period characteristics, random fluctuations, and trend changes of microservice loads, substantially improving prediction accuracy and system robustness, and providing a more reliable decision-making basis for resource scheduling.

[0108] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0114] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A microservice load composition prediction method based on deep deterministic policy gradient, characterized in that, include: S1, after normalizing the historical load data, input it into the pre-trained load prediction model pool. The load prediction model pool includes multiple basic prediction models. Each basic prediction model independently generates load prediction values ​​and model loss based on the normalized historical load data. S2, combine the normalized historical load data and the model loss set of the previous time step to generate a state vector, and input the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool. S3, the weighted combination of the model combination weights and the load prediction values ​​of each basic prediction model are weighted to obtain the final load prediction result.

2. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 1, characterized in that, Before step S1, the method also includes the step of constructing and pre-training the load prediction model pool, including: Based on the multi-dimensional characteristics of microservice load, a load prediction model pool is constructed using multiple deep learning-based time series prediction models. These multiple deep learning-based time series prediction models include at least the following: LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), BiLSTM (Bidirectional Long Short-Term Memory), TCN (Temporal Convolutional Network), Transformer, and Informer; Each of the basic prediction models is trained using normalized historical load window data as input, and each of the basic prediction models independently outputs the load prediction value and model loss for the corresponding prediction load window.

3. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 1, characterized in that, In step S1, the historical load data is normalized using the min-max normalization method, with the following formula: in, The load data before normalization at time i. This represents the normalized load data at time i. This represents the minimum load on the dataset. This indicates the maximum load on the dataset.

4. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 1, characterized in that, In step S1, the model set M in the load prediction model pool captures normalized historical load data. The time series dependencies are determined, and the model prediction set of length H is output at time t: in, This refers to the load prediction values ​​of multiple basic models in the load prediction model pool at a prediction time step i.

5. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 1, characterized in that, In step S2, the state vector is obtained by concatenating the normalized historical load window data and the model loss of the load prediction model pool at the previous time step, and is expressed as: in, This represents normalized historical load window data. This represents the model loss of the load prediction model pool at the previous time step. Represented as: in, Indicates the previous moment. j The model loss of each model. j ∈[1, m ], m This represents the number of basic models in the load prediction model pool.

6. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 5, characterized in that, In step S2, the step of inputting the state vector into the model combination decision module pre-trained by the deep deterministic policy gradient algorithm to dynamically generate the model combination weights of the load prediction model pool includes: The state vector An Actor network is input to a model combination decision module pre-trained using a deep deterministic policy gradient algorithm, and the Actor network outputs actions in its continuous action space. The action Composed of the model combination score at each prediction time step: in, Indicates the first i Model combination score for each prediction time step H For the forecast load window size; for each forecast time step i Model combination score Represented as: in, R represents the model combination score of the j-th basic model in the load forecasting model pool at the i-th forecasting time step, where R represents the real number field. For the model combination score Perform Softmax processing and map it to the model combination weights. : in, This represents the combined model weights at prediction time step i; By combining the model combination weights at each prediction time step, the model combination weights output by the prediction load window of the model combination decision module are obtained: 。 7. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 6, characterized in that, Step S3 includes: The final load prediction value of the basic prediction model with prediction step size i is obtained by multiplying the load prediction value of each basic prediction model in the load prediction model pool with its corresponding model combination weight. : The final load forecast result is obtained by combining the final load forecast values ​​from all the basic forecast models. : 。 8. The microservice load combination prediction method based on deep deterministic policy gradient as described in claim 7, characterized in that, It also includes the process of pre-training the agent of the model combination decision module using a deep deterministic policy gradient algorithm, wherein the agent is based on an Actor-Critic network structure, including an Actor main network, a target Actor network, a Critic main network and a target Critic network, wherein the Actor main network is used to deterministically output actions based on the current state, and the Critic main network is used to evaluate the value of state-action pairs; The training process includes: (1). Collect experience quadruples through interaction between the agent and the environment. The data is then stored in the experience replay pool, and subsequently, small batches of experience data are randomly sampled from the pool for network updates; among which, Let be the state vector at time t. Let t be the action at time t. To enable the agent to perform actions at time step t The immediate reward obtained afterwards This is the state vector at time step t+1; (2). Update the Critic main network, including: Calculate the TD target: in, Let i be the target of TD, i be the current time, and i+1 be the next time. This is the reward value at the current moment. As a discount factor, The value evaluation function for the target Critic network. For the state at the next moment, The target Actor network in the next time step state The output value, For the learnable parameters of the target Actor network, The learnable parameters of the target Critic network. This indicates that the target Actor network and the target Critic network are used to determine the state at the next time step. To assess its future long-term value; The difference between the TD target and the output value of the Critic main network is used as the loss function L: in, These are the learnable parameters of the Critic main network. This is the output value of the Critic main network. n The number of samples in the current batch of experience data sampled from the experience replay pool; The Critic master network value function is optimized by minimizing the loss function L, and the weights of the Critic master network are updated accordingly. (3) Update the target Critic network, including: Based on a soft update mechanism, the parameters of the target Critic network are slowly copied from the master Critic network to update the parameters of the target Critic network, as follows: in, For the update range, ; (4). Update the Actor main network, including: The parameters of the Actor's main network are updated using policy gradients, as follows: in, Represents the policy gradient. J For a given strategy, n The number of samples in the current batch of experience data sampled from the experience replay pool. Indicates the gradient calculation point. This indicates that the output value of the Critic main network is relative to the input action. a gradient, The action output by the Actor's main network. This represents the learnable parameters of the Actor main network. Indicates the action output by the Actor main network. Relative to its own parameters The gradient, calculated during state. Pick ; (5) Update the target Actor network, including: Based on the soft update mechanism, the parameters of the main Actor network are slowly copied to update the parameters of the target Actor network, which is represented as: 。 9. A microservice load combination prediction method based on deep deterministic policy gradient as described in claim 8, characterized in that, During the training process of the agent, random noise is introduced into the actions generated by the policy. Let's explore: in, Let t be the action output by the Actor's main network. This is the result after adding random noise to the action output of the Actor main network at time t.

10. A microservice load composition prediction system based on deep deterministic policy gradient, characterized in that, include: The data preprocessing module is configured to receive historical load data and perform normalization processing to generate normalized historical load data. The load prediction model pool module, connected to the data preprocessing module, includes multiple parallel basic prediction models. Each basic prediction model is configured to receive the normalized historical load data and independently generate the corresponding load prediction value and model loss. The state construction module is configured to combine normalized historical load data from the data preprocessing module with the model loss set from the previous time step from the load prediction model pool module to generate a state vector. A model combination decision module, connected to the state construction module, uses an agent pre-trained with a deep deterministic policy gradient algorithm. This agent is configured to receive the state vector and output dynamically generated model combination weights. The load prediction synthesis module is connected to the load prediction model pool module and the model combination decision module, respectively, and is configured to perform a weighted combination of the model combination weights and the load prediction values ​​generated by each basic prediction model to output the final load prediction result.

Images