A method for predicting cloud platform resource scheduling based on a phased policy gradient

By combining long-term time series prediction and deep reinforcement learning methods, cloud platform resource scheduling is optimized, solving the problems of excessive resource reservation and high cost in existing technologies, and achieving efficient resource utilization and cost control.

CN118193209BActive Publication Date: 2025-12-26SOUTH CHINA UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410358894.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-27
Publication Date
2025-12-26
Estimated Expiration
2044-03-27

AI Technical Summary

Technical Problem

Existing cloud computing resource management solutions tend to over-reserve resources when dealing with sudden load spikes, leading to increased cloud service usage costs and an inability to fully utilize resource potential. Furthermore, existing predictive elastic scaling fails to comprehensively consider cluster status when making decisions.

Method used

By combining long-term time-series prediction models with deep reinforcement learning, and by constructing policy networks and value networks, using historical load data for training, resource scheduling decisions are optimized, and reward functions are used to update model parameters, thus achieving resource scheduling that comprehensively considers cluster status and load prediction.

Benefits of technology

While ensuring service quality, we can reduce the number of virtual machine nodes used, lower cloud service costs, and improve resource utilization efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118193209B_ABST
    Figure CN118193209B_ABST
Patent Text Reader

Abstract

The application discloses a kind of prediction formula cloud platform resource scheduling methods based on staged policy gradient, including building long-term time series prediction neural network, historical load data is trained;Deep reinforcement learning model is constructed;The load data of node is collected;The load value of future period of service is predicted, and load prediction sequence is obtained;The state of container cluster node and load prediction sequence are input into deep reinforcement learning model, and the actual action of resource scheduling is obtained;Container cluster scheduler carries out resource scheduling in container cluster node according to the action actually executed;The processing condition of all requests handled in the period is counted, and the reward in this period is calculated according to reward function;The network parameters of deep reinforcement learning model are updated according to the reward, the network parameters of deep reinforcement learning model are optimized, and the scheduling performance is improved.The application reduces the use amount of virtual machine node while guaranteeing service quality, reduces the cloud cost for user.
Need to check novelty before this filing date? Find Prior Art

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing and reinforcement learning, and particularly relates to a predictive cloud platform resource scheduling method based on a phased policy gradient. BACKGROUND

[0002] Docker containerization technology can package application services and their dependencies into a lightweight, portable container and run it in any Docker-supported environment, thereby improving the efficiency of application service deployment and resource utilization. Application services are usually packaged in the form of Docker and run on virtual machines, and are managed by a container orchestration platform. The Kubernetes container orchestration platform can implement automated operations of containers, including deployment, expansion and management of containerized application services.

[0003] In the cloud computing environment, technologies such as Docker and Kubernetes have significantly improved the efficiency of application service deployment and operation, but existing resource management schemes face many challenges. The default elastic scaling strategy of Kubernetes adjusts resource allocation according to real-time resource usage and performance indicators. Although this strategy can adapt to immediate changes in load, it may have adverse effects on service quality when dealing with sudden load peaks due to the time delay in starting containers and application services. Predictive elastic scaling strategies adjust resources in advance by predicting cluster load, avoiding the adverse effects of container and application service startup delays on service quality. However, current predictive elastic scaling has the following problems: (1) Elastic scaling mainly combines single-point time series prediction and does not consider long-term load transformation. (2) Elastic scaling cannot consider the cluster state for decision-making compared to resource scheduling. These problems may lead to over-provisioning of resources in some cases, increasing the cost of using cloud services and failing to fully utilize the potential of cloud computing resources. SUMMARY

[0004] In order to overcome the above-mentioned shortcomings and deficiencies of the prior art, the purpose of the present application is to provide a predictive cloud platform resource scheduling method based on a phased policy gradient.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] A predictive cloud platform resource scheduling method based on a phased policy gradient, comprising:

[0007] Building a container cluster based on a Kubernetes container orchestration platform, building a long-term time series prediction neural network, and training it using historical load data, the container cluster including a master node and multiple Node nodes;

[0008] Construct a deep reinforcement learning model, including a policy network and a value network, where the policy network contains an action head and a value head, and the action head is a multi-output classification layer;

[0009] The input to the deep reinforcement learning model is the state S of the container cluster nodes. cluster and load prediction sequence S load′ The output is the actual action of resource scheduling. s ;

[0010] The container cluster scheduler executes actions based on the actual actions performed. s Resource scheduling is performed on container cluster nodes;

[0011] The system analyzes the processing status of all requests handled by the server service during the current period and calculates the reward for this period based on the reward function, which consists of service quality constraints and virtual machine usage costs.

[0012] The network parameters of the deep reinforcement learning model are updated based on the rewards, thus optimizing the network parameters of the deep reinforcement learning model.

[0013] Furthermore, the load prediction sequence S load′ The specific acquisition process is as follows:

[0014] At the beginning of each cycle, load data of the Node is collected, based on the recorded past L... s Load data for each cycle, for the future L of the service p The load value for each cycle is predicted using a long-term time-series prediction neural network to obtain the load prediction sequence S. load′ .

[0015] Furthermore, the input to the deep reinforcement learning model is the state S of the container cluster nodes. cluster and load prediction sequence S load′ The output is the actual action of resource scheduling. s Specifically:

[0016] The node status S of the container cluster nodes cluster and load prediction sequence S load′ The input sequence is formed and serves as the input state S for the deep reinforcement learning model.

[0017] The input state S is fed into the policy network of the deep reinforcement learning model for forward propagation, and the output is the action probability distribution dist for resource scheduling. action .

[0018] Based on the action probability distribution of resource scheduling, dist action Get the actual action executed. s .

[0019] Further, the policy network and the value network are both fully connected neural networks, and the activation function is a tanh activation function.

[0020] Further, the action head is a multi-output classification layer, specifically:

[0021] The resource scheduling problem of the cloud platform is modeled as a multi-dimensional decision problem, and each action dimension represents a resource decision scheme, i.e., a resource configuration of a virtual machine;

[0022] The action head is a multi-output classification layer, and outputs the probability distribution of each action dimension. The number of containers that can be configured for each virtual machine, i.e., action dimension, is determined by the probability distribution output by the corresponding dimension, wherein the action with the highest probability represents the optimal number of container configurations.

[0023] Further, the resource scheduling in the container cluster node is specifically:

[0024] A separate Deployment is created for each node, and when resource scheduling is performed for a node, the Deployment configuration corresponding to the node is modified.

[0025] Further, the load data of the Node node is collected, specifically:

[0026] The load data of the Node node is collected using Promethues and stored in a queue data structure.

[0027] Further, the network parameters of the deep reinforcement learning model are updated according to the reward, specifically:

[0028] The reward is standardized to avoid affecting the calculation of the advantage due to the difference in the scale of the reward.

[0029] The advantage and the reward are calculated using generalized advantage estimation.

[0030] The loss of the policy network and the value network is calculated, and the parameters of the policy network and the value network are updated using an optimizer, and the epoch ppg is repeated.

[0031] The policy network and the value network are updated every interval aux times, and an auxiliary network update phase is performed once.

[0032] Further, the long-term time series prediction neural network sequentially includes an adaptive frequency domain decomposition module, a hybrid granularity feature extraction module, a fully connected feature mapping module, and a fully connected layer.

[0033] Further, the adaptive frequency domain decomposition module includes two independent fully connected layers.

[0034] The mixed granularity feature extraction module comprises four mixed granularity feature extraction sub-modules, and each mixed granularity feature extraction sub-module comprises single granularity feature extraction modules with multi_samp different sampling rates.

[0035] The full connection feature mapping module network structure comprises four multilayer perceptrons.

[0036] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0037] The present application combines a long-term time series prediction model with deep reinforcement learning, realizes scheduling decision-making by comprehensively considering the current cluster state and the predicted future request distribution, and generates a corresponding resource scheduling strategy. The prediction type cloud platform resource scheduling method based on the phase strategy gradient provided by the present application sets a suitable reward function for deep reinforcement learning, realizes reducing the use amount of virtual machine nodes while ensuring the quality of service, and reduces the cloud computing cost for users. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 is a workflow diagram of the present application;

[0039] Figure 2 is a structural schematic diagram of the time series prediction neural network of the present application;

[0040] Figure 3 is a structural diagram of the mixed granularity feature extraction sub-module of the present application;

[0041] Figure 4 is a structural diagram of the single granularity feature extraction module of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below in combination with embodiments, but the implementation manner of the present application is not limited thereto.

[0043] Embodiment:

[0044] As shown in the figure, a prediction type cloud platform resource scheduling method based on a phase strategy gradient comprises the following steps: Figures 1-4

[0045] Step S1: build a container cluster based on a Kubernetes container orchestration platform, build a long-term time series prediction neural network, and train using historical load data of cloud platform application services;

[0046] Specifically:

[0047] Step S1.1: build a Kubernetes cluster, and the cluster comprises one master node and vms Node nodes.

[0048] ​In this embodiment, vms is 6, and the cluster has 6 Node nodes in total.

[0049] Step S1.2: Deploy Node Exporter in the form of Daemonset in the cluster, to ensure that a Node Exporter Pod runs on each node in the cluster.

[0050] Step S1.3: Expose the Node Exporter Pod using the Node Exporter Service, so that Prometheus can discover and scrape the metrics on these Pods.

[0051] Step S1.4: Define ClusterRole, ServiceAccount, ClusterRoleBinding to provide Prometheus with the necessary permissions to enable it to obtain information from the Kubernetes API server.

[0052] Step S1.5: Deploy Prometheus to the master node in the form of Deployment.

[0053] Step S1.6: Build a long-term time series prediction neural network, which sequentially includes an adaptive frequency domain decomposition module, a hybrid granularity feature extraction module, a fully connected feature mapping module, and a fully connected layer.

[0054] Step S1.7: Train the long-term time series prediction neural network using the historical load data of the cloud platform application service.

[0055] In this embodiment, the load data is specifically CPU usage.

[0056] In this embodiment, in the long-term time series prediction neural network:

[0057] (1) The adaptive frequency domain decomposition module includes two independent fully connected layers, each with an input feature dimension of the input sequence length L s and an output feature dimension of 1.

[0058] In this embodiment, L s is set to 336.

[0059] (2) The hybrid granularity feature extraction module is composed of 4 hybrid granularity feature extraction submodules, each containing multi_samp single granularity feature extraction modules with different sampling rates. For a single granularity feature extraction module with a sampling rate of samp, it contains samp multi-layer perceptrons. The structure of each multi-layer perceptron is a fully connected layer, a Gaussian error linear unit, and a fully connected layer.

[0060] In this embodiment, the specific structure of the mixed granularity feature extraction submodule is as shown in Figure 3 The multi_samp is 3, containing 3 single granularity feature extraction modules, and the sampling rates are 6, 8 and 12 respectively. The specific structure of the single granularity feature extraction module with a sampling rate of n is as shown in Figure 4 It contains n multi-layer perceptrons, each of which has the same structure and independent parameters. The specific structure is a fully connected layer, a Gaussian error linear unit and a fully connected layer. The input dimension of the first fully connected layer is The output dimension is 512, the input dimension of the second fully connected layer is 512, and the output dimension is

[0061] (3) The network structure of the fully connected feature mapping module is as shown in Figure 2 It contains 4 multi-layer perceptrons, each of which has the same structure and independent parameters. The specific structure is a fully connected layer, a Gaussian error linear unit and a fully connected layer. The input dimension of the first fully connected layer is L s , and the output dimension is hidden_dim1. The input dimension of the second fully connected layer is hidden_dim1, and the output dimension is L p , and L p is the prediction sequence length.

[0062] In this embodiment, hidden_dim1 is 512, and L p is 12.

[0063] (4) The input dimension of the fully connected layer is 4, and the output dimension is 1.

[0064] Step S1.7, import the trained model parameters into the long-term time series prediction neural network.

[0065] Step S2: Construct a deep reinforcement learning model including a policy network and a value network, wherein the policy network includes an action head and a value head, and the action head is a multi-output classification layer.

[0066] Further, the policy network sequentially includes a hidden layer, an action head and a value head. The hidden layer is a fully connected neural network, the input dimension is state_dim, and the output dimension is hidden_dim2. The state_dim is the dimension of the input state, and the calculation formula is as follows:

[0067] state_dim = vms * 2 + L p

[0068] Further, the action head is a multi-output classification layer, including a fully connected layer and a softmax function, the input dimension of the fully connected layer is hidden_dim2, and the output dimension is action_dim*vms. The reshaped dimension of the output of the fully connected layer is (vms, action_dim), and then the softmax function is applied to obtain the probability distribution dist of each dimension action action . action_dim is the value range of the number of node containers.

[0069] Further, the value head is a fully connected layer, the input dimension is hidden_dim2, and the output dimension is 1.

[0070] In the embodiment, hidden_dim2 is 256, action_dim is 3, and the value range of the number of node containers is {0, 1, 2}.

[0071] Step S3: At the beginning of each period, the load data of the Node node is collected and stored. According to the recorded load data of the past L s periods, the load value of the future L p periods of the service is predicted to obtain a load prediction sequence;

[0072] In the embodiment, the preferred mode of step S3 is:

[0073] Step S3.1: A query request is sent to the API address of Prometheus through PromQL, and the response is parsed and returned to obtain the CPU usage of the Node node.

[0074] Step S3.2: The CPU usage is stored in a queue data structure with a size of L s .

[0075] Step S3.3: All load values stored in the queue data structure are combined to form an input sequence, which is input into the long-term time series prediction neural network for prediction to obtain a CPU usage prediction sequence S load′ .

[0076] Step S4: The state S cluster of the container cluster node and the load prediction sequence S load′ are input into the deep reinforcement learning model to obtain the actual action action s of resource scheduling.

[0077] Specifically:

[0078] Step S4.1: The node state S cluster of the container cluster node and the CPU usage prediction sequence S load′The input sequence, as the input state S of the deep reinforcement learning model, is represented as follows:

[0079] S = [S cluster ,S load′ ]

[0080] The node state S of the container cluster node is represented as follows: cluster

[0081]

[0082] wherein, represents the remaining survival time of the i-th virtual machine, represents the number of containers running on the i-th virtual machine.

[0083] The CPU usage prediction sequence S is represented as follows: load′

[0084]

[0085] wherein, CPU t+k represents the CPU usage prediction value at the k-th time point in the future from the current time t.

[0086] The overall state input into the deep reinforcement learning model is represented as follows:

[0087]

[0088] Step S4.2: input the input state S into the policy network in the deep reinforcement learning model, perform forward propagation, and output the action probability distribution dist action of resource scheduling.

[0089] Step S4.3: according to the action probability distribution dist action of resource scheduling, obtain the actually executed action action s .

[0090] Specifically,

[0091] The shape of dist action is (vms, action_dim), vms is the number of dimensions of the action space, and action_dim is the number of actions in each dimension.

[0092] Step S4.3.1: for each dimension dim i of the action space, construct a Categorical distribution of the action probability of the dim i -th dimension, and the calculation formula is as follows:

[0093] dist i ​​= Categorical(a[:, i])

[0094] Step S4.3.2: During training, randomly sample an action from dist i ; during testing, select the action with the highest probability, and obtain the action actually performed in this dimension action i .

[0095] Step S4.3.3: Calculate the log probability of the action actually performed, and the calculation formula is as follows:

[0096] log_prob i = log(dist i (action i ))

[0097] Step S4.3.4: Concatenate the actions action i in all dimensions into the action vector action s actually performed, and concatenate the log probabilities log_prob i in all dimensions to obtain log_prob. In this embodiment, action s is represented as follows:

[0098] [c_num1, c_num2, c_num3, c_num4, c_num5, c_num6]

[0099] Where, when c_num i is not 0, it means placing c_num containers at the i-th virtual machine (if the i-th virtual machine is not started before performing the action, it is started first and then placed); when c_num i is 0, it means setting the i-th virtual machine to the off state. For example, when action s = [0, 1, 2, 0, 0, 0], it means setting the 1st, 4th, 5th and 6th virtual machines to the off state, placing 1 container on the 2nd virtual machine, and placing 2 containers on the 3rd virtual machine.

[0100] Step S5: The container cluster scheduler performs resource scheduling on the container cluster nodes according to the action actually performed action s .

[0101] Specifically:

[0102] Step S5.1: Initialize the connection with the Kubernetes cluster according to the user's kubeconfig file.

[0103] Step S5.2: Obtain all worker nodes in the cluster as candidate nodes for deploying Pods.

[0104] Step S5.3: According to action s and worker nodes, create or update Deployment for each node. If Deployment on a given node does not exist, create a new Deployment; if Deployment already exists, update its replica number. Each worker node has an independent Deployment.

[0105] Step S6: Count the processing of all requests processed in the period, and calculate the reward in this period according to the reward function composed of service quality constraints and virtual machine usage cost;

[0106] Further, the specific formula of the reward function in step S6 is as follows:

[0107] R t = γ p R p + γ vm Cost vm

[0108] Wherein, R p represents the reward for processing requests in the period; Cost vm represents the cost of new virtual machines in the period; γ p is the weight coefficient of the reward for processing requests; γ vm is the weight coefficient of the cost of new virtual machines.

[0109] Further, the specific formula of the reward R p for processing requests is as follows:

[0110] R request = N fast ·reward fast +N slow ·reward slow -N fail ·penalty

[0111] Wherein, N fast is the number of successfully processed requests with processing speed not greater than speed fast , N slow is the number of successfully processed requests with processing speed greater than speed fast , and N fail is the number of failed requests.

[0112] Further, the specific formula of the cost Cost vm of new virtual machines is as follows:

[0113] Costvm = N vm · Cost vm

[0114] wherein, N vm is the number of newly added virtual machines in the period, Cost vm is the use cost of the minimum life cycle of the virtual machine.

[0115] In the embodiment, γ p is 1, γ vm is 1, reward fast is 3, reward slow is 2, penalty is 10, Cost vm is 1200, speed fast is 2000ms.

[0116] Step S7: updating the network parameters of the deep reinforcement learning model according to the reward;

[0117] Specifically:

[0118] Step S7.1: reward standardization, avoiding the influence on the advantage calculation due to the difference in the scale of the reward, the calculation formula is as follows:

[0119]

[0120] wherein, μ reward is the mean of reward, σ reward is the standard deviation of reward.

[0121] Step S7.2: using generalized advantage estimation (GAE) to calculate the advantage and reward, the calculation formula is as follows:

[0122] δ t = r t + γV value (s t+1 ) - V value (s t )

[0123]

[0124] R t = A t + V value (s t )

[0125] wherein, r t is the standardized reward at time t, γ is the discount factor, λ gae is the smoothing parameter, T is the terminal time step of the sequence, Vvalue (s) is the output of the value network in state s.

[0126] In this embodiment, γ is 0.8254, λ gae is 0.95.

[0127] Step S7.3: Calculate the loss of the policy network and the value network, and update the parameters of the policy network and the value network using the optimizer, repeat the epoch ppg times.

[0128] Specifically:

[0129] Step S7.3.1: Calculate the new-old policy ratio r t (θ), since the resource scheduling action space designed by the present application is a multi-dimensional action space, it is difficult to directly calculate r t (θ), therefore, the action space of each dimension is processed independently to decouple the mutual relationship between them, that is, to decouple the control dependency relationship, after removing the control dependency relationship, the new-old policy ratio r t (θ) is calculated as follows:

[0130]

[0131]

[0132] Where d=vms, the dimension of the action space.

[0133] Step S7.3.2: The policy network uses the loss of the PPO-CLIP version, and the calculation formula is as follows:

[0134]

[0135]

[0136] Where ∈ is the clipping parameter, S(π θ (s)) is the entropy of the policy, which is used to encourage exploration.

[0137] In this embodiment, ∈ is set to 0.2, γ entropy is set to 0.01.

[0138] Step S7.3.3: The value network uses the MSE loss function, and the calculation formula is as follows:

[0139]

[0140] Step S7.4: Auxiliary training, after executing step S7.3 aux interval times, execute an auxiliary network update phase.

[0141] Further, the auxiliary network updating phase in step S7.4 is specifically implemented as follows:

[0142] Step S7.4.1: input the input state into the policy network to obtain the action probability πθ of the old policy in the corresponding state old (a|s).

[0143] Step S7.4.2: update the policy network in the auxiliary phase, and the loss function calculation formula used is as follows:

[0144]

[0145] Wherein, V policy (s t ) is the output of the state s t of the policy network value head, and KL is the KL divergence, which guarantees the stability of the policy update.

[0146] Step S7.4.3: update the value network in the auxiliary phase, and use the MSE loss function to calculate, as in step S7.3.3.

[0147] Step S7.4.4: repeat the execution of epoch aux times of steps S7.4.2 and S7.4.3.

[0148] In this embodiment, interval aux is 32, and epoch aux is 6.

[0149] Step S8: repeat steps S3-S7 to optimize the network parameters of the deep reinforcement learning model, improve the scheduling performance, obtain the optimized deep reinforcement learning model, and use it for predicting cloud platform resource scheduling.

[0150] The above embodiment is a preferred embodiment of the present application, but the embodiments of the present application are not limited by the above-mentioned embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application are equivalent replacement methods, which are all included in the protection scope of the present application.

Claims

1. A method for predicting cloud platform resource scheduling based on a stage strategy gradient, characterized in that, The application relates to a cloud platform resource scheduling method based on deep reinforcement learning, and belongs to the technical field of cloud computing. The application comprises the following steps: A container cluster is built based on a Kubernetes container orchestration platform, a long-term time series prediction neural network is built, and the long-term time series prediction neural network is trained using historical load data, wherein the container cluster comprises one master node and multiple Node nodes; The input of the deep reinforcement learning model is the state S of the container cluster node cluster And the load prediction sequence S load′ , and the output is the actual action action of resource scheduling s ; The container cluster scheduler schedules the actual action action s Resource scheduling is performed on the container cluster node; A deep reinforcement learning model is built, which comprises a policy network and a value network, wherein the policy network comprises an action head and a value head, and the action head is a multi-output classification layer; The processing conditions of all requests processed by a period container service are counted, and the reward in the period is calculated according to a reward function, wherein the reward function is composed of a service quality constraint and a virtual machine use cost; reward = γ p R p + γ vm Cost vm wherein, reward represents the reward of processing requests in a period; Cost vm represents the cost of adding virtual machines in a period; γ p is the weight coefficient of the reward of processing requests; γ vm is the weight coefficient of the cost of adding virtual machines; The R p The specific formula is as follows: R p = N fast • reward fast + N slow • reward slow - N fail • penalty N fast is the number of successful processing requests with processing speed not greater than speed fast N slow is the number of successful processing requests with processing speed greater than speed fast N fail is the number of failed processing requests; The newly added virtual machine cost Cost vm The specific formula is as follows: Cost vm = N vm · Cost min wherein, N vm is the number of virtual machines added in the period, Cost min is the usage cost of the minimum life cycle of the virtual machine; The specific formula of the reward function is as follows: The network parameters of the deep reinforcement learning model are updated according to the reward, the network parameters of the deep reinforcement learning model are optimized, and an optimized deep reinforcement learning model is obtained; The network parameters of the deep reinforcement learning model are updated according to the reward, and the specific process is as follows: where μ reward is the mean of the reward, σ reward is the standard deviation of the reward; The reward is standardized to avoid the influence of the calculation of the advantage due to the difference in the scale of the reward, and the calculation formula is as follows: delta t = r t + gamma V value (s t+1 ) - V value (s t ) A t = δ t + (γλ gae )δ t+1 +…+(γλ gae ) T-t+1 δ T-1 R t = A t + V value (s t ) where r t is the normalized reward at time t, g is the discount factor, l gae is the smoothing parameter, T is the terminal time step of the sequence, V value (s) is the output of the value network at state s; Compute the loss of the policy network and the value network, and update the parameters of the policy network and the value network using an optimizer, repeat the epoch ppg seconds; Assist training, every interval aux The next step, perform an assist network update phase once. 2.The method of claim 1, wherein, The load prediction sequence S load′ The specific obtaining process is as follows: At the beginning of each cycle, the load data of the Node node is collected, and the load value of the future L s cycles of the service is predicted through a long-term time series prediction neural network according to the recorded load data of the past L p cycles, to obtain a load prediction sequence S load′ . 3.The method of claim 1, wherein, The input of the deep reinforcement learning model is a state S of a container cluster node cluster and a load prediction sequence S load′ , and the output is an actual action action of resource scheduling s , specifically: composing an input sequence of node states S cluster and load prediction sequences S load′ as input states S for a deep reinforcement learning model; input the input state S into a policy network in a deep reinforcement learning model, perform forward propagation, and output an action probability distribution dist of resource scheduling action ; According to the action probability distribution dist of the resource scheduling action Get the actual executed action action s . 4.The method of claim 1, wherein, The advantage and the reward are calculated using generalized advantage estimation, and the calculation formula is as follows:

5. The predictive cloud platform resource scheduling method of claim 1, wherein, The policy network and the value network are both fully connected neural networks, and the activation function is a tanh activation function. The action head is a multi-output classification layer, and the specific process is as follows: The resource scheduling problem of the cloud platform is modeled as a multi-dimensional decision problem, and each action dimension represents a resource decision scheme, that is, a resource configuration of a virtual machine; 6. The predictive cloud platform resource scheduling method of claim 1, wherein, The action head is a multi-output classification layer, and the probability distribution of each action dimension is output; the number of containers that can be configured for each virtual machine, that is, each action dimension, is determined by the probability distribution output by the corresponding dimension, wherein the action with the highest probability represents the optimal number of container configurations. The resource scheduling is performed on the container cluster nodes, and the specific process is as follows:

7. The predictive cloud platform resource scheduling method of claim 2, wherein, An independent Deployment is created for each node, and when the resource scheduling is performed on a certain node, the Deployment configuration corresponding to the node is modified. Load data of the Node nodes is collected, and the specific process is as follows:

8. The predictive cloud platform resource scheduling method of any of claims 1-7, wherein, The load data of the Node nodes is collected using Promethues and stored in a queue data structure. 9.The method of claim 8, wherein, The long-term time series prediction neural network comprises, in sequence, an adaptive frequency domain decomposition module, a hybrid granularity feature extraction module, a fully connected feature mapping module and a fully connected layer. The adaptive frequency domain decomposition module comprises two independent fully connected layers; The hybrid granularity feature extraction module comprises four hybrid granularity feature extraction submodules, each hybrid granularity feature extraction submodule comprises multi_samp single-granularity feature extraction modules with different sampling rates; The network structure of the fully connected feature mapping module comprises four multilayer perceptrons.

Citation Information

Patent Citations

  • Virtual machine optimization scheduling method for cloud computing

    CN115016889A

  • Computing resource configuration method and device

    CN115357339A