A resource management method and system in a cloud computing environment

Abstract

A resource management method and system in a cloud computing environment, relating to the technical field of cloud computing, is used to solve the problem that the existing static resource management of virtual nodes in a cloud computing environment cannot effectively improve their operating performance. The technical points of the present invention include: collecting the CPU, memory and bandwidth resource information of the virtual node, proposing a DDPG-based resource allocation algorithm to calculate the target resource configuration amount of each virtual node according to the above resource information to obtain the result of the resource configuration decision, and dynamically adjust the resources of each virtual node according to the result, thereby improving the operating performance of the virtual node on the basis of improving the overall resource utilization; further, in the DDPG-based resource allocation algorithm, the Actor network is used to generate the corresponding action output, and the ratio-based output is converted into the target resource configuration size through an action decoding process. The present invention can shorten the completion time of different applications as a whole, and will not generate a large amount of system overhead.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a resource management method and system in a cloud computing environment. Background Art

[0002] In recent years, with the development and rise of cloud computing technology, virtualization technology has begun to reappear in people's sight. The purpose of virtualization technology is to virtualize physical resources and reasonably allocate virtualized physical resources to multiple virtual nodes for use, and to ensure that multiple virtual nodes are independent of each other and do not affect the operation of their respective tasks. In many cloud computing scenarios, static resources are allocated to virtual nodes at the beginning of creation, including CPU, memory, and bandwidth resources. However, as the load of virtual nodes changes dynamically during operation, the static resources of virtual nodes sometimes cannot handle the load tasks well, resulting in the limitation of the operation performance of virtual nodes. Summary of the invention

[0003] In view of the above problems, the present invention proposes a resource management method and system in a cloud computing environment to solve the problem that the existing static resource management of virtual nodes in the cloud computing environment cannot effectively improve their operating performance.

[0004] According to one aspect of the present invention, a resource management method in a cloud computing environment is provided, the method comprising the following steps:

[0005] Step 1: Collect resource information of each virtual node in real time, including CPU usage time, free memory and bandwidth information;

[0006] Step 2: Obtain the CPU, memory and bandwidth resource configuration of each virtual node;

[0007] Step 3: Calculate and extract based on the resource information and the CPU, memory and bandwidth resource configuration amounts to obtain the CPU, memory and bandwidth usage rates of each virtual node; the CPU, memory and bandwidth resource configuration amounts and the CPU, memory and bandwidth usage rates together constitute a state set;

[0008] Step 4: Use a resource management algorithm based on deep deterministic policy gradient to train and learn the state set to obtain resource configuration decisions for each virtual node;

[0009] Step 5: Dynamically adjust the resources of each virtual node according to the resource allocation decision.

[0010] Furthermore, in step 3, the CPU usage uc i 、Memory usage um i and bandwidth usage ub i The calculation formulas are:

[0011]

[0012]

[0013]

[0014] Where cpu(t) represents the CPU usage time at time t; c i Indicates the amount of CPU resource configuration; f i Indicates free memory; m i represents the amount of memory resource configuration; recv(t) represents the number of bandwidth packets received at time t; trans(t) represents the number of bandwidth packets sent at time t; b i Indicates the bandwidth resource configuration amount.

[0015] Furthermore, the specific steps of step 4 include:

[0016] Step 41: Initialize the current Actor network π θ (S), current critic network Q ω (S,A), target actor network π′ θ′ (S) and the target Critic network Q′ ω′ (S,A), and create an experience pool;

[0017] Step 42: Initialize the environment and obtain the current state information at time t;

[0018] Step 43: The current state s at time t is concentrated in the state t As input, according to the Actor network π θ (S) Calculate the discrete action and add noise to obtain the action a at the current time t t :

[0019]

[0020] In the formula, π θ (s t ) represents the Actor network at the current time t; It means that the mean is μ and the variance is σ 2 Normally distributed noise;

[0021] Step 44: According to the current state s at time t t The action a at the current time t is decoded using a ratio-based action decoding algorithm. t Decode and obtain the decoded resource configuration action;

[0022] Step 45: Perform resource configuration according to the decoded resource configuration action;

[0023] Step 46: Get the state s at time t+1 in the state set t+1 , and calculate the instant reward r according to the following reward function formula t :

[0024] r t = -λ1Bads-λ2Vars+λ3Evav

[0025] In the formula, λ1, λ2, and λ3 represent weight parameters; Bads represents the number of virtual instances in the state of over-utilization or under-utilization of resources; Vars represents the sum of the variances of CPU, memory, and bandwidth usage. The smaller the value, the more balanced the resource utilization of the virtual node; Evav represents the evaluation value of each resource management action operation. The larger the value, the more recognized it is.

[0026] Step 47: Set the current state s t , action a at time t t , state s at time t+1 t+1 and instant rewards t Right now t ,a t ,s t+1 ,r t >Deposit into experience pool;

[0027] Step 48: Sample a mini-batch from the experience pool and perform a gradient update on the parameter ω in the critic network at the current time t based on the following formula:

[0028]

[0029] In the formula, E represents the expected value; Q ω (s t ,a t ) represents the Critic network at the current time t; Represents the derivative of the parameter ω of the current Critic network; y t represents the target value, y t =r t +γQ′ ω′ (s t+1 ,π′ θ′ (s t+1 )), γ represents the loss rate, Q′ ω′ (s t+1 ,π′ θ′ (s t+1 )) represents the target Critic network at time t+1, π′ θ′ (s t+1 ) represents the target Actor network at time t+1;

[0030] ​Step 49: Sample a mini-batch from the experience pool and perform a gradient update on the parameter θ in the Actor network at the current time t based on the following formula:

[0031]

[0032] In the formula, Indicates the derivation of the parameter a of the current Critic network; Represents the derivative of the parameters θ of the current Actor network;

[0033] Step 40: Update the parameter θ′ of the target Actor network by the following formula: θ′←τθ+(1-τ)θ′;

[0034] Update the parameter ω′ of the target Critic network by the following formula: ω′←τω+(1-τ)ω′; where τ represents the momentum factor;

[0035] Step 41: Iterate and loop through Step 42 to Step 40 until the maximum number of iterations is reached, and then stop iterating.

[0036] Furthermore, the specific steps of step 44 include:

[0037] Step 441: Decode the memory action of the i-th virtual node:

[0038] a. For memory action value The corresponding memory resource configuration is:

[0039]

[0040] b. For memory action value Calculate the amount of memory that can be allocated And obtain the sum of the positive action value ratio, then the corresponding memory resource configuration is:

[0041]

[0042] Step 442: Decode the memory action of the i-th virtual node:

[0043] CPU Action Value When the value is in the range of [-1, -0.2), [-0.2, 0.2], and (0.2, 1], it corresponds to the following CPU resource configuration amounts: the CPU resource configuration amount decreases by 1, the CPU resource configuration amount remains unchanged, and the CPU resource configuration amount increases by 1. The corresponding action results are:

[0044] Step 443: Decode the bandwidth action of the i-th virtual node:

[0045] Bandwidth Action Value The corresponding bandwidth resource configuration is:

[0046] Furthermore, step 443 also includes step 444: after determining the resource configuration amount obtained by the virtual node after executing the resource configuration action, if the resource configuration amount is lower than the preset minimum resource configuration amount or exceeds the preset maximum resource configuration amount, then the resource configuration action is changed to the corresponding preset minimum resource configuration amount or preset maximum resource configuration amount.

[0047] According to another aspect of the present invention, a resource management system in a cloud computing environment is provided, the system comprising an information collection module, a resource calculation module and a resource adjustment module; wherein:

[0048] The information collection module is used to collect resource information of each virtual node in real time, the resource information includes CPU usage time, free memory and bandwidth information; obtain the CPU, memory and bandwidth resource configuration of each virtual node;

[0049] The resource calculation module is used to calculate and extract according to the resource information and the CPU, memory and bandwidth resource configuration amounts to obtain the CPU, memory and bandwidth utilization rate of each virtual node; the CPU, memory and bandwidth resource configuration amounts and the CPU, memory and bandwidth utilization rates together constitute a state set; the state set is trained and learned using a resource management algorithm based on deep deterministic policy gradient to obtain a resource configuration decision for each virtual node;

[0050] The resource adjustment module is used to dynamically adjust the resources of each virtual node according to the resource configuration decision.

[0051] Furthermore, the CPU usage rate uc in the resource calculation module i 、Memory usage um i and bandwidth usage ub i The calculation formulas are:

[0052]

[0053]

[0054]

[0055] Where cpu(t) represents the CPU usage time at time t; c i Indicates the amount of CPU resource configuration; f i Indicates free memory; m irepresents the amount of memory resource configuration; recv(t) represents the number of bandwidth packets received at time t; trans(t) represents the number of bandwidth packets sent at time t; b i Indicates the bandwidth resource configuration amount.

[0056] Furthermore, the resource calculation module uses a resource management algorithm combining an Actor-Critic network and a deep deterministic policy gradient to perform training and learning based on the state set, and the specific steps of obtaining a resource configuration decision for each virtual node include:

[0057] Step 41: Initialize the current Actor network π θ (S), current critic network Q ω (S,A), target actor network π′ θ′ (S) and the target Critic network Q′ ω′ (S,A), and create an experience pool;

[0058] Step 42: Initialize the environment and obtain the current state information at time t;

[0059] Step 43: The current state s at time t is concentrated in the state t As input, according to the Actor network π θ (S) Calculate the discrete action and add noise to obtain the action a at the current time t t :

[0060]

[0061] In the formula, π θ (s t ) represents the Actor network at the current time t; It means that the mean is μ and the variance is σ 2 Normally distributed noise;

[0062] Step 44: According to the current state s at time t t The action a at the current time t is decoded using a ratio-based action decoding algorithm. t Decode and obtain the decoded resource configuration action;

[0063] Step 45: Perform resource configuration according to the decoded resource configuration action;

[0064] Step 46: Get the state s at time t+1 in the state set t+1 , and calculate the instant reward r according to the following reward function formula t :

[0065] r t = -λ1Bads-λ2Vars+λ3Evav

[0066] In the formula, λ1, λ2, and λ3 represent weight parameters; Bads represents the number of virtual instances in the state of over-utilization or under-utilization of resources; Vars represents the sum of the variances of CPU, memory, and bandwidth usage. The smaller the value, the more balanced the resource utilization of the virtual node; Evav represents the evaluation value of each resource management action operation. The larger the value, the more recognized it is.

[0067] Step 47: Set the current state s t , action a at time t t , state s at time t+1 t+1 and instant rewards t Right now t ,a t ,s t+1 ,r t >Deposit into experience pool;

[0068] Step 48: Sample a mini-batch from the experience pool and perform a gradient update on the parameter ω in the critic network at the current time t based on the following formula:

[0069]

[0070] In the formula, E represents the expected value; Q ω (s t ,a t ) represents the Critic network at the current time t; Represents the derivative of the parameter ω of the current Critic network; y t represents the target value, y t =r t +γQ′ ω′ (s t+1 ,π′ θ′ (s t+1 )), γ represents the loss rate, Q′ ω′ (s t+1 ,π′ θ′ (s t+1 )) represents the target Critic network at time t+1, π′ θ′ (s t+1 ) represents the target Actor network at time t+1;

[0071] Step 49: Sample a mini-batch from the experience pool and perform a gradient update on the parameter θ in the Actor network at the current time t based on the following formula:

[0072]

[0073] In the formula, Indicates the derivation of the parameter a of the current Critic network;​ Represents the derivative of the parameters θ of the current Actor network;

[0074] Step 40: Update the parameter θ′ of the target Actor network by the following formula: θ′←τθ+(1-τ)θ′;

[0075] Update the parameter ω′ of the target Critic network by the following formula: ω′←τω+(1-τ)ω′; where τ represents the momentum factor;

[0076] Step 41: Iterate and loop through Step 42 to Step 40 until the maximum number of iterations is reached, and then stop iterating.

[0077] Furthermore, the resource calculation module calculates the current state s at time t. t The action a at the current time t is decoded using a ratio-based action decoding algorithm. t The specific steps of decoding and obtaining the decoded resource configuration action include:

[0078] Step 441: Decode the memory action of the i-th virtual node:

[0079] a. For memory action value The corresponding memory resource configuration is:

[0080]

[0081] b. For memory action value Calculate the amount of memory that can be allocated And obtain the sum of the positive action value ratio, then the corresponding memory resource configuration is:

[0082]

[0083] Step 442: Decode the memory action of the i-th virtual node:

[0084] CPU Action Value When the value is in the range of [-1, -0.2), [-0.2, 0.2], and (0.2, 1], it corresponds to the following CPU resource configuration amounts: the CPU resource configuration amount decreases by 1, the CPU resource configuration amount remains unchanged, and the CPU resource configuration amount increases by 1. The corresponding action results are:

[0085] Step 443: Decode the bandwidth action of the i-th virtual node:

[0086] Bandwidth Action Value The corresponding bandwidth resource configuration is:

[0087] Step 444: After determining the resource configuration amount obtained by the virtual node after executing the resource configuration action, if the resource configuration amount is lower than the preset minimum resource configuration amount or exceeds the preset maximum resource configuration amount, the resource configuration action is changed to the corresponding preset minimum resource configuration amount or preset maximum resource configuration amount.

[0088] The beneficial technical effects of the present invention are:

[0089] The present invention provides a dynamic resource management method and system for virtual nodes in a cloud computing environment, and dynamically configures virtual node resources to improve the operating performance of virtual nodes. The present invention first collects the required information, including the CPU, memory and bandwidth resource information of the virtual node, and proposes a DDPG-based resource allocation algorithm to calculate the target resource configuration amount of each virtual node according to the acquired resource information to obtain the result of the resource configuration decision, and dynamically adjust the resources of each virtual node according to the result, thereby improving the operating performance of the virtual node on the basis of improving the overall resource utilization; further, in the DDPG-based resource allocation algorithm, the Actor network is used to generate the corresponding action output, and the ratio-based output is converted into the target resource configuration size through an action decoding process. Experiments have shown that the present invention can shorten the completion time of different applications as a whole, and will not generate a large amount of system overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a schematic diagram of a resource management system architecture in a cloud computing environment of the present invention;

[0091] Figure 2 is an example diagram of evaluation rules for each resource management action operation in the present invention;

[0092] Figure 3 1 is a performance result comparison diagram in a KVM virtual machine environment in an embodiment of the present invention; wherein, Figure (a) represents a single benchmark program; Figure (b) represents an Httpload task under different benchmark program loads; Figure (c) represents each benchmark program under Httpload load;

[0093] Figure 4 1 is a performance comparison diagram of the Docker container environment in an embodiment of the present invention; wherein, Figure (a) represents a single benchmark program; Figure (b) represents the Httpload task under different benchmark program loads; Figure (c) represents each benchmark program under the Httpload load;

[0094] Figure 5 1 is a comparison diagram of energy consumption on a KVN host in an embodiment of the present invention; wherein, Figure (a) represents CPU utilization; Figure (b) represents memory utilization;

[0095] Figure 6 1 is a comparison diagram of energy consumption on a Docker host in an embodiment of the present invention; wherein, Figure (a) represents CPU utilization; and Figure (b) represents memory utilization. DETAILED DESCRIPTION

[0096] In order to enable those skilled in the art to better understand the solutions of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0097] The present invention designs a dynamic resource management framework for virtual nodes in a cloud computing environment, thereby realizing dynamic configuration of virtual node resources, hoping to improve the operating performance of virtual nodes in this way. The virtual nodes studied in the present invention include two virtualization technologies, KVM and Docker, which are currently widely used, and the managed resources include CPU, memory, and bandwidth resources.

[0098] The system of the present invention generally includes three modules, namely, an information collection module, a resource calculation module and a resource adjustment module. Among them, the information collection module is responsible for collecting the required information through the underlying interface function, including the CPU, memory and bandwidth resource information of the virtual node, and transmitting the collected information to the resource calculation module; after the resource calculation module obtains the information, it will call the algorithm to obtain the result of the resource configuration decision, and transmit the result to the resource adjustment module; the resource adjustment module dynamically adjusts the resources of each virtual node according to the result, thereby improving the virtual node operation performance on the basis of improving the overall resource utilization. The overall framework of the system is as follows: Figure 1 shown.

[0099] In the resource management system, it is necessary to first obtain the CPU, memory and network information of each virtual node. As shown in Table 1, Table 1 lists the resource information of KVM and Docker virtual nodes.

[0100] Table 1 Virtual node resource information

[0101]

[0102]

[0103] The virtualization management software libvirt is used to collect information about the virtual machines in the middle. It is a widely used tool in KVM hypervisors. The CPU and memory information are obtained through virDomain.info() and virDomain.MemoryStats() respectively. The fourth parameter returned in the former is the CPU usage time in nanoseconds (ns). The free memory size of each virtual machine is also obtained. In addition, the communication packet information is obtained by observing / proc / net / dev.

[0104] These raw information need to be converted into input data for the resource calculation module in order to serve the subsequent resource management algorithm. i , respectively, by using formulas (1)-(3) to obtain the CPU usage uc i 、Memory usage um i and bandwidth usage ub i .

[0105]

[0106]

[0107]

[0108] Among them, c i 、m i and b i They are v i CPU, memory, and bandwidth resource configuration.

[0109] The CPU, memory, and network resource information of the Docker container is obtained through the docker stats command. It first reads the data values ​​of the CPU and memory parameters under the cgroup file system through the Docker Daemon record, then accumulates the data values, forms an array through the read values, and finally obtains the corresponding CPU and memory utilization, as well as the number of network bytes received and sent, through the calculation formula.

[0110] The resource calculation module periodically obtains the resource information of all virtual nodes from the information collection module and makes global decisions for the resource adjustment module to automatically schedule resources between multiple virtual nodes. In order to maintain synchronization between collection and calculation, a periodic program with an interval of 5 seconds (s) is set to monitor and schedule resources. In each cycle, the resource calculation module uses the resource scheduling algorithm to calculate the target resource allocation for each virtual node according to the current workload. The core of the resource calculation module is the resource scheduling algorithm.

[0111] Although over-allocation of resources for virtual nodes has become a common practice in various clusters, it is impossible to completely avoid competition interference during over-allocation between running nodes, which leads to unpredictable service time. In order to cope with sudden demand workloads, the present invention proposes a DDPG-based resource allocation algorithm, called DDPG-RA, which calculates the target resource configuration amount of each virtual node based on the acquired resource information. In the DDPG-RA algorithm, the Actor network is used to generate the corresponding action output. Due to the complexity of multi-resource management, the ratio-based output is converted into the target resource configuration size through an action decoding process. The resource scheduling algorithm of the present invention is introduced in detail below.

[0112] 1. Problem Modeling

[0113] First, a resource management model is established to describe the management problem of CPU, memory, and bandwidth resources of each virtual node in the physical server. Table 2 shows the key symbols used in the algorithm.

[0114] Table 2 Main symbols

[0115]

[0116]

[0117] Define V = {v1,v2,…,v n} is a set of virtual nodes, where n is the number of virtual nodes. For the convenience of explanation, M = {m1, m2, …, m n} is defined as the memory resource configuration set of the virtual node; at the same time, UM={um1,um2,…,um n} is set to the memory utilization of these virtual nodes, where each value ranges from um i ∈[0,1]. Here, the number of vCPUs of each virtual node is used as the adjustment dimension, and C = {c1,c2,…,c n} is the vCPU resource configuration set for each virtual node, and defines UC = {uc1, uc2, ..., uc n} is the CPU utilization set of the virtual node, and each value ranges from uc i ∈[0,1]. In order to satisfy users to enjoy better network service quality, especially when users need high bandwidth, higher bandwidth can be allocated to them through coordination. Similarly, define the set B = {b1,b2,…,b n} represents the bandwidth configuration of the virtual node, and defines UB = {ub1, ub2, …, ub n} is the bandwidth resource utilization rate, and each value ranges from ub i ∈[0,1].

[0118] Define the maximum and minimum values ​​of each resource as mmax i 、cmax i 、bmax i 、mmin i 、cmin i 、bmin i By adjusting the previously unreasonable resource configuration state, resources are fully utilized. Assume that the number of virtual nodes is a constant n and does not change. In order to avoid differences between multiple resources, virtual nodes v i The CPU, memory and bandwidth resource configuration is standardized to cnor i (t), mnor i (t), bnor i (t), the standardized formula is as follows:

[0119]

[0120]

[0121]

[0122] uc i (t),um i (t), ub i (t) The values ​​of these three resource utilizations are all in the range of [0,1], so no normalization is required.

[0123] The system goal is to make full use of resources, so each virtual node should release its underutilized resources and increase its overutilized resources. Here, two thresholds are defined for each resource. For memory utilization, Defined as the lower limit of memory utilization, when the memory utilization is lower than this threshold, it is considered that the memory is not fully utilized and its memory resources should be released; Defined as the upper limit of memory utilization. When the memory utilization is greater than this threshold, the memory on the virtual instance is considered to be overloaded and its memory resources should be increased. Similarly, two sets of thresholds are defined for CPU and bandwidth utilization respectively. and and These thresholds are used to determine whether the CPU and bandwidth are fully utilized or overloaded. Thus, the management system framework will adjust the resources accordingly to manage the virtual nodes within reasonable thresholds. In the present invention, the upper and lower thresholds of the three resources are set to 80% and 20%. Each virtual node should consider the corresponding upper and lower thresholds to avoid overload and underload states.

[0124] Bads is further defined as the number of virtual instances in abnormal states, including resource over-utilization and under-utilization; Vars is defined as the sum of the variances of the three resource utilizations.

[0125]

[0126] Among them, when When x i =0; otherwise x i =1.

[0127] Vars = γ m *var(UM)+γ c *var(UC)+γ b *var(UB) (8)

[0128] Where var(·) represents variance calculation; γ m , γ c and γ b They represent weight coefficients respectively, which can be determined by the user according to the actual situation. The smaller the Vars value, the more balanced the resource utilization of the virtual node. In addition, the variable Evav is defined to represent the evaluation value of each resource management operation. The larger the value, the more recognized it is. Specifically, it is necessary to evaluate the resource adjustment of each virtual node. i The evaluation result is defined as e i , then we can get Through v i The resource adjustment effect is used to obtain the evaluation value, including the evaluation of CPU, memory and bandwidth resource adjustment, and the corresponding evaluation values ​​are defined as as well as Thus we get Next, if Figure 2 As shown in Figure 2, evaluation rules are formulated according to the changes in resource utilization status to obtain as well as The value of Figure 2 Taking the arrows numbered (2) and (5) as examples, they represent the resource utilization from underutilized state to normal state and from normal state to overloaded state, respectively, and the corresponding evaluation values ​​are -0.1 and -10.

[0129] Therefore, the objective function H of the resource management problem can be expressed as:

[0130] minH=λ1Bads+λ2Vars-λ3Evav (9)

[0131] Among them, λ1, λ2 and λ3 represent weight parameters respectively, and users can set the values ​​according to actual needs.

[0132] 2. Problem Analysis

[0133] In the resource management problem, complex dependencies and irregular load changes have not been fully understood to outline resource requirements, so the following is a preliminary analysis of the problem. At time t, the objective function H(t) is as follows:

[0134] H(t)=λ1Bads(t)+λ2Vars(t)-λ3Evav(t) (10)

[0135] Let H(T t ) is the sum of the values ​​of H(t) from time 0 to time t, expressed as follows:

[0136] H(T t )=λ1Bads(T t )+λ2Vars(T t )-λ3Evav(T t ) (11)

[0137] In the formula,

[0138]

[0139]

[0140]

[0141] According to the above formula, we can get:

[0142]

[0143] In summary, we can conclude that the resource management problem obeys a first-order Markov process. Therefore, reinforcement learning algorithms can be used to solve this problem. First, the problem needs to be transformed into a Markov decision problem. t ,A t ,S t+1 ,R t > represents the resource management problem, where S t is the set of all current states, A t is the set of all actions, S t+1 is the set of all next states, R t It is the immediate reward you get after taking an action.

[0144] a. Status: Because memory, CPU, and bandwidth need to be managed comprehensively, memory, CPU, and bandwidth must be included in the status information. In order to make effective decisions, the status set contains each virtual node v i Resource information, including three configuration resources {c i ,m​i ,b i} and the corresponding utilization of the three resources {uc i ,um i ,ub i}. Therefore, the state s at time t t ∈S t (s t ∈R 1*6n ) is defined as:

[0145]

[0146]

[0147] in, Represents a virtual node v i State information at time t; c i (t),m i (t) and b i (t) represents the CPU, memory, and bandwidth resource configuration at time t. These resource configuration information is obtained through initial configuration parameters and configuration actions. i (t),um i (t) and ub i (t) represents the resource utilization of CPU, memory, and bandwidth at time t.

[0148] b. Action: includes operations on three resources, action a at time t t ∈A t (a t ∈R 1*3n ) is defined as follows:

[0149]

[0150]

[0151] in, Represents a virtual node v i The result of the ratio-based action at time t; as well as Respectively represent the relevant ratio values ​​used to adjust the CPU, memory, and bandwidth.

[0152] c. Next state: After executing the corresponding action, the system changes the state from s t Updated to t+1 , get new s t+1 Status information, including resource configuration and resource utilization of each virtual node.

[0153] d. Instant reward: In order to maximize the objective function, the reward function is set to the opposite value of the objective function formula (9), as shown below:

[0154] r t =-λ1Bads-λ2Vars+λ3Evav (20)

[0155] 3. Resource management algorithm based on DDPG

[0156] The present invention aims to achieve full utilization of resources by scheduling previously unreasonable resource configurations, assuming that the number of virtual instances is a constant n and will not change. The present invention designs a DDPG-RA algorithm to manage multiple resources between virtual instances (Algorithm 1), which is designed based on Deep Deterministic Policy Gradient (DDPG). The DDPG-RA algorithm of the present invention combines Actor and Critic networks, uses the Actor network to generate actions, and uses the Critic network to guide the Actor network to generate better actions. The DDPG network consists of four main parts: the current Actor network, the current Critic network, the target Actor network, and the target Critic network.

[0157] a. The current Actor network is based on the current state s t As input, the discrete action a is calculated according to the ε-greedy algorithm t , after executing the action, the next state s is obtained t+1 The DDPG-based algorithm calculates the immediate reward r according to the reward function t , the four-tuple t ,a t ,s t+1 ,r t >Put it into replay memory (experience pool) and define the current Actor network as π θ (S), the output is action a t .

[0158] b. The current Critic network is responsible for the iterative update of the current Actor network, according to s t and a t Calculate the potential reward value. The potential reward value is the weighted sum of the expected values ​​of all future step rewards starting from the current state. Define the current Critic network as Q ω (S,A).

[0159] ​c. The target Actor network is used to update the current Critic network. It usually copies parameters from the current Actor network at a certain period of time. The target Actor network is defined as π′ θ′ (S), which is the s in the replay memory sampling tuple t+1 Calculate a t+1 action; finally, and s t+1 Passed to the target Critic network to calculate the potential reward value.

[0160] d. The target Critic network is used to update the current Critic network. It also periodically copies parameters from the current Critic network and defines the target Critic network as Q′ ω′ (S,A), which calculates a t+1 and t+1 The potential reward, Q′ ω′ (s t+1 ,a t+1 ).

[0161] When training the current Actor network, this is a deterministic strategy, not a predictive strategy like the Critic network. For the same system state, the action generated by the current Actor network should make the potential reward value predicted by the current Critic network as large as possible, so that it can be considered to obtain more rewards. Specifically, the loss function of the current Actor network needs to obtain the maximum potential reward value of the current Critic network. In order to minimize the policy loss function, the opposite of the output of the current Critic network is used as the loss function of the current Actor network:

[0162]

[0163] Among them, Q ω (s,a) is the potential reward function value of the Critic network, Q ω (s t ,a t The specific calculation process of ) is as follows:

[0164] Q ω (s t ,a t )=E[r t +γQ ω (s t+1 ,a t+1 )] (twenty two)

[0165] Among them, γ∈[0,1] represents the loss rate.

[0166] The parameters θ in the current Actor network are sampled from random mini-batch experiences in the replay memory. t ,a t ,s t+1 ,r t >(t∈{1,2,…,X}) to perform policy gradient update as follows:

[0167]

[0168] Specifically, in each training step, θ is updated as follows:

[0169]

[0170] Among them, α π is the learning rate of the current Actor network.

[0171] For the current Critic network training, the mean square error function is used to define its loss function as follows:

[0172] J(ω)=E[(y t -Q ω (s t ,a t )) 2 ] (25)

[0173] Among them, y t Is a target value calculation formula, as shown below:

[0174] y t =r t +γQ′ ω′ (s t+1 ,π′ θ′ (s t+1 )) (26)

[0175] Among them, Q′ ω′ (s t+1 ,π′ θ′ (s t+1 )) is calculated by the target Actor and Critic network.

[0176] The parameter ω in the current Critic network is also sampled from the random mini-batch experience of the replay memory. t ,a t ,s t+1 ,r t >(t∈{1,2,…,X}) to perform gradient update as follows:

[0177] ​​

[0178] During each training process, the ω update process is as follows:

[0179]

[0180] Among them, α Q is the learning rate of the current Critic network.

[0181] Finally, in each episode, the parameters of the target network are updated using the following formula:

[0182] ω′←τω+(1-τ)ω′ (29)

[0183] θ′←τθ+(1-τ)θ′ (30)

[0184] Among them, the momentum factor τ = 0.05.

[0185] The DDPG-RA algorithm directly determines the number of resources that need to be increased or decreased, rather than a fixed-size resource adjustment step. Since the output value of the Actor network using tanh as the activation function is in the range of [-1,1], it cannot directly reflect the meaning of the action. Therefore, a ratio-based action decoding method is proposed. Through action decoding, the size of allocated or recycled resources can be directly obtained, thereby improving allocation efficiency.

[0186] According to formula (9), first, the algorithm aims to eliminate all virtual nodes with resource shortages through resource scheduling, that is, to minimize Bads; second, in order to make full use of resources, the algorithm tries to ensure that the variance of the three resource utilization rates of each virtual node is as small as possible, that is, to minimize Vars; finally, the action is evaluated according to the change in resource utilization status after taking the action, and the resource utilization status of each virtual node v is obtained respectively. i The maximum evaluation value of is minimized, that is, -Evav. The pseudo code of the DDPG-RA algorithm implementation process is shown below.

[0187]

[0188]

[0189] In each episode of the above algorithm 1, the current resource status of the virtual node is first obtained through the information acquisition module (line 5), and the current Actor network generates the corresponding action result (line 7). Since the output value of the Actor network using tanh as the activation function is in the range of [-1,1] and cannot directly reflect the meaning of the action, a ratio-based action decoding method is designed. The action is decoded according to the ratio-based action decoding method, namely Algorithm 2, and the decoded resource configuration action is obtained (line 8). Arema executes the resource configuration action (line 9); next, the resource status s at the next moment is obtained. t+1 , and calculate the reward according to formula (20) (line 10); then t ,a t ,s t+1 ,r t >Stored in replay memory (line 11); When there are no virtual nodes that exceed the normal resource utilization threshold, DDPG-RA will jump out of the current episode (lines 12-14). In order to overcome the correlation problem between empirical data and non-stationary distribution, the sampling training method in replay memory is used to randomly sample from previous state transition experience for training (lines 16-19), so that the empirical data can be fully used and the correlation of consecutive samples can be reduced, avoiding large variance in the training parameter update process.

[0190] In order to increase the randomness of the learning process and increase the coverage of learning, a certain amount of noise is added to the action selected in line 7 The final action generation formula for interacting with the environment is as follows:

[0191]

[0192] Among them, noise The mean is μ and the variance is σ. 2 The normal distribution of Initially, μ = 0 and σ = 0.8 are set. As the number of training episodes increases, the value of σ decreases by 0.05 every 100 episodes until σ = 0.2.

[0193] In Algorithm 1, the action decoding algorithm is used to convert the ratio-based action into an executable action, denoted as da t . t The information contains the specific size of each virtual node resource configuration, and then you can t Perform resource configuration.

[0194]

[0195] ​

[0196] in, Represents a virtual node v i The result of resource allocation actions at time t; as well as Respectively represent the values ​​used to configure CPU, memory, and bandwidth resources.

[0197] In the action decoding process, action a t is a continuous value in [-1,1], and negative and positive values ​​are considered to decrease and increase resources, respectively. The pseudo code of the decoding process is shown below.

[0198]

[0199]

[0200] For memory action decoding, for memory action For example, its memory recovery amount needs to be converted into And the virtual node v i Memory resource recycling This means that the memory resources of the virtual node should be configured as (Line 4). Here is an explanation of the use To convert memory resource recovery amount, instead of In the tanh function, the function value corresponding to a large number of values ​​in the domain is approximately equal to -1. Need to recycle i All memory resources, this is obviously not feasible, especially in the early stages of training, it is very easy to Therefore, it is not recommended to use Convert to v i The amount of memory to reclaim. Afterwards, this will greatly reduce the probability of bad movements and facilitate training.

[0201] For memory actions For example, the reclaimed memory is allocated to these virtual nodes according to the action value. The allocatable memory size M is calculated (line 5), and the sum of the positive action value ratios is obtained (line 7). Then, Arema increases the memory resources of each virtual node according to the ratio between the action values. In other words, Arema will allocate 100% memory resources to each virtual node for each virtual node v. i Configuration In this case, virtual nodes with large action values ​​can obtain more memory, and nodes with small action values ​​will also alleviate resource shortage.

[0202] For CPU action decoding, vCPUs in a virtual node are executed as tasks on physical cores, and their number can exceed the number of physical cores. However, multiprocessors have an unpredictable architecture by causing interference and communication delays between different tasks executed on different cores. If the task scheduling strategy on the multiprocessor is not considered, each vCPU cannot get a fixed time slice on the physical core to execute tasks. Therefore, CPU resources are difficult to quantify and control, and are not suitable for large-scale adjustments. Management is performed at the adjustment granularity of 1 vCPU at a time. More specifically, when the CPU action value When in the range of [-1,-0.2], [-0.2,0.2] and (0.2,1], Arema calculates v i The operations of reducing 1 vCPU, keeping it unchanged, and increasing 1 vCPU are respectively as follows: as well as (Lines 16-22).

[0203] For bandwidth action decoding, bandwidth is allocated by limiting the maximum upload and download bandwidth of each virtual network card. Here, the same bandwidth resources are configured for upload and download bandwidth, and the bandwidth action value converted is (Line 23).

[0204] In addition to action decoding, the safety of actions must be ensured, and malicious behavior should be stopped (for example, releasing almost all memory of a virtual node). In order to ensure the normal operation of each virtual node, a virtual node v i The minimum resource configuration for CPU, memory and bandwidth is set as cmin i , mmin i , bmin i During the adjustment process, each virtual node cannot fall below these minimum resources. Before executing an action, it is necessary to determine how much resources each virtual node can obtain after executing the action. If it falls below the minimum resource or exceeds the maximum resource configuration, the operation is invalid and the minimum or maximum resource amount is selected instead (lines 24-32).

[0205] After calculating these target resource configuration values, Arema performs resource allocation through the execution module. For KVM, resource allocation is performed: the virsh management tool can be used to control the allocation of memory and vCPU, and the Linux TC tool is used to control the maximum communication bandwidth of each virtual network card. The TC tool uses the qdisc-class-filter tree structure to implement hierarchical control of the traffic flowing through each virtual network card through the htb queue and netem. Based on the OVS network architecture, upload traffic flows from the vNIC to the qvb device, and download traffic flows from the qvo device to the tap device. Therefore, the qvb (qvo) device is set to limit the maximum upload (download) bandwidth. For Docker, resource allocation is performed: the docker update configuration tool is used to control the allocation of memory and vCPU through the parameters cpus and memory. Similar to virtual machines, the Linux TC tool is used to allocate bandwidth resources for containers.

[0206] The technical effect of the present invention is further verified through experiments.

[0207] Classic benchmark programs are used to demonstrate the performance of the system of the present invention. Multiple hosts are used to build the OpenStack platform. Each host uses an Intel(R) Xeon(R) Silver 4116 processor with 48 cores, 128GB of memory, and a disk size of 3.3TB. The operating system is Centos-release-7-5x86 64. A nova (KVM) and a nova-docker (Docker) computing node are mainly used in the OpenStack platform. Multiple virtual nodes are created in the KVM and Docker nodes to evaluate various types of resource management including memory, CPU, and bandwidth. The system of the present invention, the Arema system, is written in Python, and its acquisition, computing, and execution modules are deployed together on the computing nodes. Each virtual node is started according to the initial configuration in Table 3.

[0208] Table 3 Initial configuration of virtual machines and containers

[0209]

[0210] Since the goal is to improve the efficiency of virtual nodes by carefully allocating multiple types of resources, the performance is measured by comparing the benchmark program performance and resource utilization. The specific experimental settings are as follows.

[0211] (1) Comparison method

[0212] In order to evaluate the present invention, the system of the present invention is not only compared with the default case where no resource scheduling scheme is used (this default case is called Default), but also compared with other resource management system prototypes in the following KVM and Docker-based technologies: For the comparative test in the virtualization environment based on virtual machines, the algorithm Adares suitable for the KVM environment is compared. [1] For comparison, it uses the context slot machine framework to control the CPU and memory resources of the virtual machine; for the comparative experiment in the container-based virtualization environment, it is compared with Thoth [2] For comparison, this is a dynamic Q-learning resource management system for appropriate resource adjustment for container-based cloud platforms.

[0213] (2) Benchmark Program

[0214] Using DaCapo [3] The test set evaluates performance overhead, which is a Java benchmark suite. It includes different types of test applications, including compute, memory, and disk intensive applications. The embodiment of the present invention selects several benchmark applications to evaluate performance overhead, including memory intensive (i.e., h2), compute intensive (i.e., jython, pmd, jython, avrora, sunflow, fop, xalan, lusearch, batik), and disk intensive (i.e., eclipse, luindex) benchmarks. In all evaluations, each selected DaCapo application is run ten times.

[0215] Httpload is a web server performance testing tool based on Linux platform, which is used to test the throughput and load of web servers. Given a fixed number of concurrent users p and a fixed number of access times f, the network performance can be estimated based on the completion time of the test. In all evaluations, p=50 and f=1000 are set.

[0216] To create a resource contention scenario, several workload applications were run along with the benchmark during the experiment, including:

[0217] Computational load program: This is an infinite loop of continuous addition operations, which uses a non-blocking lock to control the termination of the computing task. Given a number x, it will create the corresponding x processes, each of which will occupy one vCPU. When the main process releases the non-blocking lock, the child process will end the computing task.

[0218] Static Memory Loader: This is also a real memory request test application, given a fixed number of size y consuming memory load, it requests y amount of memory usage.

[0219] Dynamic Memory Loader: In Mono Programs [4] Given a workload range [low, high], Mono works in two phases. In the first phase, Mono initially requests a low size of memory, and then it monotonically increases its memory request to high. In the second phase, Mono gradually decreases its memory request from high to low and releases all workloads. In the experiment, Mono and h2 are run together as a set of test sets to observe the performance of h2 under Mono's dynamic memory workload.

[0220] (3) Algorithm parameter setting

[0221] Some parameters in the DDPG-RA algorithm can be set through empirical analysis or actual needs. Table 4 gives the parameter value settings that appear in equations (8) and (9).

[0222] Table 4 Parameter settings

[0223]

[0224] Although deep reinforcement learning is widely used, its training has always been a problem. The training of neural networks is mainly an exploratory process, and malicious behaviors (such as crashes caused by too little memory allocation) are likely to occur when there is insufficient experience in the early stages. Therefore, a pre-training method is needed to accelerate the learning process and avoid these behaviors as much as possible. However, this remains a challenge. On the one hand, neural networks require a large amount of training data; on the other hand, when the network interacts directly with the real environment, the data collection efficiency will be very low. Therefore, the present invention constructs a load simulator to pre-train the network, and then migrates the model to the actual environment, thereby reducing training costs and improving security. The simulator configures the virtual nodes with the same resource configuration in Table 3. At the same time, in order to fully understand the knowledge of various application loads, 1000 different initial load vectors are generated for each experimental scale, including CPU, memory and bandwidth loads. At the beginning of each episode, a state is randomly selected and the execution of lines 6-16 in Algorithm 1 begins. During the process, the load vector remains unchanged. When there is no virtual node that exceeds the normal resource utilization threshold, the current episode is jumped out. Each time the jump is made, the training model is sampled in the replay memory. The agents used in the subsequent 5 and 15 scale experiments were trained for 50,000 episodes, respectively, for about 30 and 50 hours, respectively. Thus, the model of the present invention can obtain a comprehensive training process and is applicable to various scenarios. Since in the actual system environment, multiple types of applications are executed synchronously, the dependencies are complex, and the resource requirements are irregular, it is difficult to outline the resource requirements. However, from the perspective of machine learning, the model only needs reasonable and experienced initial values, and then interacts with the actual environment, and then continuously trains and updates the experience. In this way, the model integrates the knowledge of the initial simulator and the actual system, thereby reducing the scope and time of random exploration and improving efficiency.

[0225] In the performance experiment, 15 virtual nodes are created to verify whether the present invention can shorten the completion time of the hybrid benchmark program composed of DaCapo and Httpoad. Different benchmark program combinations are deployed simultaneously in all 15 virtual nodes, and each combination in Table 5 runs on a virtual node. The DaCapo program is run 10 times, and the results obtained include the average running time and error bars. The error bars represent the standard deviation of the results of ten runs of the DaCapo program to quantify the difference in its overall performance. According to the initial configuration in Table 3 and the settings of the test program, these 15 virtual instances will compete for the CPU resources of the host.

[0226] Table 5 Experimental test combination information

[0227]

[0228]

[0229] In the KVM virtual machine experiment, Figure 3 The following figure shows the final results of the benchmark test in a KVM environment. Figure 3 It can be concluded that the results of the Default case are the worst. Therefore, we mainly focus on the comparison between the system Arema of the present invention and the comparative solution Adares. Figure 3 In (a), the results of avrora running under Arema of the present invention achieved the highest speed-up, with an improvement of 113.10%. Similarly, when managed by Arema, the results of h2, h2[mono], and eclipse were improved by 89.65%, 55.65%, and 49.38% times, respectively. Sunflow and lusearch were also reduced in completion time by 14.78% and 20.78%, respectively. However, the speed of jython decreased by less than 5%, which is a small performance loss.

[0230] When testing a combination of network-intensive and computation-intensive benchmark programs, Arema, the system of the present invention, achieved a speed improvement of more than 102% for the Httpload program under various computation-intensive programs, compared to Adares. Figure 3 (b). This is mainly because Adares does not perform bandwidth management when dealing with multi-resource management. As for the results of the corresponding computationally intensive benchmarks, such as Figure 3 As shown in (c), compared with Adares, fop and lusearch-fix achieved 87.64% and 72.06% times faster running speed respectively under Arema. pmd and eclipse also achieved 61.13% and 42.91% times faster running speed respectively under Arema. The results of luindex and jython were improved by 29.00% and 26.47% times respectively. In addition, xalan's performance was also slightly improved by 4.65% running speed. This is because Adares ignores the CPU resource competition caused by network-intensive tasks. However, batik's running speed decreased by 4.70% under the Arema scheme.

[0231] In the Docker container experiment, the comparison results are as follows Figure 4 As shown. Figure 4In the results, compared with Thoth, the first two memory-intensive h2-related results in Arema were reduced in completion time by 48.40% and 58.53%, respectively. jython, eclipse, avrora, and sunflow reduced their running time by 22.82%, 20.37%, 19.20%, and 13.87%, respectively. However, the results in lusearch showed a 3.52% performance degradation. As for Figure 4 The results of the combined operation of the network and computing intensive benchmark programs in (b) and (c) show that when various DaCapo programs are running simultaneously, the Arema system of the present invention reduces the running time of Httpload by at least 59.96% compared with the Thoth solution. Figure 4 (b) as shown. Figure 4 (c) shows the results of the corresponding computationally intensive benchmark programs. Compared with the Thoth solution, the system Arema of the present invention reduces the completion time of eclipse, jython, pmd, fop, xalan, batik, lusearch-fix and luindex by 45.58%, 27.53%, 20.87%, 18.99%, 16.68%, 15.75%, 11.50% and 11.05%, respectively.

[0232] Figure 3 and Figure 4 These results shown in verify that the system of the present invention, Arema, can reduce the completion time of different applications as a whole. Compared with Adares and Thoth, Arema reduces the running completion time of each benchmark program by an average of 38.95% and 37.95%, respectively. Although the performance of the application with two virtual nodes is slightly reduced, it is still considered an acceptable result. Because it is impossible to improve the performance of each node in many running virtual instances, the ideal result is to improve the performance of most applications.

[0233] The energy consumption of the system was further analyzed. The system overhead of Arema, the system of the present invention, on CPU and memory was measured. 15 virtual machines and containers were started on the nova and nova-docker computing nodes, respectively, which was called the Default situation, and the CPU and memory utilization were observed within one hour. In addition, Arema was used to adjust the CPU, memory, and bandwidth for all virtual instances once a minute, with the adjustment ranges of [1vCPU, 2vCPU], [1024MB, 2048MB], and [1MB / s, 2MB / s], respectively. The CPU and memory utilization within one hour was also observed. In this way, the CPU and memory overhead of Arema, which performs resource scheduling operations, was evaluated. Figure 5 and Figure 6In the figure, the red lines show the CPU and memory overheads generated when the CPU, memory and bandwidth of virtual machines and container nodes are frequently scheduled, respectively; while the black lines are the CPU and memory overheads of Default. Although there are many unknown system processes that interfere with the results during the observation process, it can be found that Arema has CPU and memory overheads similar to those of KVM or Docker hosts in the Default case, and Arema does not generate a large amount of overhead, which proves the availability of the system Arema of the present invention.

[0234] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

[0235] The documents cited in the present invention are as follows:

[0236] [1]IACano, Optimizing distributed systems using machine learning.Ph.D.dissertation, University of Washington, Seattle, USA, 2019.

[0237] [2] A. Sangpetch, O. Sangpetch, N. Juangmarisakul, and S. Warodom, Thoth: Automatic resource management with machine learning for container-based cloud platform. in CLOSER, 2017, pp.75–83.

[0238] [3]S.M.Blackburn,R.Garner,C.Hoffmann,A.M.Khang,K.S.McKinley,R.Bentzur,A.Diwan,D.Feinberg,D.Frampton,S.Z.Guyer et al.,The dacapobenchmarks:Java benchmarking development and analysis,in Proceedings of the21st annual ACM SIGPLAN conference on Object-oriented programming systems,languages,and applications,2006,pp.169–190.

[0239] [4]W.Zhao,Z.Wang,and Y.Luo,Dynamic memory balancing for virtualmachines,ACM SIGOPS Operating Systems Review,vol.43,no.3,pp.37–47,2009.

Claims

1. A resource management method in a cloud computing environment, characterized in that: The following steps are involved: Step 1: Collect resource information of each virtual node in real time, including CPU usage time, free memory and bandwidth information; Step 2: Obtain the CPU, memory and bandwidth resource configuration of each virtual node; Step 3: Calculate and extract based on the resource information and the CPU, memory and bandwidth resource configuration amounts to obtain the CPU, memory and bandwidth usage rates of each virtual node; the CPU, memory and bandwidth resource configuration amounts and the CPU, memory and bandwidth usage rates together constitute a state set; Step 4: Use a resource management algorithm based on deep deterministic policy gradient to train and learn the state set to obtain resource configuration decisions for each virtual node; The specific steps include: Step 41: Initialize the current Actor network π θ (S), current critic network Q ω (S,A), target Actor network π θ ″(S) and target critic network Q′ ω′ (S,A), and create an experience pool; Step 42: Initialize the environment and obtain the current state information at time t; Step 43: The current state s at time t is concentrated in the state t As input, according to the Actor network π θ (S) Calculate the discrete action and add noise to obtain the action a at the current time t t : In the formula, π θ (s t ) represents the Actor network at the current time t; It means that the mean is μ and the variance is σ 2 Normally distributed noise; Step 44: According to the current state s at time t t The action a at the current time t is decoded using a ratio-based action decoding algorithm. t Decode and obtain the decoded resource configuration action; Step 45: Perform resource configuration according to the decoded resource configuration action; Step 46: Get the state s at time t+1 in the state set t+1 , and calculate the instant reward r according to the following reward function formula t : r t =-λ1Bads-λ2Vars+λ3Evav In the formula, λ1, λ2, and λ3 represent weight parameters; Bads represents the number of virtual instances in the state of over-utilization or under-utilization of resources; Vars represents the sum of the variances of CPU, memory, and bandwidth usage. The smaller the value, the more balanced the resource utilization of the virtual node; Evav represents the evaluation value of each resource management action operation. The larger the value, the more recognized it is. Step 47: Set the current state s t , action a at time t t , state s at time t+1 t+1 and instant rewards t That is t ,a t ,s t+1 ,r t Deposit into experience pool; Step 48: Sample a mini-batch from the experience pool and perform a gradient update on the parameter ω in the critic network at the current time t based on the following formula: In the formula, E represents the expected value; Q ω (s t ,a t ) represents the Critic network at the current time t; Represents the derivative of the parameter ω of the current Critic network; y t represents the target value, y t =r t +γQ′ ω′ (s t+1 ,π′ θ′ (s t+1 )), γ represents the loss rate, Q′ ω′ (s t+1 ,π θ ″(s t+1 )) represents the target Critic network at time t+1, π θ ″(s t+1 ) represents the target Actor network at time t+1; Step 49: Sample a mini-batch from the experience pool and perform a gradient update on the parameter θ in the Actor network at the current time t based on the following formula: In the formula, Indicates the derivation of the parameter a of the current Critic network; Represents the derivative of the parameters θ of the current Actor network; Step 40: Update the parameter θ′ of the target Actor network by the following formula: θ′←τθ+(1-τ)θ′; Update the parameter ω′ of the target Critic network by the following formula: ω′←τω+(1-τ)ω′; where τ represents the momentum factor; Step 41, iteratively executing step 42 to step 40 until the preset maximum number of iterations is reached and then the iterative execution is stopped; Step 5: Dynamically adjust the resources of each virtual node according to the resource allocation decision.

2. The resource management method in a cloud computing environment according to claim 1, characterized in that: CPU usage uc in step 3 i 、Memory usage um i and bandwidth usage ub i The calculation formulas are: Where cpu(t) represents the CPU usage time at time t; c i Indicates the amount of CPU resource configuration; f i Indicates free memory; m i represents the amount of memory resource configuration; recv(t) represents the number of bandwidth packets received at time t; trans(t) represents the number of bandwidth packets sent at time t; b i Indicates the bandwidth resource configuration amount.

3. The resource management method in a cloud computing environment according to claim 2, characterized in that: The specific steps of step 44 include: Step 441: Decode the memory action of the i-th virtual node: a. For memory action value The corresponding memory resource configuration is: b. For memory action value Calculate the amount of memory that can be allocated And obtain the sum of the positive action value ratio, then the corresponding memory resource configuration is: Step 442: Decode the memory action of the i-th virtual node: CPU Action Value When the value is in the range of [-1, -0.2), [-0.2, 0.2], and (0.2, 1], it corresponds to the following CPU resource configuration amounts: the CPU resource configuration amount decreases by 1, the CPU resource configuration amount remains unchanged, and the CPU resource configuration amount increases by 1. The corresponding action results are: Step 443: Decode the bandwidth action of the i-th virtual node: Bandwidth Action Value The corresponding bandwidth resource configuration is:

4. The resource management method in a cloud computing environment according to claim 3, characterized in that: Step 443 also includes step 444: after determining the resource configuration amount obtained by the virtual node after executing the resource configuration action, if the resource configuration amount is lower than the preset minimum resource configuration amount or exceeds the preset maximum resource configuration amount, the resource configuration action is changed to the corresponding preset minimum resource configuration amount or preset maximum resource configuration amount.

5. A resource management system in a cloud computing environment, characterized in that: It includes information collection module, resource calculation module and resource adjustment module; among them, The information collection module is used to collect resource information of each virtual node in real time, the resource information includes CPU usage time, free memory and bandwidth information; obtain the CPU, memory and bandwidth resource configuration of each virtual node; The resource calculation module is used to calculate and extract according to the resource information and the CPU, memory and bandwidth resource configuration amounts to obtain the CPU, memory and bandwidth usage of each virtual node; the CPU, memory and bandwidth resource configuration amounts and the CPU, memory and bandwidth usage together constitute a state set; the state set is trained and learned using a resource management algorithm based on deep deterministic policy gradient to obtain a resource configuration decision for each virtual node; the specific steps include: Step 41: Initialize the current Actor network π θ (S), current critic network Q ω (S,A), target Actor network π θ ″(S) and target critic network Q′ ω′ (S,A), and create an experience pool; Step 42: Initialize the environment and obtain the current state information at time t; Step 43: The current state s at time t is concentrated in the state t As input, according to the Actor network π θ (S) Calculate the discrete action and add noise to obtain the action a at the current time t t : In the formula, π θ (s t ) represents the Actor network at the current time t; It means that the mean is μ and the variance is σ 2 Normally distributed noise; Step 44: According to the current state s at time t t The action a at the current time t is decoded using a ratio-based action decoding algorithm. t Decode and obtain the decoded resource configuration action; Step 45: Perform resource configuration according to the decoded resource configuration action; Step 46: Get the state s at time t+1 in the state set t+1 , and calculate the instant reward r according to the following reward function formula t : r t =-λ1Bads-λ2Vars+λ3Evav In the formula, λ1, λ2, and λ3 represent weight parameters; Bads represents the number of virtual instances in the state of over-utilization or under-utilization of resources; Vars represents the sum of the variances of CPU, memory, and bandwidth usage. The smaller the value, the more balanced the resource utilization of the virtual node; Evav represents the evaluation value of each resource management action operation. The larger the value, the more recognized it is. Step 47: Set the current state s t , action a at time t t , state s at time t+1 t+1 and instant rewards t That is t ,a t ,s t+1 ,r t Deposit into experience pool; Step 48: Sample a mini-batch from the experience pool and perform a gradient update on the parameter ω in the critic network at the current time t based on the following formula: In the formula, E represents the expected value; Q ω (s t ,a t ) represents the Critic network at the current time t; Represents the derivative of the parameter ω of the current Critic network; y t represents the target value, y t =r t +γQ′ ω′ (s t+1 ,π θ ″(s t+1 )), γ represents the loss rate, Q′ ω′ (s t+1 ,π θ ″(s t+1 )) represents the target Critic network at time t+1, π θ ″(s t+1 ) represents the target Actor network at time t+1; Step 49: Sample a mini-batch from the experience pool and perform a gradient update on the parameter θ in the Actor network at the current time t based on the following formula: In the formula, Indicates the derivation of the parameter a of the current Critic network; Represents the derivative of the parameters θ of the current Actor network; Step 40: Update the parameter θ′ of the target Actor network by the following formula: θ′←τθ+(1-τ)θ′; Update the parameter ω′ of the target Critic network by the following formula: ω′←τω+(1-τ)ω′; where τ represents the momentum factor; Step 41, iteratively executing step 42 to step 40 until the preset maximum number of iterations is reached and then the iterative execution is stopped; The resource adjustment module is used to dynamically adjust the resources of each virtual node according to the resource configuration decision.

6. The resource management system in a cloud computing environment according to claim 5, characterized in that: The CPU usage uc in the resource calculation module i 、Memory usage um i and bandwidth usage ub i The calculation formulas are: Where cpu(t) represents the CPU usage time at time t; c i Indicates the amount of CPU resource configuration; f i Indicates free memory; m i represents the amount of memory resource configuration; recv(t) represents the number of bandwidth packets received at time t; trans(t) represents the number of bandwidth packets sent at time t; b i Indicates the bandwidth resource configuration amount.

7. A resource management system in a cloud computing environment according to claim 6, characterized in that: The resource calculation module is based on the current state s at time t. t The action a at the current time t is decoded using a ratio-based action decoding algorithm. t The specific steps of decoding and obtaining the decoded resource configuration action include: Step 441: Decode the memory action of the i-th virtual node: a. For memory action value The corresponding memory resource configuration is: b. For memory action value Calculate the amount of memory that can be allocated And obtain the sum of the positive action value ratio, then the corresponding memory resource configuration is: Step 442: Decode the memory action of the i-th virtual node: CPU Action Value When the value is in the range of [-1, -0.2), [-0.2, 0.2], and (0.2, 1], it corresponds to the following CPU resource configuration amounts: the CPU resource configuration amount decreases by 1, the CPU resource configuration amount remains unchanged, and the CPU resource configuration amount increases by 1. The corresponding action results are: Step 443: Decode the bandwidth action of the i-th virtual node: Bandwidth Action Value The corresponding bandwidth resource configuration is: Step 444: After determining the resource configuration amount obtained by the virtual node after executing the resource configuration action, if the resource configuration amount is lower than the preset minimum resource configuration amount or exceeds the preset maximum resource configuration amount, the resource configuration action is changed to the corresponding preset minimum resource configuration amount or preset maximum resource configuration amount.

Images