Method and device for virtual machine relocation in heterogeneous cluster based on scalable reinforcement learning

Abstract

The application discloses a heterogeneous cluster virtual machine relocation method based on scale-variable reinforcement learning, which comprises the following steps: firstly, collecting cluster runtime state and service quality data, and then updating a host mask vector through a virtual machine relocation rule; on this basis, a feature representation method of a heterogeneous host type embedding is established to reflect the similarity and difference between hosts; then, a feature extraction network of a host historical utilization rate sequence is constructed to capture the dynamic change of resource utilization; finally, a reinforcement learning base model is designed to realize the scale-variable virtual machine relocation method, when the cluster scale changes due to host downtime or maintenance, the mask mechanism is used to limit the virtual machine relocation range, and the effectiveness and stability of resource scheduling are ensured. The application realizes a flexible and efficient virtual machine relocation strategy according to the characteristics of a heterogeneous cluster.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud resource scheduling, and in particular relates to a method and device for relocating virtual machines in a heterogeneous cluster based on scalable reinforcement learning. Background Art

[0002] Cloud data centers are the infrastructure for cloud computing, providing users with a variety of computing services. Due to their large scale, cloud data centers consume a significant amount of energy, placing significant pressure on the environment and the economy. To reduce energy consumption in cloud data centers, a common approach is virtual machine relocation. This involves migrating virtual machines from one host to another, consolidating and load balancing virtual machines, thereby reducing the number of hosts and power consumption. However, virtual machine relocation also faces challenges and drawbacks. For example, resource competition between virtual machines can lead to degraded service quality, virtual machine migration incurs additional overhead, and virtual machine relocation decisions must consider multiple objectives and constraints.

[0003] To address the problem of VM relocation, existing methods are primarily categorized into two types: rule-based and learning-based. Rule-based methods relocate VMs according to pre-defined relocation rules or heuristic algorithms. While these methods are simple and easy to implement, they lack the ability to adapt to dynamic changes in cloud workloads and require manual adjustment of parameters and thresholds. Learning-based methods utilize techniques such as machine learning or reinforcement learning to automatically learn relocation strategies. While these methods are adaptable to complex environments, they require extensive training data and computing resources, and struggle to handle high-dimensional state and action spaces. Furthermore, neither rule-based nor learning-based methods fully consider the heterogeneity and scalability of cloud computing clusters. Hosts in a cluster may have varying hardware configurations and computing power, and may dynamically join or leave the cluster due to maintenance, failures, or capacity expansion. These factors can impact the effectiveness and efficiency of VM relocation. Summary of the Invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning to solve the problem that the state space dimension changes caused by the high heterogeneity of existing cluster hosts and the change of cluster scale cannot be applied to fixed-dimensional neural networks.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] In a first aspect, the present invention provides a method for relocating virtual machines in a heterogeneous cluster based on scalable reinforcement learning, comprising the following steps:

[0007] Collect cluster runtime information, including host CPU utilization and energy consumption data, and virtual machine quality of service violation rate; the virtual machine quality of service violation rate is the ratio of the time the virtual machine experiences insufficient resources to the total active time;

[0008] According to the cluster status and virtual machine relocation rules, the host mask vector is updated; the host mask vector at time t is represented as Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i =0 indicates that a virtual machine can be placed; the length of the host mask vector is equal to the number of hosts in the cluster, and the order is consistent with the order of the hosts in the cluster;

[0009] Establish a state feature representation method for embedding heterogeneous host types, mapping different types of hosts in the cluster into the same space to reflect the similarities and differences between hosts;

[0010] Establish a feature extraction network for the host's historical utilization sequence and convert each host's CPU utilization sequence into a fixed-length feature vector;

[0011] A reinforcement learning-based model for solving the virtual machine scheduling problem is established. The model inputs are the cluster state vector, host mask vector, and host type embedding, and the model output is the action vector for virtual machine relocation.

[0012] A scalable virtual machine relocation method is established, wherein the virtual machine relocation method is to migrate virtual machines on overloaded or underloaded hosts according to the state of the cluster and the virtual machine relocation rules to achieve the purpose of minimizing energy consumption and maximizing service quality.

[0013] As a preferred technical solution, the collection of cluster runtime information is specifically as follows:

[0014] For a cluster with N hosts, define the cluster utilization at time t where h i,t =(u i,t-k+1 ,…,u i,t ) represents the CPU utilization sequence of host i in the past k moments, u i,t represents the CPU utilization of host i at time t; the cluster energy consumption at time t is defined as where e i,t Indicates the current energy consumption value of host i; in addition, defines the virtual machine service quality at time t Where N is the total number of hosts in the cluster, n i is the number of virtual machines on host i, and They represent the insufficient resource demand time and total active time of virtual machine j at time t respectively.

[0015] As a preferred technical solution, the virtual machine relocation rules include:

[0016] (1) An overloaded host cannot host any more virtual machines;

[0017] (2) VMs on an underloaded host can only be migrated to other non-overloaded hosts or powered-off hosts of different types;

[0018] (3) The host that is down or under maintenance cannot host any more virtual machines; the host mask vector at time t is represented by Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i =0 means that a virtual machine can be placed.

[0019] As a preferred technical solution, the method for establishing a state feature representation method for embedding heterogeneous host types is specifically as follows:

[0020] For a cluster with k types of heterogeneous hosts, the heterogeneous host types are embedded as Among them G i,k is the embedding vector representation of the k-th type of host, and d is the feature dimension of the host type embedding.

[0021] As a preferred technical solution, the feature extraction network for establishing the host historical utilization sequence is specifically as follows:

[0022] The characteristics of the host historical utilization sequence are obtained through a recursive neural network. The recursive neural network part adopts a standard RNN network structure, and its input is the historical utilization sequence of N hosts. Among them, the historical utilization sequence of each host is h i,t It is input into the RNN network structure in parallel. At time step t, the hidden state of the RNN output is represented as the feature of the host utilization sequence and serves as part of the input of the reinforcement learning model for further virtual machine scheduling decisions.

[0023] As a preferred technical solution, the establishment of a reinforcement learning basic model for solving the virtual machine scheduling problem is specifically as follows:

[0024] The reinforcement learning basic model for solving the virtual machine scheduling problem includes state space, action space and reward function; the designed state space includes the historical utilization sequence of each host in the cluster, the heterogeneous host type embedding and the host mask vector, which is specifically represented as S t =[H t,Mask t ,Embedding]; the designed action space allows the selection of the target host for virtual machine placement. There are N possible choices. The action space is represented by A = [1, 0, 0, ..., 0], which means that the first host is selected as the placement object. The dimension of the action space A is N. The reward function consists of two parts: energy consumption reward and service quality reward. The energy consumption reward is represented by r energy =E t-1 -E t , the service quality reward is expressed as r sla =Q t-1 -Q t , the final reward is expressed as r = αr energy +βr sla , where α and β are the weights of energy consumption reward and service quality reward respectively.

[0025] As a preferred technical solution, the method for establishing a scalable virtual machine relocation method is specifically as follows:

[0026] The virtual machine relocation method includes a virtual machine placement method based on reinforcement learning, a virtual machine selection method and a host status determination method;

[0027] For the VM placement method based on reinforcement learning, a reinforcement learning algorithm is used to select an optimal target host for each VM that needs to be relocated based on the cluster's state vector and host mask vector to maximize the cumulative reward;

[0028] For the virtual machine selection method, the system first calculates the CPU utilization and computing capacity corresponding to the optimal energy efficiency of the overloaded host, expressed in millions of operations per second (MIPS). Then, it traverses all virtual machines on the overloaded host and calculates the difference between the host utilization after migration and the optimal energy efficiency utilization for each virtual machine. The result is stored. Then, the virtual machine with the smallest difference is selected as the migration target. The above process is repeated until the host is no longer overloaded.

[0029] For host status determination, first, set the overload host threshold, that is, the host with a current utilization rate exceeding 90% is considered an overloaded host and requires virtual machine relocation; then the underload host threshold is set to L = quantile 1 / 4 (u 1,t ,…,u N,t ), which is the 1 / 4 quantile of the current utilization of N hosts. Hosts with utilization lower than L are considered underloaded and need to be shut down to save energy;

[0030] Next, the status of all hosts is determined. Based on the comparison between utilization and threshold, the hosts are classified into three states: overloaded, normal, or underloaded.

[0031] Finally, the virtual machines that need to be migrated are determined from the overloaded and underloaded hosts, and then reinforcement learning is used to complete the relocation of the virtual machines. After the virtual machines are migrated, the underloaded hosts are shut down, and no operation is performed on the normal hosts.

[0032] In a second aspect, the present invention provides a heterogeneous cluster virtual machine relocation system based on scalable reinforcement learning, which is applied to the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning, including an information acquisition module, a mask vector update module, a state feature representation module, a feature extraction module, a reinforcement learning module and a relocation module;

[0033] The information collection module is used to collect cluster runtime information, including the CPU utilization and energy consumption data of the host and the service quality violation rate of the virtual machine; the service quality violation rate of the virtual machine is the ratio of the time when the virtual machine experiences insufficient resources to the total active time; the mask vector update module is used to update the host mask vector according to the cluster status and the virtual machine relocation rule; the host mask vector at time t is represented by Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i =0 indicates that a virtual machine can be placed; the length of the host mask vector is equal to the number of hosts in the cluster, and the order is consistent with the order of the hosts in the cluster;

[0034] The state feature representation module is about establishing a state feature representation method for embedding heterogeneous host types, mapping different types of hosts in the cluster into the same space to reflect the similarities and differences between the hosts;

[0035] The feature extraction module is used to establish a feature extraction network for the host historical utilization sequence and convert the CPU utilization sequence of each host into a feature vector of fixed length;

[0036] The reinforcement learning module is used to establish a reinforcement learning basic model for solving the virtual machine scheduling problem. The input of the model is the cluster state vector, the host mask vector and the host type embedding. The output of the model is the action vector of the virtual machine relocation;

[0037] The relocation module is used to establish a scalable virtual machine relocation method. The virtual machine relocation method is to migrate virtual machines on overloaded or underloaded hosts according to the cluster status and virtual machine relocation rules to achieve the purpose of minimizing energy consumption and maximizing service quality.

[0038] In a third aspect, the present invention provides an electronic device comprising:

[0039] at least one processor; and,

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning.

[0043] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0044] (1) The innovation of the present invention is to use host type embedding and recursive neural networks to encode the state and heterogeneous characteristics of the cluster into a state vector, then use Mask to process cluster scale changes, and then use deep reinforcement learning to select appropriate virtual machines for relocation based on the state vector, thereby achieving dual-objective optimization of minimizing energy consumption and maximizing service quality. The social benefit of the present invention is that it can reduce the operating costs and environmental impact of cloud data centers and improve user satisfaction and competitiveness of cloud computing services. The technical contribution of the present invention is to propose a virtual machine relocation method that adapts to the heterogeneity and scale changes of cloud computing clusters, effectively solving the problems existing in existing methods and improving the performance and scalability of virtual machine relocation.

[0045] (2) By leveraging host type embedding and recurrent neural networks, this invention can more accurately capture the heterogeneity and dynamic changes of cloud computing clusters. Compared to traditional rule-based methods, this invention eliminates the need for manual setting and adjustment of complex parameters and thresholds. Instead, it uses deep reinforcement learning technology to enable the system to automatically learn and adapt to different load changes, thereby achieving higher adaptability.

[0046] (3) The present invention uses Mask to handle cluster size changes, effectively resolving the challenges faced by existing methods when handling dynamic host additions or exits. Furthermore, because the method of the present invention has online learning capabilities, it exhibits strong robustness against various uncertainties and interference factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 This is a flow chart of a method for relocating virtual machines in a heterogeneous cluster based on scalable reinforcement learning according to an embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of relocating virtual machines in a heterogeneous cluster according to an embodiment of the present invention;

[0050] Figure 3 This is a diagram of a virtual machine relocation framework based on reinforcement learning according to an embodiment of the present invention;

[0051] Figure 4 This is a graph showing the change in the number of active hosts at different times of the day using the embodiment of the present invention and the heuristic comparison algorithm;

[0052] Figure 5 This is a graph showing energy consumption changes under different cluster sizes according to an embodiment of the present invention;

[0053] Figure 6 This is a graph showing changes in the quality of service violation rate under different cluster sizes according to an embodiment of the present invention;

[0054] Figure 7 This is a block diagram of a heterogeneous cluster virtual machine relocation system based on scalable reinforcement learning according to an embodiment of the present invention.

[0055] Figure 8 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0057] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0058] like Figure 1 As shown, the method for relocating virtual machines in a heterogeneous cluster based on scalable reinforcement learning in this embodiment includes the following steps:

[0059] Step 1: Cluster runtime information collection. Install a monitoring component on each host to collect CPU utilization and energy consumption data, as well as the VM's quality of service violation rate. This rate is calculated using the VM's resource requirements and allocated resources. The VM's quality of service violation rate is the ratio of the time the VM experiences resource shortage to its total active time. Time when the requested resources are less than the allocated resources is considered resource shortage time.

[0060] Furthermore, in this embodiment, the monitoring component uses the open source tool Prometheus to regularly send data to the distributed database InfluxDB; ipmitool is installed on the server and the Prometheus server job configuration is added to obtain the hardware monitoring capability; the cluster utilization collected at time t is is the CPU utilization sequence of host i in the past k moments; the cluster energy consumption collected at time t is where e i,t represents the energy consumption value of host i at time t; the virtual machine service quality at time t Where M is the total number of hosts in the cluster, n i is the number of virtual machines on host i, and They represent the insufficient resource demand time and total active time of virtual machine j at time t respectively.

[0061] Step 2: Figure 2 As shown in Figure 2, virtual machine relocation requires determining a legitimate target host. Setting relocation rules can avoid unnecessary exploration and improve algorithm performance. The host mask vector is updated based on the cluster status and virtual machine relocation rules.

[0062] Furthermore, the virtual machine relocation rules include: (1) overloaded hosts cannot place virtual machines; (2) virtual machines on underloaded hosts can only be migrated to other non-overloaded hosts or different types of powered-off hosts; (3) downtime or maintenance hosts cannot place virtual machines; the host mask vector at time t is represented by Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i =0 means that a virtual machine can be placed.

[0063] It is understandable that the virtual machine relocation rules are customized and adjusted according to specific application scenarios and requirements. For example, if the CPU utilization of the host exceeds the threshold, then the host cannot be used as the target host for virtual machine migration; if the host energy efficiency is lower than the energy efficiency of the host where the virtual machine to be migrated is located, then the host cannot be used as the target host; if the host is in maintenance / downtime state, then the host cannot be used as the target host; before each virtual machine scheduling, all hosts are traversed according to the relocation rules and the host mask vector that can be placed is determined. The host mask vector at time t is represented as Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i = 0 indicates that the virtual machine can be placed. Its length is equal to the number of hosts in the cluster, and its order is consistent with the order of the hosts in the cluster. For example, if there are 5 hosts in the cluster, and the 1st and 3rd hosts can be the target hosts for the virtual machine, then the host mask vector is [1, 0, 1, 0, 0].

[0064] Step 3: Using the state feature representation method of heterogeneous host type embedding, different types of hosts in the cluster are mapped into the same space to reflect the similarities and differences between the hosts. The heterogeneous host type embedding is a real number vector of length d that represents the type characteristics of the host.

[0065] Furthermore, the method for generating heterogeneous host type embedding is as follows: i) Classify all hosts in the cluster according to their processor types to obtain K host categories, each category containing several hosts; ii) For each host category, randomly initialize a real number vector of length d as its type embedding vector; iii) For each host, assign the corresponding type embedding vector according to the category it belongs to. Let the host type embedding at time t be Embedding = [G 1,1 ,G 2,j ,…,G N,k ], G i,kEmbedding vector for the type of host i.

[0066] Step 4: Use the feature extraction network of the host's historical utilization sequence to convert the CPU utilization sequence of each host into a fixed-length feature vector.

[0067] Furthermore, the interval of the host's CPU historical utilization sequence is customized according to the scenario and requirements. For example, the host's CPU utilization is collected every 5 minutes, and then these data are connected into a one-dimensional real number sequence h i,t =(u i,t-k+1 ,…,u i,t This sequence can reflect the host's load characteristics, such as peak value, average value, and fluctuation range. A recurrent neural network (RNN) is then used to convert the input time series data into a real number vector of a specific length as a feature representation of the host's utilization status. The RNN network has 1 neuron at the input, d neurons at the hidden layer, and d / 2 neurons at the output. These neurons recursively calculate k times and output sequence features of dimension d / 2. This vector can reflect the host's load characteristics and can reflect the host's periodicity and trend.

[0068] Step 5: Figure 3 As shown, using the reinforcement learning basic model, the cluster state vector, host mask vector and host type embedding S are input. t =[H t ,Mask t ,Embedding], output the action vector A = [1,0,0,…,0] of the virtual machine relocation. Define the goal of reinforcement learning, which is to maximize the cumulative reward, that is, to minimize the energy consumption and maximize the service quality of the relocated cluster within a certain period of time. The cumulative reward refers to the discounted sum of the rewards after each relocation, that is, r = r1 + γr2 + γ 2 r3+…, where r1, r2, r3,… are the rewards after each relocation, and γ is a discount factor between 0 and 1, which is used to adjust the importance of future rewards; the reward is set to r = αr energy +βr sla , where α and β are the weights of energy consumption reward and service quality reward respectively, and energy consumption reward r energy =E t-1 -E t , service quality reward r sla =Q t-1 -Q t The basic algorithm of reinforcement learning adopts the SAC algorithm. In addition to the conventional optimization objectives, the SAC algorithm also takes action entropy as the optimization objective. Compared with algorithms such as DQN and PPO, it has a faster convergence speed. The parameters of the algorithm are shown in Table 1.

[0069] Table 1 Algorithm parameters

[0070]

[0071]

[0072] In step six, the virtual machine relocation method refers to migrating virtual machines on overloaded or underloaded hosts according to the cluster status and virtual machine relocation rules, so as to minimize energy consumption and maximize service quality.

[0073] Furthermore, the virtual machine relocation method includes three sub-steps: a virtual machine placement method based on reinforcement learning, a virtual machine selection method, and a host status determination method.

[0074] Furthermore, the reinforcement learning-based VM placement method uses a reinforcement learning algorithm to select an optimal target host for each VM to be relocated based on the cluster's state vector and host mask vector to maximize the cumulative reward. This method is described in step 5.

[0075] Furthermore, the virtual machine selection method is to select one or more virtual machines for migration for each overloaded host based on the CPU utilization corresponding to its optimal energy efficiency value and the host's computing capacity, so that the utilization of the overloaded host is reduced to near the optimal energy efficiency utilization. The specific process of this method is as follows: For each overloaded host, obtain the CPU utilization corresponding to its optimal energy efficiency value, denoted as u * , and the host's computing capacity, denoted as C, where computing capacity is expressed in millions of instructions per second (MIPS). This information can be obtained from the host's specifications or monitoring tools and updated regularly. All virtual machines on the overloaded host are traversed. For each virtual machine, the difference between the utilization rate of the overloaded host and the optimal energy efficiency utilization rate after it is migrated from the overloaded host is calculated and stored, denoted as d, that is, d = |u * -(um / C)|, where u is the current utilization of the overloaded host and m is the computing demand of the virtual machine, expressed in MIPS. Note that the calculation here assumes that the host utilization after the virtual machine is migrated is equal to the optimal energy efficiency utilization u * The virtual machine with the smallest difference (d) is selected from all virtual machines as the migration target, i.e., d = min(d). It is added to the list of virtual machines to be relocated, and the utilization of the overloaded host is updated, i.e., u = um / C. This process is repeated until the utilization of the overloaded host is less than or equal to the overloaded host threshold U, at which point the virtual machine selection process stops.

[0076] Furthermore, the host status determination method is to classify the host into three states: overloaded, normal, and underloaded according to the host's CPU utilization and a preset threshold, so as to perform the corresponding virtual machine relocation operation. The specific process of this method is as follows: set the overload host threshold U = 0.9, indicating that the host with a current utilization greater than 90% is an overloaded host and needs to be relocated; then set the underload host threshold to L, where L = quantile 1 / 4 (u 1,t ,…,u N,t ) represents the 1 / 4 quantile of the current utilization of N hosts. When the utilization of a host is lower than L, it is considered to be an underloaded host and needs to be shut down to save energy. Then, the status of all hosts is determined, and the host utilization is compared with the threshold to classify the hosts into three states: overloaded, normal, and underloaded. Finally, the virtual machine selection method and the reinforcement learning-based virtual machine placement method are executed for overloaded hosts, the virtual machine migration method and the host shutdown method are executed for underloaded hosts, and no operation is performed on normal hosts.

[0077] Finally, we used the CloudSim simulation method to simulate virtual machine requests using PlanetLab's real load data and used Poisson distribution to generate virtual machine requests. Through this simulation method, we can evaluate and compare the performance and effects of reinforcement learning without affecting the real cluster. The simulation environment parameters are shown in Table 2.

[0078] Table 2 Simulation environment parameters

[0079] parameter value Number of hosts [20,100,200] Number of virtual machines [100,500,1000] Task arrival rate that satisfies Poisson distribution 10s Mission length 2000 Host overload threshold 0.9 Simulation data Planetlab-20110303

[0080] Figure 4 This paper presents the number of active hosts at each moment using different methods for the same cluster size. We selected three of the most commonly used algorithms for comparison. The results show that these algorithms tend to frequently change host states, a consequence of their inability to account for dynamic environments and heterogeneity between hosts. In contrast, our proposed reinforcement learning-based approach maintains a minimum number of active hosts and optimizes long-term returns. Figure 5 and Figure 6 The changes in energy consumption and service quality of the method proposed in this invention under clusters of different sizes are demonstrated. The results show that under different cluster sizes, the reinforcement learning-based method can maintain a low energy consumption and service quality violation rate.

[0081] Based on the same concept as the scalable reinforcement learning-based heterogeneous cluster virtual machine relocation method described in the above-mentioned embodiment, the present invention also provides a scalable reinforcement learning-based heterogeneous cluster virtual machine relocation system, which can be used to execute the above-mentioned scalable reinforcement learning-based heterogeneous cluster virtual machine relocation method. For ease of explanation, the structural diagram of the embodiment of the scalable reinforcement learning-based heterogeneous cluster virtual machine relocation system only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and the device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0082] See also Figure 7 In another embodiment of the present application, a heterogeneous cluster virtual machine relocation system 100 based on scalable reinforcement learning is provided, the system comprising an information acquisition module 101, a mask vector update module 102, a state feature representation module 103, a feature extraction module 104, a reinforcement learning module 105 and a relocation module 106;

[0083] The information collection module 101 is used to collect cluster runtime information, including the CPU utilization and energy consumption data of the host, and the service quality violation rate of the virtual machine; the service quality violation rate of the virtual machine is the ratio of the time when the virtual machine experiences insufficient resources to the total active time; the mask vector update module 102 is used to update the host mask vector according to the cluster status and the virtual machine relocation rule; the host mask vector at time t is represented by Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i =0 indicates that a virtual machine can be placed; the length of the host mask vector is equal to the number of hosts in the cluster, and the order is consistent with the order of the hosts in the cluster;

[0084] The state feature representation module 103 is about establishing a state feature representation method for embedding heterogeneous host types, mapping different types of hosts in the cluster into the same space to reflect the similarities and differences between the hosts;

[0085] The feature extraction module 104 is used to establish a feature extraction network for the host historical utilization sequence and convert the CPU utilization sequence of each host into a feature vector of fixed length;

[0086] The reinforcement learning module 105 is used to establish a reinforcement learning basic model for solving the virtual machine scheduling problem. The input of the model is the cluster state vector, the host mask vector and the host type embedding. The output of the model is the action vector of the virtual machine relocation.

[0087] The relocation module 106 is used to establish a scalable virtual machine relocation method, which is to migrate virtual machines on overloaded or underloaded hosts according to the cluster status and virtual machine relocation rules to achieve the purpose of minimizing energy consumption and maximizing service quality.

[0088] It should be noted that the heterogeneous cluster virtual machine relocation system based on scalable reinforcement learning of the present invention corresponds one-to-one to the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning are applicable to the embodiment of the heterogeneous cluster virtual machine relocation based on scalable reinforcement learning. For specific details, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0089] In addition, in the implementation of the heterogeneous cluster virtual machine relocation system based on scalable reinforcement learning in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be distributed to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the heterogeneous cluster virtual machine relocation system based on scalable reinforcement learning is divided into different program modules to complete all or part of the functions described above.

[0090] See also Figure 8 In one embodiment, an electronic device for implementing a method for relocating heterogeneous cluster virtual machines based on scalable reinforcement learning is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a heterogeneous cluster virtual machine relocation program 203 based on scalable reinforcement learning.

[0091] Among them, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Furthermore, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed on the electronic device 200, such as the code of the heterogeneous cluster virtual machine relocation program 203 based on scalable reinforcement learning, but can also be used to temporarily store data that has been output or is to be output.

[0092] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0093] Figure 8 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 8 The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0094] The heterogeneous cluster virtual machine relocation program 203 based on scalable reinforcement learning stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve:

[0095] Collect cluster runtime information, including host CPU utilization and energy consumption data, and virtual machine service quality violation rate; the virtual machine service quality violation rate is the ratio of the time the virtual machine experiences insufficient resources to the total active time; update the host mask vector based on the cluster status and virtual machine relocation rules; the host mask vector at time t is represented by Mask t =(m1,m2,…,m N |m i ∈{0,1}), m i =1 means that the virtual machine cannot be placed, m i =0 indicates that a virtual machine can be placed; the length of the host mask vector is equal to the number of hosts in the cluster, and the order is consistent with the order of the hosts in the cluster;

[0096] Establish a state feature representation method for embedding heterogeneous host types, mapping different types of hosts in the cluster into the same space to reflect the similarities and differences between hosts;

[0097] Establish a feature extraction network for the host's historical utilization sequence and convert each host's CPU utilization sequence into a fixed-length feature vector;

[0098] A reinforcement learning-based model for solving the virtual machine scheduling problem is established. The model inputs are the cluster state vector, host mask vector, and host type embedding, and the model output is the action vector for virtual machine relocation.

[0099] A scalable virtual machine relocation method is established, wherein the virtual machine relocation method is to migrate virtual machines on overloaded or underloaded hosts according to the state of the cluster and the virtual machine relocation rules to achieve the purpose of minimizing energy consumption and maximizing service quality.

[0100] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0101] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0102] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0103] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for relocating virtual machines in a heterogeneous cluster based on scalable reinforcement learning, characterized in that: The following steps are involved: Collect cluster runtime information, including CPU utilization series and energy consumption data of N hosts, as well as the service quality violation rate of virtual machines; the service quality violation rate of virtual machines is the ratio of the time the virtual machine experiences insufficient resources to the total active time; Update the host mask vector based on the cluster state and VM relocation rules; The host mask vector has a length equal to the number of hosts in the cluster and an order consistent with the order of the hosts in the cluster; The virtual machine relocation rules include: (1) no virtual machines can be placed on an overloaded host; (2) virtual machines on an underloaded host can only be migrated to other non-overloaded hosts or powered-off hosts of different types; (3) no virtual machines can be placed on a host that is down or undergoing maintenance; Establish a state feature representation method for embedding heterogeneous host types, mapping different types of hosts in the cluster into the same space to reflect the similarities and differences between hosts; Establish a feature extraction network for the host's historical utilization sequence and convert each host's CPU utilization sequence into a fixed-length feature vector; A reinforcement learning basic model for solving the virtual machine scheduling problem is established. The reinforcement learning basic model for solving the virtual machine scheduling problem includes a state space, an action space, and a reward function. The state space includes the historical utilization sequence of each host in the cluster, the heterogeneous host type embedding, and the host mask vector. The action space allows the selection of the target host for virtual machine placement, with a total of N possible choices. The reward function consists of two parts: energy consumption reward and service quality reward. The energy consumption reward is expressed as , the service quality reward is expressed as , the final reward is expressed as ,in α and β are the weight values ​​of energy consumption reward and service quality reward respectively; the input of the reinforcement learning basic model is the cluster state vector, host mask vector and host type embedding, and the output of the reinforcement learning basic model is the action vector of virtual machine relocation; A scalable virtual machine relocation method is established. The virtual machine relocation method refers to migrating virtual machines on overloaded or underloaded hosts according to the state of the cluster and the virtual machine relocation rules to minimize energy consumption and maximize service quality. Specifically, the virtual machine relocation method includes a virtual machine placement method based on reinforcement learning, a virtual machine selection method and a host state determination method; for the virtual machine placement method based on reinforcement learning, a reinforcement learning algorithm is used to select an optimal target host for each virtual machine that needs to be relocated according to the state vector of the cluster and the host mask vector to maximize the cumulative reward; for the virtual machine selection method, the system first calculates the CPU utilization and computing capacity corresponding to the optimal energy efficiency of the overloaded host, and the computing capacity is expressed in millions of MIPS per second, and then traverses all virtual machines on the overloaded host and calculates its migration efficiency for each virtual machine. The difference between the utilization of the final host and the optimal energy efficiency utilization is calculated and the result is stored; then, the virtual machine with the smallest difference is selected from all virtual machines as the migration object, and the virtual machine selection process is repeated until the host is no longer overloaded; for host status determination, first, the overload host threshold is set, and the host with a current utilization rate exceeding 90% is regarded as an overloaded host and needs to be relocated; then the underload host threshold is set to the 1 / 4 quantile of the current utilization rates of N hosts, and the host with a utilization rate lower than the underload host threshold is regarded as underloaded and needs to be shut down to save energy; then, the status of all hosts is determined, and the hosts are classified into three states: overloaded, normal, or underloaded based on the comparison of utilization rates with the threshold; finally, the virtual machines that need to be migrated are determined from the overloaded and underloaded hosts, and then reinforcement learning is used to complete the relocation of the virtual machines. After the virtual machines are migrated out, the underloaded hosts are shut down, and no operation is performed on the normal hosts.

2. The method for relocating virtual machines in a heterogeneous cluster based on scalable reinforcement learning according to claim 1, characterized in that: The feature extraction network for establishing the host historical utilization sequence is specifically as follows: The characteristics of the host historical utilization sequence are obtained through a recurrent neural network. The recurrent neural network adopts a standard RNN network structure. Its input is the historical utilization sequence of N hosts. The historical utilization sequence of each host is input into the RNN network structure in parallel. t The hidden state of the RNN output is represented as the feature of the host utilization sequence and is used as part of the input of the reinforcement learning model for further virtual machine scheduling decisions.

3. A heterogeneous cluster virtual machine relocation system based on scalable reinforcement learning, characterized by: It includes information acquisition module, mask vector update module, state feature representation module, feature extraction module, reinforcement learning module and relocation module; The information collection module is used to collect cluster runtime information, including CPU utilization sequence and energy consumption data of N hosts, and service quality violation rate of virtual machines; the service quality violation rate of virtual machines is the ratio of the time when the virtual machine experiences insufficient resources to the total active time; The mask vector update module is used to update the host mask vector according to the cluster status and the virtual machine relocation rule; The host mask vector has a length equal to the number of hosts in the cluster and an order consistent with the order of the hosts in the cluster; The virtual machine relocation rules include: (1) no virtual machines can be placed on an overloaded host; (2) virtual machines on an underloaded host can only be migrated to other non-overloaded hosts or powered-off hosts of different types; (3) no virtual machines can be placed on a host that is down or undergoing maintenance; The state feature representation module is used to establish a state feature representation method for embedding heterogeneous host types, mapping different types of hosts in the cluster into the same space to reflect the similarities and differences between the hosts; The feature extraction module is used to establish a feature extraction network for the host historical utilization sequence and convert the CPU utilization sequence of each host into a feature vector of fixed length; The reinforcement learning module is used to establish a reinforcement learning basic model for solving the virtual machine scheduling problem. The reinforcement learning basic model for solving the virtual machine scheduling problem includes a state space, an action space, and a reward function. The state space includes the historical utilization sequence of each host in the cluster, the heterogeneous host type embedding, and the host mask vector. The action space allows the selection of a target host for virtual machine placement, with a total of N possible choices. The reward function consists of two parts: energy consumption reward and service quality reward. The energy consumption reward is expressed as , the service quality reward is expressed as , the final reward is expressed as ,in α and β are the weight values ​​of energy consumption reward and service quality reward respectively; the input of the reinforcement learning basic model is the cluster state vector, host mask vector and host type embedding, and the output of the reinforcement learning basic model is the action vector of virtual machine relocation; The relocation module is used to establish a scalable virtual machine relocation method. The virtual machine relocation method refers to migrating virtual machines on overloaded or underloaded hosts according to the state of the cluster and the virtual machine relocation rules to minimize energy consumption and maximize service quality. Specifically, the virtual machine relocation method includes a virtual machine placement method based on reinforcement learning, a virtual machine selection method and a host state determination method; for the virtual machine placement method based on reinforcement learning, a reinforcement learning algorithm is used to select an optimal target host for each virtual machine that needs to be relocated according to the state vector of the cluster and the host mask vector to maximize the cumulative reward; for the virtual machine selection method, the system first calculates the CPU utilization and computing capacity corresponding to the optimal energy efficiency of the overloaded host, and the computing capacity is expressed in millions of indexes per second (MIPS). Then, all virtual machines on the overloaded host are traversed, and for each virtual machine, the optimal target host is selected. The difference between the host utilization after migration and the optimal energy efficiency utilization is calculated and the result is stored; then, the virtual machine with the smallest difference is selected from all virtual machines as the migration object, and the virtual machine selection process is repeated until the host is no longer overloaded; for host status determination, first, an overloaded host threshold is set, and hosts with a current utilization exceeding 90% are regarded as overloaded hosts and need to be relocated; then the underloaded host threshold is set to the 1 / 4 quantile of the current utilization of N hosts, and hosts with a utilization lower than the underloaded host threshold are regarded as underloaded and need to be shut down to save energy; then, the status of all hosts is determined, and based on the comparison of utilization with the threshold, the hosts are classified into three states: overloaded, normal, or underloaded; finally, the virtual machines that need to be migrated are determined from the overloaded and underloaded hosts, and then reinforcement learning is used to complete the relocation of the virtual machines. After the virtual machines are migrated, the underloaded hosts are shut down, and no operation is performed on the normal hosts.

4. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning as described in any one of claims 1-2.

5. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the heterogeneous cluster virtual machine relocation method based on scalable reinforcement learning according to any one of claims 1-2 is implemented.

Images