Virtual machine placement method based on confidence upper bound and adaptive learning rate improved Q-learning

Abstract

This invention discloses a virtual machine placement method based on improved Q-learning using confidence upper bounds and adaptive learning rates, belonging to the field of virtual machine placement technology. It solves the problems of low efficiency and limitations of traditional virtual machine placement methods in the prior art. The invention includes the following steps: S1. Based on the improved influence coefficient and migration time, an IFMT virtual machine selection algorithm is constructed to select the most suitable virtual machine for migration; S2. Through confidence upper bound technology and adaptive learning rate adjustment, a state space, action space, and reward function are set to construct an improved Q-learning virtual machine placement model to find the optimal placement host; S3. The optimal placement host is used to place the most suitable virtual machine for migration, realizing the virtual machine placement strategy of virtual machine integration. This invention improves the convergence speed and flexibility of the algorithm, optimizes virtual machine placement decisions, and can be applied to virtual machine placement design.

Description

Technical Field

[0001] This invention relates to virtual machine placement methods, and more particularly to virtual machine placement methods based on confidence upper bounds and adaptive learning rates to improve Q-learning, belonging to the field of virtual machine placement technology. Background Technology

[0002] A suitable virtual machine selection strategy is fundamental to achieving efficient resource allocation and stable system operation in cloud computing. In actual operation, selecting appropriate virtual machines for migration or scheduling can effectively prevent host overload, improve resource utilization, and ensure user service quality. Conversely, improper virtual machine selection can lead to uneven resource allocation, with some hosts overloaded while others remain idle. This not only reduces overall resource utilization but may also cause system performance degradation, severely impacting user experience. Therefore, developing scientific and efficient virtual machine selection algorithms has always been a key research focus in the field of cloud computing.

[0003] While existing virtual machine (VM) selection algorithms have proven effective in certain scenarios, their shortcomings are becoming increasingly apparent as cloud computing environments become more complex and diverse. These algorithms typically focus on a single factor, such as VM resource usage, neglecting the time required for VM migration and its impact on other VMs and the host. This one-dimensional approach makes it difficult for the algorithms to make optimal decisions in complex and ever-changing cloud environments. Furthermore, most current research on VM selection strategies focuses on the resource correlation between VMs, while research on the correlation between the host and VMs is relatively lacking. For example, one algorithm calculates the impact coefficient of VMs on host overload to assess the differences between different VMs and identify VMs that have a significant impact on host overload, helping overloaded hosts quickly return to normal load. However, this algorithm uses cosine similarity to measure the similarity of two vector directions. This approach only considers whether the vector directions are the same, without considering the magnitude and linear relationship, failing to fully reflect the actual relationship between vectors. Moreover, this algorithm does not adequately consider migration time; excessively long migration times may negatively impact overall system performance.

[0004] Meanwhile, with the rapid development of the cloud computing era, the importance of virtual machine placement strategy is becoming increasingly prominent. Virtual machine placement is directly related to the resource utilization, energy consumption and performance of the data center. The selection of virtual machines will also affect the effect of virtual machine placement. By optimizing the placement of virtual machines, resource waste can be effectively reduced, resource utilization can be improved, energy consumption costs can be reduced, and users can be provided with more efficient, reliable and economical cloud services.

[0005] While existing virtual machine placement algorithms can meet basic needs in some conventional scenarios, they are revealing a series of problems as cloud computing environments become increasingly complex and application scenarios become more diverse. Traditional virtual machine placement algorithms mostly rely on pre-defined rules or simple heuristic strategies. These methods lack flexibility in the face of dynamically changing cloud computing loads, struggle to adapt to real-time fluctuations in resource demand, and are prone to unreasonable resource allocation, resulting in low resource utilization. In recent years, reinforcement learning-based virtual machine placement algorithms have gained attention. Among them, Q-learning algorithms, due to their decision-making based on state-action value functions, have shown certain advantages in the field of virtual machine placement, optimizing virtual machine placement strategies through continuous trial and error. However, Q-learning algorithms suffer from low learning efficiency and slow convergence speed. In large-scale cloud computing environments with numerous states and actions, they struggle to quickly find the optimal placement strategy. Their fixed learning rate and exploration strategy also make it difficult for the algorithm to properly handle the relationship between exploring new strategies and utilizing existing experience in complex and ever-changing environments, easily getting trapped in local optima and failing to fully discover the best virtual machine placement solution.

[0006] In summary, the virtual machine placement method for Q-learning needs to be improved based on confidence upper bounds and adaptive learning rates. Summary of the Invention

[0007] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0008] In view of this, in order to solve the problems of low efficiency and limitations of traditional virtual machine placement methods in the prior art, this invention provides a virtual machine placement method based on confidence upper bound and adaptive learning rate to improve Q-learning.

[0009] The technical solution is as follows: An improved virtual machine placement method for Q-learning based on confidence upper bounds and adaptive learning rates, including the following steps:

[0010] S1. Based on the improved impact coefficient and migration time, the IFMT virtual machine selection algorithm is constructed to select the virtual machine and obtain the most suitable virtual machine for migration;

[0011] S2. By using confidence upper bound techniques and adaptive learning rate adjustment, we set the state space, action space, and reward function to construct an improved Q-learning virtual machine placement model and find the optimal placement host.

[0012] S3. By improving the virtual machine placement model of Q-learning, the most suitable virtual machine for migration is placed on the optimal placement host, thereby realizing the virtual machine placement strategy of virtual machine consolidation.

[0013] Furthermore, in S1, the influence coefficient is improved. Represented as:

[0014] (1)

[0015] (2)

[0016] in, For overloaded host, For overloaded host numbers, For overloaded host The virtual machine, Indicates overloaded host and its first virtual machine The Pearson correlation coefficient between resource usage, Indicates overloaded host CPU usage Indicates overloaded host The virtual machine CPU usage, where d represents the resource dimension. and These represent overloaded main units. and its first virtual machine The various resources, and These represent overloaded main units. and its first virtual machine The average value of each resource;

[0017] Migration time required for virtual machines Represented as:

[0018] (3)

[0019] in, Represents virtual machine Current memory usage Indicates overloaded host Available remaining network bandwidth;

[0020] The IFMT virtual machine selection algorithm is expressed as follows:

[0021] (4)

[0022] Filtered using the IFMT virtual machine selection algorithm The virtual machine with the highest value is selected as the most suitable virtual machine for migration, while the resource utilization of the overloaded host is updated.

[0023] Furthermore, in S2, the state space Represented as:

[0024] (5)

[0025] (6)

[0026] in, The average CPU utilization of all active hosts. The number of overloaded hosts, The number of underloaded hosts. For overloaded host Total CPU capacity The number of active hosts;

[0027] Action space Represented as:

[0028] (7)

[0029] in, For possible hosts, For all possible host selections A set;

[0030] reward function Represented as:

[0031] (8)

[0032] (9)

[0033] (10)

[0034] in, Let be the energy consumption of the host in the current state space and action space, which is expressed as a linear function of utilization. This indicates a breach of the SLA after selecting a host. This is the penalty coefficient for SLA default. Indicates dissatisfaction In this situation, This represents the maximum power consumption of the host computer. This represents the power consumption of the host in idle state.

[0035] By employing the confidence upper bound technique, we construct the Q-learning update formula and the UCB formula. Under the UCB policy, we construct the criterion formula for selecting actions, that is, at each time step, based on the reward function... Evaluate the effectiveness of the current state action, update the Q-value of adaptive learning, and complete the construction of the improved Q-learning virtual machine placement model;

[0036] The Q-learning update formula is expressed as:

[0037] (11)

[0038] in, γ is the learning rate, which represents the step size for updating the Q-value, and γ is the discount factor. For the next state The maximum Q-value, which is used to update the Q-value of the current state-space and action-space pair. For action;

[0039] The UCB formula is expressed as:

[0040] (12)

[0041] Where c is a coefficient controlling the exploration. For state space Total number of visits For the current state space Select Action The number of times;

[0042] The formula for selecting actions is expressed as follows:

[0043] (13)

[0044] The adaptive learning rate adjustment formula is expressed as:

[0045] (14).

[0046] The beneficial effects of this invention are as follows: This invention provides a virtual machine placement method based on confidence upper bounds and adaptive learning rate-based improved Q-learning. By using a virtual machine selection strategy algorithm based on improved influence coefficients and migration time, it accurately identifies the virtual machine that has the most significant impact on the overloaded host and has the shortest migration time, which is beneficial for quickly restoring the load of the overloaded host. By balancing exploration and utilization through confidence upper bound technology, and adjusting the learning rate according to the dynamic changes in learning, the convergence speed and flexibility of the algorithm are improved to optimize the virtual machine placement decision, find the optimal placement host, and finally implement the virtual machine placement using an improved Q-learning virtual machine placement model. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0048] Figure 1 This is a flowchart illustrating the improved virtual machine placement method for Q-learning based on confidence upper bounds and adaptive learning rates.

[0049] Figure 2 A pseudocode diagram of the IFMT algorithm;

[0050] Figure 3 A pseudocode diagram illustrating the virtual machine placement model execution process for improving Q-learning;

[0051] Figure 4 A schematic diagram illustrating the structural principle of the virtual machine placement model for improving Q-learning. Detailed Implementation

[0052] To make the technical solutions and advantages of the embodiments of the present invention clearer, the exemplary embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not an exhaustive list of all embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0053] refer to Figures 1-4 This embodiment describes a virtual machine placement method for Q-learning based on confidence upper bounds and adaptive learning rates, specifically including the following steps:

[0054] S1. Based on the improved impact coefficient and migration time, the IFMT virtual machine selection algorithm is constructed to select the virtual machine and obtain the most suitable virtual machine for migration;

[0055] S2. By using confidence upper bound techniques and adaptive learning rate adjustment, we set the state space, action space, and reward function to construct an improved Q-learning virtual machine placement model and find the optimal placement host.

[0056] S3. By improving the virtual machine placement model of Q-learning, the most suitable virtual machine for migration is placed on the optimal placement host, thereby realizing the virtual machine placement strategy of virtual machine consolidation.

[0057] Furthermore, in S1, the influence coefficient is improved. Represented as:

[0058] (1)

[0059] (2)

[0060] in, For overloaded host, For overloaded host numbers, For overloaded host The virtual machine, Indicates overloaded host and its first virtual machine The Pearson correlation coefficient between resource usage, Indicates overloaded host CPU usage Indicates overloaded host The virtual machine CPU usage, where d represents the resource dimension. and These represent overloaded main units. and its first virtual machine The various resources, and These represent overloaded main units. and its first virtual machine The average value of each resource;

[0061] Migration time required for virtual machines Represented as:

[0062] (3)

[0063] in, Represents virtual machine Current memory usage Indicates overloaded host Available remaining network bandwidth;

[0064] The IFMT virtual machine selection algorithm is expressed as follows:

[0065] (4)

[0066] Filtered using the IFMT virtual machine selection algorithm The virtual machine with the highest value is selected as the most suitable virtual machine for migration, while the resource utilization of the overloaded host is updated.

[0067] Specifically, according to the definition of the impact coefficient, under a specific resource dimension, the larger the value of the impact coefficient, the closer the resource consumption pattern of the overloaded host is to the resource demand pattern of the virtual machine. This indicates that the virtual machine exerts a more significant impact on the overloaded host. Therefore, it is necessary to select virtual machines with higher precision and screen out the most suitable virtual machines for migration.

[0068] The IFMT virtual machine selection algorithm carefully balances the dual impacts of virtual machine resource usage and migration time on host overload. Its core objective is to accurately identify the most suitable virtual machines for migration on an overloaded physical host—those that have the most significant impact on the overloaded host and require the shortest migration time. Using the IFMT algorithm, the host can maintain high resource utilization efficiency while reducing the number of virtual machine migration operations, quickly and effectively adjusting the host load to a suitable state. (Reference) Figure 2 The IFMT virtual machine selection algorithm first initializes the host resource utilization list, and at the same time obtains the virtual machine list of the current host. For each host, it obtains its resource utilization list, calculates the average resource utilization of the host, and obtains the resource utilization status at the current moment. Next, for each virtual machine (VM) on the host, the algorithm obtains its overload host detection and VM selection strategy resource utilization list, calculates the average resource utilization of the VMs, and obtains the resource utilization at the current moment. Based on the above data, the algorithm calculates the Pearson correlation coefficient to measure the correlation between host and VM resource utilization and calculates the impact coefficient (IF) to evaluate the impact of VMs on host resource utilization. Then, after estimating the time required for VM migration (Migrate_time), the algorithm calculates the ratio of impact coefficient to migration time (IFMT) as an indicator for selecting VMs to be migrated. Under the conditions that the current resource utilization is greater than the target resource utilization and the VM list is not empty, the VM with the largest IFMT value is found and added to the list of VMs to be migrated. At the same time, the host resource utilization is updated. Finally, the algorithm returns the list of VMs to be migrated, selected_vms, completing the entire VM selection process.

[0069] Furthermore, in S2, the state space Represented as:

[0070] (5)

[0071] (6)

[0072] in, The average CPU utilization of all active hosts. The number of overloaded hosts, The number of underloaded hosts. For overloaded host Total CPU capacity The number of active hosts;

[0073] Action space Represented as:

[0074] (7)

[0075] in, For possible hosts, For all possible host selections A set;

[0076] reward function Represented as:

[0077] (8)

[0078] (9)

[0079] (10)

[0080] in, Let be the energy consumption of the host in the current state space and action space, which is expressed as a linear function of utilization. This indicates a breach of the SLA after selecting a host. The penalty factor for SLA default is usually set to a large constant (e.g., 10) to emphasize the severity of SLA default. Indicates dissatisfaction In this situation, Maximum host power consumption, for;

[0081] By employing the confidence upper bound technique, we construct the Q-learning update formula and the UCB formula. Under the UCB policy, we construct the criterion formula for selecting actions, that is, at each time step, based on the reward function... Evaluate the effectiveness of the current state action, weigh energy consumption and SLA default penalty in the reward function, update the Q value of adaptive learning to learn the optimal policy, and complete the construction of the improved Q-learning virtual machine placement model;

[0082] The Q-learning update formula is expressed as:

[0083] (11)

[0084] in, γ is the learning rate, which represents the step size for updating the Q-value, and γ is the discount factor used to weigh the importance of current rewards against future rewards. For the next state The maximum Q-value, which is used to update the Q-value of the current state-space and action-space pair. For action;

[0085] The UCB formula is expressed as:

[0086] (12)

[0087] Where c is a coefficient controlling the exploration. For state space Total number of visits For the current state space Select Action The number of times it is used to balance the exploration of known information and new information;

[0088] The formula for selecting actions is expressed as follows:

[0089] (13)

[0090] The adaptive learning rate adjustment formula is expressed as:

[0091] (14)

[0092] Specifically, the state space describes the resource utilization of the cloud environment and mainly includes three parts;

[0093] The action space represents the selection of hosts to which virtual machines are assigned; it includes all possible host choices. The set of virtual machines, each with an allocation problem, involves the action of selecting a possible host. (i.e., a∈A) is assigned, and in adaptive learning, this selection is based on the current state space. The optimal adaptive learning Q-value or exploration strategy (such as UCB and ϵ-greedy strategy) is determined;

[0094] For the Q-Learning algorithm, the learning rate This is a relatively critical parameter, which determines the degree of influence of newly acquired information on the current Q value. In different virtual machine placement models of improved Q-learning, the adjustment method of the learning rate will be different. This application designs an adaptive learning rate adjustment formula, and to avoid division by 0, 1 is added to the denominator in the formula.

[0095] In the initial stage, when a specific state and its corresponding action are accessed for the first time, the number of accesses is 0, and the learning rate is set at this time. Setting it to 1 means that newly acquired information will completely replace the current Q-value. As the number of times this state and action are accessed increases, the learning rate... The learning rate will gradually decrease, meaning that the impact of newly acquired information on the current Q value will gradually decrease. By adjusting the learning rate in the above way, the virtual machine placement model of Q-learning can dynamically balance the impact of new information and existing information at different learning stages, thereby improving the efficiency and stability of learning. This method of using an adaptive learning rate can achieve rapid exploration in the initial stage of the model and maintain the robustness of learning in the later stage, preventing overfitting.

[0096] refer to Figure 3 First, initialize the Q-value list and learning rate. The algorithm uses a discount factor γ, an exploration rate ε, and a coefficient c to control exploration. For each virtual machine (VM) to be allocated, it obtains the current system state, including the average resource utilization of all active hosts, the number of overloaded hosts, and the number of underloaded hosts. Then, based on the conditions, it determines whether to randomly select an action with probability ε or to select an action using the UCB strategy. The UCB strategy calculates the UCB value of each action in the current state and selects the action with the largest UCB value. According to the selected action, it attempts to allocate the VM to the corresponding host. If the host cannot accommodate the VM, it attempts to randomly select a host that can accommodate the VM and has the lowest energy consumption, or activate an inactive host for placement. After the placement is completed, the algorithm obtains the next system state, calculates the reward obtained from this allocation, updates the Q value according to the Q-Learning update formula, dynamically adjusts the learning rate according to the number of action accesses, and updates the number of state accesses and the number of action accesses.

[0097] refer to Figure 4Based on the current state of the cloud data center composed of hosts, an action is executed, which is to assign the virtual machine to be migrated to a suitable host. The system then provides a reward based on the execution result. This placement action will change the environment state, bringing the entire environment to a new state.

[0098] Although the invention has been described with reference to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of interpreting or limiting the subject matter of the invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the invention is illustrative and not restrictive, and the scope of the invention is defined by the appended claims.

Claims

1. A virtual machine placement method based on confidence upper bound and adaptive learning rate improved Q-learning, characterized in that, Includes the following steps: S1. Based on the improved impact coefficient and migration time, the IFMT virtual machine selection algorithm is constructed to select the virtual machine and obtain the most suitable virtual machine for migration; S2. By using confidence upper bound techniques and adaptive learning rate adjustment, we set the state space, action space, and reward function to construct an improved Q-learning virtual machine placement model and find the optimal placement host. S3. By improving the virtual machine placement model of Q-learning, the most suitable virtual machine for migration is placed on the optimal placement host, thereby realizing the virtual machine placement strategy of virtual machine consolidation; In the S1, the improvement influence coefficient is expressed as: （1） （2） in, For overloaded host, For overloaded host numbers, For overloaded host The virtual machine, Indicates overloaded host and its first virtual machine The Pearson correlation coefficient between resource usage, Indicates overloaded host CPU usage Indicates overloaded host The virtual machine CPU usage, where d represents the resource dimension. and These represent overloaded main units. and its first virtual machine The various resources, and These represent overloaded main units. and its first virtual machine The average value of each resource; Migration time required for virtual machines Represented as: （3） in, Represents virtual machine Current memory usage Indicates overloaded host Available remaining network bandwidth; The IFMT virtual machine selection algorithm is expressed as follows: （4） Filtered using the IFMT virtual machine selection algorithm The virtual machine with the highest value is selected as the most suitable virtual machine for migration, while the resource utilization of the overloaded host is updated. In S2, the state space Represented as: （5） （6） in, The average CPU utilization of all active hosts. The number of overloaded hosts, The number of underloaded hosts. For overloaded host Total CPU capacity The number of active hosts; Action space Represented as: （7） in, For possible hosts, For all possible host selections A set; reward function Represented as: （8） （9） （10） in, Let be the energy consumption of the host in the current state space and action space, which is expressed as a linear function of utilization. This indicates a breach of the SLA after selecting a host. This is the penalty coefficient for SLA default. Indicates dissatisfaction In this situation, This represents the maximum power consumption of the host computer. This represents the power consumption of the host in idle state. By employing the confidence upper bound technique, we construct the Q-learning update formula and the UCB formula. Under the UCB policy, we construct the criterion formula for selecting actions, that is, at each time step, based on the reward function... Evaluate the effectiveness of the current state action, update the Q-value of adaptive learning, and complete the construction of the improved Q-learning virtual machine placement model; The Q-learning update formula is expressed as: （11） in, γ is the learning rate, which represents the step size for updating the Q-value, and γ is the discount factor. For the next state The maximum Q-value, which is used to update the Q-value of the current state-space and action-space pair. For action; The UCB formula is expressed as: （12） Where c is a coefficient controlling the exploration. For state space Total number of visits For the current state space Select Action The number of times; The formula for selecting actions is expressed as follows: （13） The adaptive learning rate adjustment formula is expressed as: （14）。

Images