Virtual machine scheduling method, device and system, storage medium and program product

Abstract

The invention provides a virtual machine scheduling method, device and system, a storage medium and a program product, and relates to the field of cloud computing. The method comprises the following steps: identifying a source numa node of a first virtual machine to be migrated in a first computing node according to the total vCPU utilization rate of each numa node in the first computing node; under the condition that the target numa node capable of receiving the first virtual machine exists in the first computing node, migrating the first virtual machine to the target numa node; and under the condition that migration is not successful, first event information is reported to the cluster control node to trigger virtual machine live migration between the first computing node and the second computing node.

Description

Technical Field

[0001] The present disclosure relates to the field of cloud computing, and in particular to a virtual machine scheduling method, device, system, storage medium and program product. Background Art

[0002] With the widespread application of cloud computing and the increase in the number of NUMA (non-uniform memory access) nodes in servers, the management of virtual machine memory and vCPU (Virtual Central Processing Unit) in computing nodes faces many problems. When the virtual machine vCPU and memory are fixed on a NUMA node, there is an imbalance in the vCPU utilization between NUMA nodes, which will cause different virtual machine vCPUs to affect performance due to resource competition. Summary of the invention

[0003] A technical problem to be solved by the present disclosure is to provide a virtual machine scheduling method, device, system, storage medium and program product, which can solve the problem of uneven vCPU utilization between NUMAs affecting the performance of the virtual machine and reduce the impact of hot migration on the operation of the virtual machine.

[0004] According to one aspect of the present disclosure, a virtual machine scheduling method is proposed, which is executed by a first computing node, including: identifying a source NUMA node of a first virtual machine to be migrated in the first computing node according to a total virtual central processing unit (vCPU) utilization rate of each non-consistent memory access NUMA node in the first computing node; migrating the first virtual machine to the target NUMA node if there is a target NUMA node in the first computing node that can receive the first virtual machine; and reporting first event information to a cluster control node if the migration is unsuccessful, so as to trigger hot migration of the virtual machine between the first computing node and the second computing node.

[0005] In some embodiments, hot migration of a virtual machine between a first computing node and a second computing node includes hot migrating a second virtual machine in a target numa node to the second computing node so that the first virtual machine is migrated to the target numa node.

[0006] In some embodiments, hot migration of a virtual machine between a first computing node and a second computing node includes hot migrating the first virtual machine to the second computing node.

[0007] In some embodiments, when there is no target numa node, the second event information is reported to the cluster control node so that the cluster control node hot migrates the first virtual machine to the second computing node.

[0008] In some embodiments, the target NUMA node is a NUMA node whose total vCPU utilization is less than or equal to a threshold after receiving the first virtual machine and which has sufficient memory to host the first virtual machine.

[0009] In some embodiments, identifying a source numa node of a first virtual machine to be migrated includes: identifying a numa node whose total vCPU utilization is greater than a threshold as a source numa node.

[0010] In some embodiments, the first virtual machine is a virtual machine with the lowest vCPU utilization in the source numa node.

[0011] In some embodiments, the second virtual machine is a virtual machine with the lowest vCPU utilization in the target numa node.

[0012] In some embodiments, the first event information includes an address of the first computing node, a target numa node identifier, and a vCPU utilization of each virtual machine of the target numa node.

[0013] In some embodiments, migrating the first virtual machine to the target numa node includes: migrating the vCPU and memory of the first virtual machine to the target numa node.

[0014] According to another aspect of the present disclosure, a virtual machine scheduling device is also proposed, which is located in a first computing node and includes: a CPU utilization collection module, configured to collect the total virtual central processing unit vCPU utilization of each non-consistent memory access NUMA node in the first computing node; and a virtual machine migration module, configured to identify the source NUMA node of the first virtual machine to be migrated in the first computing node according to the total vCPU utilization of each NUMA node, and if there is a target NUMA node in the first computing node that can receive the first virtual machine, migrate the first virtual machine to the target NUMA node, and if the migration is not successful, report the first event information to the cluster control node to trigger the hot migration of the virtual machine between the first computing node and the second computing node.

[0015] According to another aspect of the present disclosure, a virtual machine scheduling device is also proposed, which is located in a first computing node and includes: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the virtual machine scheduling method as described above based on instructions stored in the memory.

[0016] According to another aspect of the present disclosure, a virtual machine scheduling system is also proposed, including: a first computing node, including the above-mentioned virtual machine scheduling device; a second computing node; and a cluster control node, configured to receive first event information sent by the virtual machine scheduling device when it is determined that there is a target non-consistent memory access NUMA node, but the first virtual machine in the source NUMA node has not been successfully migrated to the target NUMA node, and perform virtual machine hot migration between the first computing node and the second computing node.

[0017] In some embodiments, the cluster control node is configured to hot migrate the second virtual machine in the target numa node to the second computing node, or to hot migrate the first virtual machine in the source numa node to the second computing node.

[0018] In some embodiments, the cluster control node is further configured to receive second event information sent by the virtual machine scheduling device when it is determined that there is no target numa node, and hot migrate the first virtual machine in the source numa node to the second computing node.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is further provided, on which computer program instructions are stored, and when the instructions are executed by a processor, the virtual machine scheduling method as described above is implemented.

[0020] According to another aspect of the present disclosure, a computer program product is also proposed, including a computer program or instructions, and the computer program or instructions implement the above-mentioned virtual machine scheduling method when executed by a processor.

[0021] In the above embodiment, when there is an imbalance in vCPU utilization between NUMA nodes in a computing node, virtual machine migration is first performed between different NUMA nodes in the same computing node. If the migration is unsuccessful, hot migration of virtual machines between different computing nodes is triggered, which solves the problem of uneven vCPU utilization between NUMA nodes affecting the performance of the virtual machine and reduces the impact of hot migration on the operation of the virtual machine.

[0022] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0024] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0025] Figure 1A flowchart of some embodiments of the virtual machine scheduling method disclosed in the present invention;

[0026] Figure 2 Schematic diagram of the flow of other embodiments of the virtual machine scheduling method disclosed in the present invention;

[0027] Figure 3 Schematic diagram of the flow of other embodiments of the virtual machine scheduling method disclosed in the present invention;

[0028] Figure 4 Schematic diagram of the flow of other embodiments of the virtual machine scheduling method disclosed in the present invention;

[0029] Figure 5 It is a structural schematic diagram of some embodiments of the virtual machine scheduling device disclosed in the present invention;

[0030] Figure 6 It is a structural schematic diagram of some other embodiments of the virtual machine scheduling device disclosed in the present invention;

[0031] Figure 7 It is a schematic diagram of the structure of some embodiments of the virtual machine scheduling system disclosed in the present invention. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0033] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0034] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0035] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0036] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0037] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0038] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0039] When creating or starting a virtual machine, fixing the vCPU and memory in a numa node can solve the problem of cross-numa access. However, in public cloud scenarios, the business type in the virtual machine is unknown, so the utilization of the virtual machine on the computing node is also unknown. Especially when the virtual machine vCPU is oversold, the vCPU load in a numa node is uncertain and has large fluctuations. When the virtual machine vCPU and memory are fixed on a numa node, there is an imbalance in the vCPU utilization between numa nodes, which will cause the performance of different virtual machine vCPUs to be affected due to resource competition.

[0040] The present disclosure provides a virtual machine scheduling solution that can solve the problem of uneven vCPU utilization between NUMAs affecting the performance of virtual machine operations, and can reduce the impact of hot migration on virtual machine operations. The solution of the present disclosure will be introduced below using a specific embodiment as an example.

[0041] Figure 1 This is a flowchart of some embodiments of the virtual machine scheduling method disclosed in the present invention. The embodiment is executed by a first computing node, which is a server on which virtual machines and other management software run. The embodiment includes steps S11-S13.

[0042] In step S11, a source numa node of a first virtual machine to be migrated in the first computing node is identified according to the total vCPU utilization of each numa node in the first computing node.

[0043] In some embodiments, the first computing node can use the CPU utilization collection module to collect and summarize the overall vCPU utilization of virtual machines on different numa nodes, and identify the numa node whose total vCPU utilization is greater than a threshold as the source numa node.

[0044] For example, if the total vCPU utilization in a numa node is greater than 50%, it is considered that a load imbalance occurs in the first computing node, and the virtual machines in the numa node with a total vCPU utilization greater than 50% need to be migrated.

[0045] In some embodiments, the first virtual machine is the virtual machine with the lowest vCPU utilization in the source numa node. The memory of the virtual machine with higher vCPU utilization is frequently accessed, and the migration of the virtual machine with higher vCPU utilization is prone to migration failure. In this embodiment, migrating the virtual machine with the lowest vCPU utilization can improve the success rate of virtual machine migration.

[0046] In step S12, if there is a target numa node in the first computing node that can receive the first virtual machine, the first virtual machine is migrated to the target numa node.

[0047] In some embodiments, the target NUMA node is a NUMA node whose total vCPU utilization is less than or equal to a threshold after receiving the first virtual machine and which has sufficient memory to host the first virtual machine.

[0048] For example, if after the first virtual machine is migrated to a numa node, the sum of the total vCPU utilization in the numa node is less than or equal to 50%, and there is enough content on the numa node to carry the first virtual machine, then the numa node is selected as the target numa node and the first virtual machine is migrated to the target numa node.

[0049] In step S13, if the migration is not successful, first event information is reported to the cluster control node to trigger hot migration of the virtual machine between the first computing node and the second computing node.

[0050] For example, if a target NUMA node is selected before the migration of the first virtual machine, but when migrating the first virtual machine, the first virtual machine cannot be migrated to the target NUMA node because a new virtual machine is created on the target NUMA node or the target NUMA node has insufficient memory due to other reasons, the first computing node reports an event to the cluster control node, thereby triggering the hot migration of the virtual machines between the first computing node and the second computing node.

[0051] The cluster control node is a server on which virtual machine control software runs to load control operations such as creation and deletion of virtual machines.

[0052] In the case of load imbalance between NUMA nodes, if the virtual machine in the NUMA node with high total vCPU utilization is directly hot migrated to other computing nodes to complete the load balancing between NUMA nodes in the computing node, the hot migration will have a temporary impact on the virtual machine and occupy the network bandwidth. In the above embodiment, in the case of vCPU utilization imbalance between NUMA nodes in the computing node, the virtual machine migration is first performed between different NUMA nodes in the same computing node. If the migration is unsuccessful, the hot migration of the virtual machine between different computing nodes is triggered, which solves the problem of the uneven vCPU utilization between NUMA nodes affecting the performance of the virtual machine, and reduces the impact of hot migration on the operation of the virtual machine.

[0053] In some embodiments, hot migration of a virtual machine between a first computing node and a second computing node includes hot migrating a second virtual machine in a target numa node to the second computing node so that the first virtual machine is migrated to the target numa node.

[0054] In this embodiment, since the memory of the target NUMA node is insufficient to carry the first virtual machine migrated from the source NUMA node, the cluster control node migrates the virtual machine in the target NUMA node to other computing nodes, freeing up enough memory in the target NUMA node to host the first virtual machine, thereby solving the problem of uneven vCPU utilization between NUMA nodes affecting the performance of the virtual machine.

[0055] In some embodiments, the second virtual machine is a virtual machine with the lowest vCPU utilization in the target numa node. This embodiment can improve the success rate of virtual machine migration and reduce the impact on virtual machines with higher vCPU utilization.

[0056] In some other embodiments, hot migration of a virtual machine between a first computing node and a second computing node includes: hot migration of the first virtual machine to the second computing node.

[0057] In this embodiment, if the memory of the target NUMA node is insufficient to carry the first virtual machine migrated from the source NUMA node, the cluster control node will hot migrate the first virtual machine to other computing nodes, thereby achieving load balancing between NUMA nodes and between computing nodes.

[0058] Figure 2 It is a flowchart diagram of other embodiments of the virtual machine scheduling method disclosed in the present invention. This embodiment is executed by the first computing node and includes steps S21-S26.

[0059] In step S21, a source numa node of a first virtual machine to be migrated in the first computing node is identified according to the total vCPU utilization of each numa node in the first computing node.

[0060] In step S22, it is identified that there is a target numa node in the first computing node that can receive the first virtual machine. If so, step S23 is executed; otherwise, step S26 is executed.

[0061] In step S23, the first virtual machine is migrated to the target numa node.

[0062] In step S24, if the migration is unsuccessful, first event information is reported to the cluster control node.

[0063] In some embodiments, the first event information includes the address of the first computing node, the target numa node identifier, and the vCPU utilization of each virtual machine of the target numa node. The address of the first computing node is, for example, an IP address. The cluster control node identifies which computing node needs to perform virtual machine hot migration based on the address of the first computing node; identifies which virtual machine in which numa node needs hot migration based on the target numa node identifier; and identifies which virtual machine needs to be migrated out of the target numa node based on the vCPU utilization of each virtual machine.

[0064] In step S25, in response to the cluster control node hot migrating the second virtual machine in the target numa node to the second computing node, the first virtual machine is migrated to the target numa node.

[0065] The cluster control node selects the virtual machine with the lowest vCPU utilization in the target numa node and hot migrates it to the second computing node to free up enough memory in the target numa node to host the first virtual machine.

[0066] In some embodiments, the cluster control node selects multiple virtual machines in the target numa node to migrate to other computing nodes to free up enough memory in the target numa node to host the first virtual machine.

[0067] In some embodiments, after the target numa node frees up memory, the first computing node will try again to migrate the first virtual machine to the target numa node. If the migration fails, the first event information will be reported to the cluster control node again.

[0068] In some embodiments, the vCPU and memory of the first virtual machine are migrated to the target numa node.

[0069] For example, when migrating a virtual machine, the first part is to migrate the vCPU. For example, by using the operating system to bind cores, use the taskset(cpumask, pid, dst_node) function to bind the virtual machine vCPU thread and other threads to the target numa node. The second part is to migrate the memory. For example, use the migratepags(pid, snode, dnode) function in the libnuma library to migrate the memory used by the virtual machine process to the target numa node. Reduce the performance issues caused by cross-numa access between the virtual machine vCPU and memory.

[0070] In step S26, the second event information is reported to the cluster control node so that the cluster control node hot migrates the first virtual machine to the second computing node.

[0071] For example, if the first computing node determines that there is no target NUMA node, it reports to the cluster control node that other NUMA nodes are insufficient in memory or overloaded, and sends the address of the first computing node, the source NUMA node identifier, and the vCPU utilization of each virtual machine in the source NUMA node to the cluster control node, or sends the first virtual machine identifier to the cluster control node, so that the cluster control node hot migrates the first virtual machine to the second computing node.

[0072] In the above embodiment, virtual machines are preferentially migrated between different NUMA nodes. If there is a target NUMA node but the migration is not successful, hot migration of virtual machines is performed between computing nodes to free up memory for completing the migration of virtual machines between NUMA nodes. If there is no target NUMA node, hot migration of virtual machines is performed directly between computing nodes to reduce the problem of virtual machine operation performance being affected by uneven vCPU utilization between NUMA nodes.

[0073] The following will introduce the solution of the present disclosure by taking a specific embodiment as an example. Figure 3 As shown, Figure 3 The flowchart of another embodiment of the virtual machine scheduling method disclosed in the present invention includes steps S31-S314. A cluster includes a cluster control node, multiple computing nodes, and each computing node has multiple NUMA nodes. The number of NUMA nodes in a computing node is related to the processor model and BIOS settings. Currently, most Intel processor computing nodes have two NUMA nodes, and some processors have four NUMA nodes.

[0074] In step S31, the host CPU topology is obtained.

[0075] The host is a physical server that runs virtual machines and usually has a Linux operating system installed. The compute node obtains the number of CPUs and numa nodes of the entire physical server by reading the information in / sys / devices / system / cpu / cpux / topology in the Linux operating system, and then forms the topology of the physical server CPU to prepare for the subsequent virtual machine migration.

[0076] In step S32, the virtual machines on the host machine are traversed.

[0077] In step S33, ensure that the virtual machine memory and the vCPU are in the same numa node.

[0078] When creating a virtual machine, the memory and vCPU may not be on the same NUMA node. In this case, you need to set the memory and vCPU of the virtual machine to the same NUMA node.

[0079] In step S34, the numa nodes are traversed to calculate the total vCPU utilization of each numa node.

[0080] Calculate the vCPU utilization of all virtual machines in the NUMA node, add up the vCPU utilization of all virtual machines, and get the total vCPU utilization of the entire NUMA node.

[0081] For example, the vCPU utilization of all virtual machines on numa node N = , where VMi represents the vCPU usage of the i-th virtual machine, and N(VMi) represents the i-th virtual machine on a certain node.

[0082] In step S35, it is determined whether there is load imbalance among the NUMA nodes. If so, step S36 is executed; otherwise, step S34 is continued.

[0083] For example, a certain algorithm is used to determine whether the entire computing node is balanced. If it is not balanced, the corresponding migration operation will be triggered. If the vCPU utilization of the virtual machine on a numa node N is greater than 50% of the overall utilization of the numa node, it means that the vCPU utilization of the virtual machine on the computing node is unbalanced, and the virtual machine on the numa node needs to be migrated to other numa nodes.

[0084] For example, the total vCPU utilization of all virtual machines on a computing node is CPUUSAGE(N) = U(vm0) + U(vm1)y + U(vm2)y 2 +…+U(vmn)y n , y is the attenuation factor, for example, the value is 0.8. vmn is the time that the virtual machine occupies the vCPU in the period T. The relationship between n and T in vmn is: n represents the number of periods T that have passed. For example, vm0 represents the current period, vm1 represents the previous period, vm2 represents the previous two periods, and so on. The value of period T is, for example, 100ms. If , it means that there is a load imbalance between the NUMA nodes of the computing node.

[0085] The load of a virtual machine is the accumulation of loads in multiple cycles. The longer the time, the smaller the impact on the overall load. Therefore, the above formula can avoid misjudgment caused by fluctuations in virtual machine vCPU utilization.

[0086] In step S36, a first virtual machine is selected from the numa nodes whose total vCPU utilization is greater than a threshold.

[0087] For example, a virtual machine with the lowest vCPU utilization on the muna node is selected as the first virtual machine.

[0088] In step S37, a target numa node is selected.

[0089] The selected target NUMA node must meet two conditions. The first condition is that after the first virtual machine is migrated to the NUMA node, the total vCPU utilization of the existing virtual machines on the NUMA node cannot be greater than 50% of the overall utilization of the target NUMA node. The second condition is that there is enough content on the NUMA node to host the first virtual machine. After the first virtual machine is migrated, the next cycle will be entered for judgment. If it is judged that there is an imbalance between the NUMA nodes again, the virtual machine will be migrated again.

[0090] In step S38, it is determined whether the numa node is successfully obtained, if so, step S39 is executed, otherwise, step S313 is executed.

[0091] In step S39, the first virtual machine is migrated to the target numa node.

[0092] In step S310, determine whether the migration is successful, if so, continue to step S34, otherwise, execute step S311.

[0093] In step S311, first event information is reported to the control node.

[0094] In step S312, wait for the virtual machine on the target numa node to be hot migrated, and then continue to execute step S37.

[0095] In step S313, the second event information is reported to the control node.

[0096] In step S314, wait for hot migration of the first virtual machine, and then continue to execute step S35.

[0097] In some embodiments, Figure 4 As shown, if the first computing node cannot implement virtual machine migration between NUMA nodes, it will collect the reasons for the failure and report the event to the cluster control node.

[0098] In step S41, the first computing node reports event information to the cluster control node.

[0099] If the virtual machine migration fails because the target numa node is not selected, the failure reason is that other numa nodes have insufficient memory or are overloaded. The event information includes the first computing node IP and the source muna node information. The source numa node information includes the source numa node ID and the vCPU utilization of each virtual machine in the source numa node.

[0100] If the target numa node is selected, but the target numa node cannot host the first virtual machine during the virtual machine transfer process, the failure reason is insufficient memory of the numa node. The event information includes the first computing node IP, the target numa node ID, and the vCPU utilization of each virtual machine on the target numa node.

[0101] In step S42, the cluster control node performs hot migration of the virtual machine.

[0102] For example, if the virtual machine migration fails because the target numa node is not selected, the cluster control node directly migrates the first virtual machine to the second computing node.

[0103] For another example, if a target numa node is selected, but during the virtual machine transfer process, the target numa node cannot host the first virtual machine, the cluster control node hot migrates one or more virtual machines in the target numa node to the second computing node until the target numa node has enough memory to host the first virtual machine. Subsequently, step S43 is also included.

[0104] In step S43, the cluster control node sends a response message to the first computing node.

[0105] In step S44, the first computing node migrates the first virtual machine in the source numa node to the target numa node.

[0106] The first computing node attempts to migrate the first virtual machine in the source numa node to the target numa node. If the migration still fails, the event will be reported again.

[0107] In the above embodiment, by dynamically collecting the virtual machine vCPU utilization of the numa node, when there is a load imbalance between the numa nodes, the vCPU and memory of the virtual machine are migrated between different numas, and hot migration is triggered when the numa node memory is insufficient. The problem of insufficient numa node memory leading to virtual machine migration failure is solved, thereby reducing the problem of load imbalance between numa nodes affecting the operating performance of virtual machines.

[0108] Figure 5 It is a structural schematic diagram of some embodiments of the virtual machine scheduling device disclosed in the present invention. The virtual machine scheduling device is located in the first computing node and includes a CPU utilization collection module 510 and a virtual machine migration module 520.

[0109] The CPU utilization collection module 510 is configured to collect the total vCPU utilization of each numa node in the first computing node.

[0110] The virtual machine migration module 520 is configured to identify the source NUMA node of the first virtual machine to be migrated in the first computing node according to the total vCPU utilization of each NUMA node, and migrate the first virtual machine to the target NUMA node if there is a target NUMA node in the first computing node that can receive the first virtual machine. If the migration is not successful, report the first event information to the cluster control node to trigger the hot migration of the virtual machine between the first computing node and the second computing node.

[0111] In some embodiments, the second virtual machine in the target numa node is hot migrated to the second computing node so that the first virtual machine is migrated to the target numa node. The second virtual machine is the virtual machine with the lowest vCPU utilization in the target numa node.

[0112] In some other embodiments, the first virtual machine is hot migrated to the second computing node. The first virtual machine is a virtual machine with the lowest vCPU utilization in the source numa node.

[0113] In some embodiments, the target NUMA node is a NUMA node whose total vCPU utilization is less than or equal to a threshold after receiving the first virtual machine and which has sufficient memory to host the first virtual machine.

[0114] In some embodiments, identifying a source numa node of a first virtual machine to be migrated includes: identifying a numa node whose total vCPU utilization is greater than a threshold as a source numa node.

[0115] In some embodiments, the first event information includes an address of the first computing node, a target numa node identifier, and a vCPU utilization of each virtual machine of the target numa node.

[0116] In some embodiments, migrating the first virtual machine to the target numa node includes: migrating the vCPU and memory of the first virtual machine to the target numa node.

[0117] In some embodiments, the virtual machine migration module 520 is further configured to report second event information to the cluster control node when there is no target numa node, so that the cluster control node hot migrates the first virtual machine to the second computing node.

[0118] In the above embodiment, when there is an imbalance in vCPU utilization between NUMA nodes in the computing node, virtual machine migration is first performed between different NUMA nodes in the same computing node, which solves the problem of uneven vCPU utilization between NUMA nodes affecting the performance of the virtual machine. In the case of unsuccessful migration, hot migration of virtual machines between different computing nodes is triggered, reducing the impact on the operation of virtual machines.

[0119] Figure 6 Schematic diagram of the structure of some other embodiments of the virtual machine scheduling device disclosed in the present invention, the virtual machine scheduling device includes a memory 610 and a processor 620. Wherein: the memory 610 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the above embodiments. The processor 620 is coupled to the memory 610 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 620 is used to execute the instructions stored in the memory.

[0120] In some embodiments, the processor 620 is coupled to the memory 610 via a BUS 630. The virtual machine scheduling device can also be connected to an external storage device 650 via a storage interface 640 to call external data, and can also be connected to a network or another computer system (not shown) via a network interface 660. No further detailed description will be given here.

[0121] Figure 7 The structure diagram of some embodiments of the virtual machine scheduling system disclosed in the present invention includes a first computing node 710 , a second computing node 720 and a cluster control node 730 .

[0122] In this embodiment, there is a cluster control node 730 and multiple computing nodes in a cluster. When each computing node is unbalanced in load, it serves as the first computing node 710. It should be understood by those skilled in the art that the first and second here are only used to distinguish different computing nodes. The first computing node 710 includes the virtual machine scheduling device in the above embodiment, and the virtual machine scheduling device includes a CPU utilization acquisition module and a virtual machine migration module. It should be understood by those skilled in the art that when the second computing node 720 is unbalanced in load, it also performs the function of the first computing node 710, and therefore, also includes a CPU utilization acquisition module and a virtual machine migration module. The cluster control node 730 is configured to receive the first event information sent by the virtual machine scheduling device when it is determined that there is a target numa node, but the first virtual machine in the source numa node has not successfully migrated to the target numa node, and perform virtual machine hot migration between the first computing node and the second computing node. A cluster scheduling module can be set in the cluster control node 730.

[0123] In some embodiments, the cluster control node 730 is configured to hot migrate the second virtual machine in the target numa node to the second computing node, or to hot migrate the first virtual machine in the source numa node to the second computing node.

[0124] like Figure 7As shown, if the numa0 node includes virtual machine 0 and virtual machine 1, the numa1 node includes virtual machine 2. If the virtual machine migration module in the first computing node 710 determines that the total vCPU utilization of the virtual machines in the numa0 node is greater than 50%, and the vCPU utilization of virtual machine 1 is the smallest, it is determined whether the total vCPU of the numa1 node is less than 50% of the overall vCPU utilization of the numa1 node, and there is enough memory to carry the virtual machine 1. If so, the virtual machine 1 is migrated to the numa1 node. During the migration process, if the numa1 memory is insufficient to carry the virtual machine 1 migrated from the numa0 node, the event is reported to the cluster control node 730. The cluster control node 730 hot migrates virtual machine 2 to the second computing node 730. In this way, the numa1 node has enough memory to carry virtual machine 1.

[0125] Of course, during the migration process, if numa1 has insufficient memory to carry virtual machine 1 migrated from numa0 node, the cluster control node 730 may directly hot migrate virtual machine 1 to the second computing node 720 .

[0126] In some embodiments, the cluster control node 730 is further configured to receive second event information sent by the virtual machine scheduling device when it is determined that there is no target numa node, and hot migrate the first virtual machine in the source numa node to the second computing node.

[0127] For example, if the numa1 node is not sufficient to carry the virtual machine 1, the cluster control node 730 can directly hot migrate the virtual machine 1 in the numa1 node to the second computing node 730.

[0128] In other embodiments, a computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the steps of the method in the above-mentioned embodiment. It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] In some embodiments, a computer program product is protected, including a computer program or an instruction, which implements the above method when the computer program or instruction is executed by a processor. The computer program product includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by the CPU, the above functions defined in the method of the embodiment of the present disclosure are executed.

[0130] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0133] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0134] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0135] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A virtual machine scheduling method, executed by a first computing node, comprising: Identify a source numa node of a first virtual machine to be migrated in the first computing node according to a total virtual central processing unit vCPU utilization rate of each non-consistent memory access numa node in the first computing node; If there is a target numa node in the first computing node that can receive the first virtual machine, migrating the first virtual machine to the target numa node; as well as If the migration is unsuccessful, first event information is reported to the cluster control node to trigger hot migration of the virtual machine between the first computing node and the second computing node.

2. The virtual machine scheduling method according to claim 1, wherein: The hot migration of the virtual machine between the first computing node and the second computing node includes: The second virtual machine in the target numa node is hot migrated to the second computing node so that the first virtual machine is migrated to the target numa node.

3. The virtual machine scheduling method according to claim 1, wherein: The hot migration of the virtual machine between the first computing node and the second computing node includes: The first virtual machine is hot migrated to the second computing node.

4. The virtual machine scheduling method according to claim 1, further comprising: In the case that the target numa node does not exist, second event information is reported to the cluster control node so that the cluster control node hot migrates the first virtual machine to the second computing node.

5. The virtual machine scheduling method according to any one of claims 1 to 4, wherein: The target NUMA node is a NUMA node whose total vCPU utilization is less than or equal to a threshold after receiving the first virtual machine and which has sufficient memory to host the first virtual machine.

6. The virtual machine scheduling method according to any one of claims 1 to 4, wherein: The identifying the source numa node of the first virtual machine to be migrated includes: A numa node whose total vCPU utilization is greater than a threshold is identified as the source numa node.

7. The virtual machine scheduling method according to any one of claims 1 to 4, wherein: The first virtual machine is a virtual machine with the lowest vCPU utilization in the source numa node.

8. The virtual machine scheduling method according to claim 2, wherein: The second virtual machine is the virtual machine with the lowest vCPU utilization in the target numa node.

9. The virtual machine scheduling method according to any one of claims 1 to 4, wherein: The first event information includes the address of the first computing node, the target numa node identifier, and the vCPU utilization of each virtual machine of the target numa node.

10. The virtual machine scheduling method according to any one of claims 1 to 4, wherein: The migrating the first virtual machine to the target numa node comprises: Migrate the vCPU and memory of the first virtual machine to the target numa node.

11. A virtual machine scheduling device, located at a first computing node, comprising: A CPU utilization collection module is configured to collect the total virtual central processing unit vCPU utilization of each non-uniform memory access numa node in the first computing node; as well as The virtual machine migration module is configured to identify the source NUMA node of the first virtual machine to be migrated in the first computing node according to the total vCPU utilization of each NUMA node, and migrate the first virtual machine to the target NUMA node if there is a target NUMA node in the first computing node that can receive the first virtual machine, and if the migration is not successful, report the first event information to the cluster control node to trigger the hot migration of the virtual machine between the first computing node and the second computing node.

12. A virtual machine scheduling device, located at a first computing node, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the virtual machine scheduling method according to any one of claims 1 to 10 based on instructions stored in the memory.

13. A virtual machine scheduling system, comprising: A first computing node, comprising the virtual machine scheduling device according to claim 11 or 12; The second computing node; as well as The cluster control node is configured to receive the first event information sent by the virtual machine scheduling device when it is determined that there is a target non-consistent memory access numa node, but the first virtual machine in the source numa node has not been successfully migrated to the target numa node, and perform virtual machine hot migration between the first computing node and the second computing node.

14. The virtual machine scheduling system according to claim 13, wherein: The cluster control node is configured to hot migrate the second virtual machine in the target numa node to the second computing node, or to hot migrate the first virtual machine in the source numa node to the second computing node.

15. The virtual machine scheduling system according to claim 13 or 14, wherein: The cluster control node is further configured to receive second event information sent by the virtual machine scheduling device when it is determined that there is no target numa node, and hot migrate the first virtual machine in the source numa node to the second computing node.

16. A computer-readable storage medium having computer program instructions stored thereon, wherein the instructions, when executed by a processor, implement the virtual machine scheduling method according to any one of claims 1 to 10.

17. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the virtual machine scheduling method according to any one of claims 1 to 10 is implemented.

Images